The problem of adding new motion capture data for human motion tracking

[rottenapple777]

In the human motion tracking task, I want to add new motion capture data, but the amc file I downloaded from CMU is different from the xml of the keyframe in mjpc. Is some format conversion program being used? How do I get the new motion capture data file in the new xml format. Thanks a lot.

[thowell]

@hartikainen

[hartikainen]

Unfortunately, I haven't open-sourced any of these functions as they're overly complicated and not very user friendly for anyone else than myself. It's been long on my todo-list to clean up the IK-code but I'm honestly not sure if I'll ever get there.

You need to transform the .amc-files into our .xml format. For that, you basically have to extract the cartesian joint/body positions for each frame and order them like they're used here:

mujoco_mpc/mjpc/tasks/humanoid/tracking/tracking.cc

Lines 69 to 73 in 1f78168

	const std::array<std::string, 16> body_names = {
	"pelvis", "head", "ltoe", "rtoe", "lheel", "rheel",
	"lknee", "rknee", "lhand", "rhand", "lelbow", "relbow",
	"lshoulder", "rshoulder", "lhip", "rhip",
	};

That should be relatively easy to do. Just note that you'll have to rescale the positions with a constant value to get them right. Then just write the each frame into an .xml file like here:

mujoco_mpc/mjpc/tasks/humanoid/tracking/keyframes/CMU-CMU-02-02_04_poses.xml

Line 3 in 1f78168

<key name="jump_1" mpos="0.0 0.0 0.9079 0.02246 -0.01716 1.48528 0.05642 0.11259 0.02555 0.05707 -0.10348 0.01632 -0.06685 0.06839 0.05535 -0.06679 -0.06662 0.05473 0.03258 0.09748 0.4423 0.05061 -0.09743 0.43688 0.07061 0.19745 0.86066 0.02813 -0.23827 0.85899 -0.07446 0.17572 1.06235 -0.11842 -0.18084 1.06022 -0.02793 0.17321 1.31954 -0.07074 -0.17717 1.31069 0.00543 0.07048 0.81714 -0.00302 -0.06647 0.81644" qpos="-0.04934 -0.00198 1.25512 0.99691 0.0161 -0.04859 -0.05959 0.00806733 -0.0921235 -0.00874875 -0.0488637 -0.0894064 0.0219203 -0.432015 -0.374747 -0.0965783 0.0391758 -0.0760225 0.132042 -0.256406 -0.350065 -0.494566 0.714612 -0.744779 -0.922223 0.818614 -0.698744 -1.00263" qvel="0.2013 -0.009899999999999999 0.0525 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0" />

That is, each key in the .xml contains 48 = 16 * 3 (cartesian xyz position for the 16 joints) mpos values.

What's a bit trickier is if you also want to use reset positions for the motions. These are controlled by the first keyframe's qpos and qvel values (like in the above example key). For these, you're gonna have to do some inverse kinematics because the .amc-files don't include those for our humanoid. For the inverse kinematics, I've used the multi-site IK functions from here: google-deepmind/dm_control#399.

[hartikainen]

I looked around and noticed that I had opened a PR (https://github.com/hartikainen/mocap-environments/pull/6/files) for converting the AMASS files (which also contains most of the CMU mocap motions) into our .xml format. That might give you some ideas on how to approach the conversion. But be warned: as I mentioned above, the code is not super usable right now and it's been so long since I opened the PR that the code might be buggy/stale already.

[yuvaltassa]

@rottenapple777 if you would clean that up and make it usable, that would be a much-appreciated contribution 👍

[rottenapple777]

@hartikainen Okay, I'll check it out. Thanks for taking the time to reply.