MoCap data from DLC

[wangcongrobot]

Hi, thank you for the nice project! I am wondering if the Data from DLC can be used for the imitation? Such as tracking the 4 feet only? Thank you

[talmo]

Hi @wangcongrobot,

Thanks for the interest! We're not quite at prime time yet to officially release the package and provide extensive external support, but if you'd like to give it a go yourself, you can check out this bit of the code for how to implement a new data loader.

Alternatively, we already support NWB via the ndx-pose extension, so if you can save out your 3D poses into that format, the existing loader should just work.

Cheers,

Talmo

[wangcongrobot]

Thanks a lot! I will try it!

[wangcongrobot]

@talmo Hi Talmo, Thank you for your response. I'm fine with creating NWB data from DLC, but I have two questions that I'm unsure about:

• Getting full mocap data, like in this paper, seems challenging. Would it be possible to work with only part of the full keypoints? For example, tracking 5-6 keypoints using a single camera in DLC.

• Is camera calibration necessary since the raw data from DLC appears to be in pixels?

Thanks in advance!

[talmo]

Right now, the expected input is 3D keypoints, so unfortunately that might not work with your existing data if they're in 2D.

Additionally, the inverse kinematics will be quite challenging without enough keypoints on the rest of the body.

[wangcongrobot]

Can we train a base motion model with full 3D keypoints and then use part of the keypoint to guide the whole-body motion?