Which Bodies does MuJoCo.Humanoid have

[Kallinteris-Andreas]

Question

Mujoco.Humanoid's documentation says it has 14 bodies (nbody=14)
https://gymnasium.farama.org/environments/mujoco/humanoid/
but I can only count 13 in the xml file
https://github.com/Farama-Foundation/Gymnasium/blob/main/gymnasium/envs/mujoco/assets/humanoid.xml

        # bodies
        torso = 0
        lwaist = 1
        pelvis = 2
        right_thigh = 3
        right_sin = 4
        right_foot = 5
        left_thigh = 6
        left_sin = 7
        left_foot = 8
        right_upper_arm = 9
        right_lower_arm = 10
        left_upper_arm = 11
        left_lower_arm = 12

am I missing something, which is the 14th body

Thanks!

[pseudo-rnd-thoughts]

I suspect there is an issue with the docs as it looks like there is a missing angle 2 given everything is mirrored

https://gymnasium.farama.org/environments/mujoco/humanoid/

[Kallinteris-Andreas]

@pseudo-rnd-thoughts no that is an unrelated issue
(here is a PR to fix it anyway: #206)

@rodrigodelazcano can you answer this?

[rodrigodelazcano]

Thanks for bringing this up @Kallinteris-Andreas. This actually opens another discussion/fix that we should make to the mujoco environments.

The quick answer is that the worldbody is also considered a body in mujoco, thus you'll have to add world=0 to the list (in mujoco the worldbody is accessed with the name world, model.body('world').id should be 0).

Now, the next question to be answered is if the observations related to this body should be included in the observation space since they will be 0 and are not intrinsic to the agent. Same happens with the other mujoco environments like Ant.

We need to test if this is true like in #214 and then actually consider a possible v5 version for this situation. This fix and new version can wait until we make the transfer of the mujoco environments to Gymnasium-Robotics since the research up til today has been done with the current observation space.

[Kallinteris-Andreas]

Damn it,
Was really hoping the answer, would not be worldbody

[Kallinteris-Andreas]

import gymnasium
import numpy as np

env = gymnasium.make("Humanoid-v4")
env.reset()

for _ in range(1000):
    env.step(env.action_space.sample())
    assert np.all(env.unwrapped.data.cinert[0] == 0)
    assert np.all(env.unwrapped.data.cvel[0] == 0)
    assert np.all(env.unwrapped.data.cfrc_ext[0] == 0)
    assert np.all(env.unwrapped.data.qfrc_actuator[:6] == 0)

1. worldbody is 100% inert
2. I have no explanation why as to qfrc_actuator[:6] is all zeroes

[rodrigodelazcano]

Yes, this is another redundant observation. Reading the MuJoCo docs I found an explanation (https://mujoco.readthedocs.io/en/latest/computation.html?highlight=moment%20arms#actuation-model)

qfrc_actuator is the actuator force in joint coordinates, joint_moment_arm * actuator_force (including joints without actuators, nv size). This means that the free joint of the head is also included in the qfrc_actuator vector and will always be 0.
Then there is this other attribute data.actuator_force which returns the actuator output torques (size nu) and is equal to the Gymnasium action since Humanoid uses motor actuators.

My suggestion is to keep qfrc_actuator in the observation space since the joint moment arms is relevant for extracting information about the bodies inertia but removing the elements related to the un-actuated free joint.

[Kallinteris-Andreas]

V4 with contact forces (BLUE) and v5 (GREEN) with contact forces, have particularly identical training.
The only difference in v5 is the exclusion of worldbody contact forces.

[pseudo-rnd-thoughts]

It is difficult to tell, but green looks to have high variances at the beginning then stabilise with blue while orange has a massive variance.

What are our guiding ideas of when to make a change and when to not?

[Kallinteris-Andreas]

Orange (no ctn), having massive variance, is because of effectively more exploration (contact forces are sparse).

Comparing blue and green. They have close enough variance. (Which is exactly the expected behavior)

[Kallinteris-Andreas]

Same for Humanoid-v5
if we compare v4 with fixed reward (includes contact forces), and v5 (which excludes the worldbody and root observations) we can see similar performance (Which is exactly the expected behavior)

Note: throughout the training of v5 it was asserted that the worldbody and root observations equal 0

[Kallinteris-Andreas]

import gymnasium
import numpy as np
import time

#env = gymnasium.make('Ant-v5', render_mode=None)
#env = gymnasium.make('Humanoid-v5', render_mode=None)
#env = gymnasium.make('Humanoid-v5', xml_file='/home/master-andreas/op3-scene.xml', render_mode=None)
env = gymnasium.make('Humanoid-v5', xml_file='/home/master-andreas/casie-scene.xml', render_mode=None)

env.reset()
#breakpoint()

for ep in range(100):
    for step in range(10000):
        env.step(env.action_space.sample())
        #time.sleep(0.05)
        assert np.all(env.unwrapped.data.cinert[0] == 0)
        assert np.all(env.unwrapped.data.cvel[0] == 0)
        assert np.all(env.unwrapped.data.cfrc_ext[0] == 0)
        assert np.all(env.unwrapped.data.qfrc_actuator[:6] == 0)
        #assert np.all(env.unwrapped.data.qfrc_actuator[-2] == 0)

        #print(env.unwrapped.data.qfrc_actuator)

#breakpoint()

For the sake of completeness I tested 3rd party models cassie & op3
and confirmed that the worldbody observations are constantly 0
also the first 6 qfrc_actuator observations are also constantly 0, because they are the freejoint (similarly to root joint in the "humanoid.xml")

[Kallinteris-Andreas]

Here is the v4 vs v5 humanoidstandup (the only different is the exclusion of the constant 0 worldbody and freejoint observations)
We can see that the behavior (as with the other environments) is similar