Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration with the Unity game engine #13

Open
SherAlex1998 opened this issue Aug 22, 2023 · 0 comments
Open

Integration with the Unity game engine #13

SherAlex1998 opened this issue Aug 22, 2023 · 0 comments

Comments

@SherAlex1998
Copy link

I'm going to reproduce the hands movement in Unity. Started with Twinkle Twinkle Little Star example.

At first, I saved the scene with all the resources:

export_with_assets(
      task.root_entity.mjcf_model,
      out_dir="/tmp/robopianist/piano_with_shadow_hands",
      out_file_name="scene.xml",
)

At second, I installed Unity plug-in from MuJoCo website.
At third, I changed scene assets from obj format to stl format like user pkrack wrote here
Finally, Unity has able to load this scene. But when I louched it I recived errors:

Element 'joint', line 1

NullReferenceException: Failed to create Mujoco runtime.
Mujoco.MjScene.StepScene () (at MuJoCo/Runtime/Components/MjScene.cs:345)
Mujoco.MjScene.FixedUpdate () (at MuJoCo/Runtime/Components/MjScene.cs:98)

The first occurrences of "range" in the xml debug file contain a comma in the values, not a period. Maybe that's the problem. But I do not know how to change the generation of this file)

<mujoco>
  <worldbody>
    <body pos="0 0 0" quat="-1 0 0 0" gravcomp="0" name="lh_shadow_hand/_0">
      <body pos="0.4 -0.15 0.13" quat="-0.5 -0.5 0.5 0.5" gravcomp="1" name="lh_shadow_hand/lh_forearm_1">
        <inertial pos="0 0 0.09" quat="-1 0 0 0" mass="3" diaginertia="0.0138 0.0138 0.00744" />
        <joint type="slide" pos="0 0 0" axis="-1 0 0" ref="0" armature="0.0002" springref="0" springdamper="0 0" damping="67.4762" stiffness="0" solreflimit="0.02 1" solimplimit="0.9 0.95 0.001 0.5 2" solreffriction="0.02 1" solimpfriction="0.9 0.95 0.001 0.5 2" frictionloss="0.01" limited="false" margin="0" range="-0.4605 0.7605" name="lh_shadow_hand/forearm_tx_3" />
        <joint type="slide" pos="0 0 0" axis="-1.192093E-07 0 1" ref="0" armature="0.0002" springref="0" springdamper="0 0" damping="67.4762" stiffness="0" solreflimit="0.02 1" solimplimit="0.9 0.95 0.001 0.5 2" solreffriction="0.02 1" solimpfriction="0.9 0.95 0.001 0.5 2" frictionloss="0.01" limited="false" margin="0" range="0 0.06" name="lh_shadow_hand/forearm_ty_4" />
        <geom density="1000" type="mesh" mesh="mesh_6" pos="0 0 0" quat="-1 0 0 0" priority="0" contype="0" conaffinity="0" group="2" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_0_5" />
        <geom density="1000" type="mesh" mesh="mesh_8" pos="0 0 0" quat="-1 0 0 0" priority="0" contype="0" conaffinity="0" group="2" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_1_7" />
        <geom density="1000" type="mesh" mesh="mesh_10" pos="0 0 0" quat="-1 0 0 0" priority="0" contype="1" conaffinity="1" group="3" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_2_9" />
        <geom density="1000" type="box" size="0.035 0.035 0.035" pos="0 -0.01000001 0.181" quat="0.9249092 0 0.3801881 0" priority="0" contype="1" conaffinity="1" group="3" condim="3" solmix="1" solref="0.005 1" solimp="0.5 0.99 0.0001" margin="0" gap="0" friction="1 0.005 0.0001" fluidshape="none" fluidcoef="5 0.25 1.5 1 1" name="lh_shadow_hand//unnamed_geom_3_11" />
        <site type="box" size="0.001 0.001 0.001" pos="0 0 0" quat="-1 0 0 0" name="lh_shadow_hand/forearm_tx_site_12" />
        <site type="box" size="0.001 0.001 0.001" pos="0 0 0" quat="-1 0 0 0" name="lh_shadow_hand/forearm_ty_site_13" />
        <body pos="0 -0.01000001 0.21301" quat="-1 0 0 0" gravcomp="1" name="lh_shadow_hand/lh_wrist_14">
          <inertial pos="0 1.490116E-08 0.029" quat="0.5 0.5 0.5 0.5" mass="0.1" diaginertia="6.4E-05 4.38E-05 3.5E-05" />
          <joint type="hinge" pos="0 0 0" axis="-1.192093E-07 -1 0" ref="0" armature="0.0002" springref="0" springdamper="0 0" damping="0.5" stiffness="0" solreflimit="0.02 1" solimplimit="0.9 0.95 0.001 0.5 2" solreffriction="0.02 1" solimpfriction="0.9 0.95 0.001 0.5 2" frictionloss="0.01" limited="false" margin="0" range="-30,00002 10" name="lh_shadow_hand/lh_WRJ2_16" />

...

I attached this file to post, you may look it.

As a result, maybe it would be better to somehow copy the frame-by-frame hand positions to make an animated clip? Where can I get these values from an xml file generated by a neural network?
Importing the environment as it is in Unity seems to be a more confusing task.
debug.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@SherAlex1998