Got a strange MuJoCo Warning

[toslunar]

My model diverged and the agent must have acted with inf or nan, but I got the MuJoCo Warning Unknown warning type Time = 0.0400.

/Users/tos/py3mj150/lib/python3.6/site-packages/chainer/optimizers/adam.py:60: RuntimeWarning: overflow encountered in multiply
  v += (1 - hp.beta2) * (grad * grad - v)
/Users/tos/py3mj150/lib/python3.6/site-packages/chainer/optimizers/adam.py:60: RuntimeWarning: invalid value encountered in add
  v += (1 - hp.beta2) * (grad * grad - v)
/Users/tos/py3mj150/lib/python3.6/site-packages/chainer/functions/math/minimum.py:28: RuntimeWarning: invalid value encountered in less_equal
  gx1 = gy * (x1 <= x2)
/Users/tos/py3mj150/lib/python3.6/site-packages/chainer/functions/math/minimum.py:29: RuntimeWarning: invalid value encountered in greater
  gx2 = gy * (x1 > x2)
/Users/tos/py3mj150/lib/python3.6/site-packages/chainer/functions/math/clip.py:33: RuntimeWarning: invalid value encountered in greater
  gy[0] * (self.x_min < x[0]) * (x[0] < self.x_max)),
/Users/tos/py3mj150/lib/python3.6/site-packages/chainer/functions/math/clip.py:33: RuntimeWarning: invalid value encountered in less
  gy[0] * (self.x_min < x[0]) * (x[0] < self.x_max)),
Traceback (most recent call last):
  File "nishiura/train_ppo_gym.py", line 212, in <module>
    main()
  File "nishiura/train_ppo_gym.py", line 206, in main
    clip_eps_decay_hook,
  File "/Users/tos/Documents/chainerrl/chainerrl/experiments/train_agent.py", line 144, in train_agent_with_evaluation
    logger=logger)
  File "/Users/tos/Documents/chainerrl/chainerrl/experiments/train_agent.py", line 54, in train_agent
    obs, r, done, info = env.step(action)
  File "/Users/tos/Documents/chainerrl/chainerrl/misc/env_modifiers.py", line 62, in step
    observation, reward, done, info = old_step(action)
  File "/Users/tos/py3mj150/lib/python3.6/site-packages/gym/core.py", line 96, in step
    return self._step(action)
  File "/Users/tos/py3mj150/lib/python3.6/site-packages/gym/wrappers/time_limit.py", line 36, in _step
    observation, reward, done, info = self.env.step(action)
  File "/Users/tos/py3mj150/lib/python3.6/site-packages/gym/core.py", line 96, in step
    return self._step(action)
  File "/Users/tos/Documents/chainerrl/nishiura/humanoid_env.py", line 31, in _step
    self.do_simulation(a, self.frame_skip)
  File "/Users/tos/Documents/chainerrl/nishiura/gym_env_mujoco150/mujoco_env.py", line 108, in do_simulation
    self.sim.step()
  File "mujoco_py/mjsim.pyx", line 99, in mujoco_py.cymj.MjSim.step
  File "mujoco_py/cymj.pyx", line 116, in mujoco_py.cymj.wrap_mujoco_warning.__exit__
  File "mujoco_py/cymj.pyx", line 76, in mujoco_py.cymj.c_warning_callback
  File "/Users/tos/py3mj150/lib/python3.6/site-packages/mujoco_py/builder.py", line 273, in user_warning_raise_exception
    raise MujocoException('Got MuJoCo Warning: {}'.format(warn))
mujoco_py.builder.MujocoException: Got MuJoCo Warning: Unknown warning type Time = 0.0400.

[toslunar]

Hopper-v1:

>>> e.step([nan, nan, nan])
WARNING: Nan, Inf or huge value in QACC at DOF 0. The simulation is unstable. Time = 0.0720.

(array([  1.24968608e+00,   1.74844428e-22,  -3.13984315e-21,
         1.57906690e-21,  -1.39832122e-20,   9.12424898e-20,
        -7.84800000e-02,   1.36862999e-20,  -7.52642810e-19,
         2.03085074e-19,  -3.48718327e-18]), nan, False, {})

Hopper-mujoco150-v1:

>>> e.step([nan, nan, nan])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tos/py3mj150/lib/python3.6/site-packages/gym/core.py", line 96, in step
    return self._step(action)
  File "/Users/tos/py3mj150/lib/python3.6/site-packages/gym/wrappers/time_limit.py", line 36, in _step
    observation, reward, done, info = self.env.step(action)
  File "/Users/tos/py3mj150/lib/python3.6/site-packages/gym/core.py", line 96, in step
    return self._step(action)
  File "/Users/tos/Documents/chainerrl/nishiura/gym_env_mujoco150/hopper.py", line 13, in _step
    self.do_simulation(a, self.frame_skip)
  File "/Users/tos/Documents/chainerrl/nishiura/gym_env_mujoco150/mujoco_env.py", line 108, in do_simulation
    self.sim.step()
  File "mujoco_py/mjsim.pyx", line 99, in mujoco_py.cymj.MjSim.step
  File "mujoco_py/cymj.pyx", line 116, in mujoco_py.cymj.wrap_mujoco_warning.__exit__
  File "mujoco_py/cymj.pyx", line 76, in mujoco_py.cymj.c_warning_callback
  File "/Users/tos/py3mj150/lib/python3.6/site-packages/mujoco_py/builder.py", line 273, in user_warning_raise_exception
    raise MujocoException('Got MuJoCo Warning: {}'.format(warn))
mujoco_py.builder.MujocoException: Got MuJoCo Warning: Unknown warning type Time = 0.0080.

[Nocorupe]

i think this is MuJoCo v1.50's bug.
i checked this warning with simple C++ program which applies NAN ctrl value based on MuJoCo's sample.
mjpro131 outs correct warning message, but mjpro150 outs "Unknown".

~/.mujoco/mjpro131/bin$ ./test ../model/humanoid.xml xs 10

Comparison of original and saved model
 Max difference : 2.61e-06
 Field name     : body_mass

Simulation WARNING: Nan, Inf or huge value in QACC at DOF 0. The simulation is unstable. Time = 0.0000.

~/.mujoco/mjpro150/bin$ ./test ../model/humanoid.xml xs 10

Comparison of original and saved model
 Max difference : 2.61e-06
 Field name     : body_mass

Simulation WARNING: Unknown warning type Time = 0.0000.

[kirk86]

@Nocorupe thanks for the response. So just to clarify that Unknown exception is because of the simulation environment returning Nan, Inf or Huge value? In that case how did you guys solved it. Apologies if my questions sound stupid I'm fairly new to the RL thing.

[DanielTakeshi]

Got the same error shown above:

in do_simulation
    self.sim.step()
  File "mujoco_py/mjsim.pyx", line 126, in mujoco_py.cymj.MjSim.step
  File "mujoco_py/cymj.pyx", line 156, in mujoco_py.cymj.wrap_mujoco_warning.__exit__
  File "mujoco_py/cymj.pyx", line 77, in mujoco_py.cymj.c_warning_callback
  File "/home/takeshidanny/miniconda3/envs/spinningup/lib/python3.6/site-packages/mujoco_py/builder.py", line 363, in user_warning_raise_excep
tion
    raise MujocoException(warn + 'Check for NaN in simulation.')
mujoco_py.builder.MujocoException: Unknown warning type Time = 0.0000.Check for NaN in simulation.

And it was due to having actions that were nans.