The performance of Mujoco Swimmer is low.

[caozhangjie]

• I have marked all applicable categories:
  • exception-raising bug
  • RL algorithm bug
  • documentation request (i.e. "X is missing from the documentation.")
  • new feature request
• I have visited the source website
• I have searched through the issue tracker for duplicates
• I have mentioned version numbers, operating system and environment, where applicable:

  import tianshou, torch, numpy, sys
  print(tianshou.__version__, torch.__version__, numpy.__version__, sys.version, sys.platform)

I have run another RL algorithm here on Mujoco Swimmer-v3 and get the reward above 300. I'm not sure why tianshou can only achieve a lower than 100 reward for Swimmer.

[Trinkle23897]

@ChenDRAG

[ChenDRAG]

@caozhangjie Achieving a reward of 300 in Mujoco Swimmer is rarely seen in the literature. As a matter of fact, I personally don't know any public algorithm that can get that high reward. The algorithm TRPO you seem to use can only achieve a reward of 120 in PPO's paper, and can only achieve a reward of ~80 in OPENAI baselines(see https://github.com/thu-ml/tianshou/tree/master/examples/mujoco).
Could you specify your detailed environment settings and provide training curves?

[caozhangjie]

You can clone his repo and replace the main.py with my attached main.py, where I write an evaluate function. I just do 20 rollouts to evaluate the policy (without discount factor). You can run python main.py --env-name Swimmer-v3 --batch-size 25000 --log-interval 10 in the repo. It will give a reward of about 300 in the 20 or 30 episodes (5 minutes with 1 cpu).
main.py.zip

[Trinkle23897]

I change one configuration (--log-interval from 5 to 10, same as changing seed):

However the performance is still better than tianshou. I think it is because you change the observation normalization, but this change also affects other tasks' performance. It is quite possible that you get the best performance on swimmer but meanwhile fails on ant/halfcheetah. (People usually run mujoco benchmark with same configuration for all environments)

[araffin]

hill-a/stable-baselines#500 (comment)

[Trinkle23897]

hill-a/stable-baselines#500 (comment)

Great thanks!!! But I have a question: how did you change the sensor from neck to head? Is there any forked repo I can refer?

[araffin]

you don't need to change the sensor (see my last comment).