Multi-Agent RL with Ray RLlib for large scenario

[liaoq-blcr]

Hi Stefan,

Many thanks for the great work!

I tried to customize my own multi-agent environment following the same structure, and found that when the number of agents is large, I got nan reward in the logs and num_env_steps_sampled_lifetime as 0 in results.json. So I tested your "mobile-large-ma-v0" environment, and similarly, I got the same problem -- seems like the sample collection and training for the large scenario do not work for large scenarios. However, with small environment, "mobile-small-ma-v0", everything looks fine with rllib.

Have you experienced something similar? Could you please give some hints? Thank you so much!

│ env_runners/episode_len_mean nan │
│ env_runners/episode_return_mean nan │
│ num_env_steps_sampled_lifetime 0 │

[stefanbschneider]

Hi @liaoq-blcr ,

Thanks for using mobile-env and reporting the issue. I tried to look into it and can reproduce the behavior that no episodes or steps are logged when switching from the small to the large multi-agent environment.

I don't really understand where this difference comes from myself. I hope to find more time to look into this and will update you if I find something.

Vice versa, if you find out what the cause is in the meantime, please let me know / post it here. Thanks!

[stefanbschneider]

@liaoq-blcr Could you try if the issue still persists with the latest version of mobile-env (2.0.2) and Ray? I just merged a PR to update some stuff, including the Ray RLlib notebook. Now, I don't see the issue with the large environment anymore.

[liaoq-blcr]

@liaoq-blcr Could you try if the issue still persists with the latest version of mobile-env (2.0.2) and Ray? I just merged a PR to update some stuff, including the Ray RLlib notebook. Now, I don't see the issue with the large environment anymore.

Hi Stefan, thank you so much! it works now also for the large environment. Many thanks for solving the issue!