Restart crashed Unity environments

[hvpeteet]

Proposed change(s)

Update the SubprocessEnvManager to restart workers when the underlying Unity environments crash.
When a worker receives an ENV_EXITED signal it will now:

1. Record all failures coming through the step queue and drop all other messages.
2. Purge any pending trajectories as they may belong to a crashed worker or be corrupted.
3. Restart all failed workers (up to a configurable limit)

This behavior can be limited via a rate limit, max lifetime limit, or both. The configuration options for both are shown below with their default values.

⚠️ Each of these options applies to a single environment, if num_envs > 1 then the limit will apply separately to each replica (num_envs = 2 will spawn 2 Unity environments which can each be restarted 10 times).

env_settings:
  # Can restart 10 times over the lifetime of the experiment.
  max_lifetime_restarts: 10
  # Rate limit of 1 failure per 60s
  restarts_rate_limit_n: 1
  restarts_rate_limit_period_s: 60

They can of course be passed via CLI arguments as well

--max-lifetime-restarts
--restarts-rate-limit-n
--restarts-rate-limit-period-s

Disabling this feature

• Rate limiting can be turned off by setting --restarts-rate-limit-n=-1
• Lifetime limiting can be turned off by setting --max-lifetime-restarts=-1

Useful links (Github issues, JIRA tickets, ML-Agents forum threads etc.)

Internal JIRA: https://jira.unity3d.com/browse/MLA-1344

Types of change(s)

• New feature

Checklist

• Added tests that prove my fix is effective or that my feature works
• Updated the changelog (if applicable)
• Updated the documentation (if applicable)

[miguelalonsojr]

Big fat green light!!!!

[maryamhonari]

Looks great!