handle multiple dones in a single step

[chriselion]

Proposed change(s)

Fix two bugs in the gym interface that could happen if an Agent called EndEpisode() multiple times in the same step:

1. The bookkeeping would try to remove the "transient" agent before it was added, which would fail. We also need to avoid adding it since it's already done.
2. For a single Agent, the checks in _sanitize_info() was too strict - we could have the case where step_result.n_agents()=3 self._n_agents=1 n_extra_agents=2 because there we 2 "done" agents in the step.

This would show up with stack traces:

Traceback (most recent call last):
  File "test_gym.py", line 38, in <module>
    observation, reward, done, info = multi_env.step(multi_env.action_space.sample())
  File "/Users/chris.elion/code/ml-agents/gym-unity/gym_unity/envs/__init__.py", line 222, in step
    step_result = self._step()
  File "/Users/chris.elion/code/ml-agents/gym-unity/gym_unity/envs/__init__.py", line 449, in _step
    return self._sanitize_info(info)
  File "/Users/chris.elion/code/ml-agents/gym-unity/gym_unity/envs/__init__.py", line 371, in _sanitize_info
    "The number of agents in the scene does not match the expected number."
gym_unity.envs.UnityGymException: The number of agents in the scene does not match the expected number.

with multiagent=False, or

Traceback (most recent call last):
  File "test_gym.py", line 32, in <module>
    observations, rewards, dones, info = multi_env.step(actions)
  File "/Users/chris.elion/code/ml-agents/gym-unity/gym_unity/envs/__init__.py", line 222, in step
    step_result = self._step()
  File "/Users/chris.elion/code/ml-agents/gym-unity/gym_unity/envs/__init__.py", line 445, in _step
    self.agent_mapper.mark_agent_done(agent_id, reward)
  File "/Users/chris.elion/code/ml-agents/gym-unity/gym_unity/envs/__init__.py", line 531, in mark_agent_done
    gym_index = self._agent_id_to_gym_index.pop(agent_id)
KeyError: 2

with multiagent=True

Useful links (Github issues, JIRA tickets, ML-Agents forum threads etc.)

https://forum.unity.com/threads/errors-when-using-gym-unity-0-14-1-and-0-15-0.852586/

#3531 (comment)

https://jira.unity3d.com/browse/MLA-805

Types of change(s)

• Bug fix
• New feature
• Code refactor
• Breaking change
• Documentation update
• Other (please describe)

Checklist

• Added tests that prove my fix is effective or that my feature works
• Updated the changelog (if applicable)
• Updated the documentation (if applicable)
• Updated the migration guide (if applicable)

Other comments

[chriselion]

This is where the reported exceptions are. We could also make a check before the call to mark_agent_done, in _sanitize_info() but it seemed a bit cleaner to do it here.

[chriselion]

Just making the mark_agent_done() change without this means that we'll register the agent and run out of "slots" later.

[chriselion]

I'm not 100% it's safe to remove this, but it's what was triggering in the single agent case (with 2 "done" agents)

[ervteng]

Looks good - though we really need to simplify the batched step result output, the gym wrapper is getting pretty fat

[chriselion]

Bypassing C# yamato tests.