[Feature Request] Support for VecMonitor for gym3-style environments

[vwxyzjn]

🚀 Feature

Gym3-style environments such a procgen directly produces the vectorized environments, so there is no chance of passing a Monitor wrapper to the envs creation process. However, there still is a need to record the episodic stats. One potential way to do this is through the VecMonitor wrapper as done in openai/baselines

### Check list

• I have checked that there is no similar issue in the repo (required)

[araffin]

Hello,

I think VecMonitor would be a good addition, however, I'm not sure that this would make SB3 compatible with Gym3 by any mean...
Also, apply VecMonitor at that stage won't give the true reward if wrappers are used before.

[vwxyzjn]

Hi Antonin,

Thanks for the prompt response.

I'm not sure that this would make SB3 compatible with Gym3 by any mean...

I think this addition should at least push SB3 closer to be compatible with Gym3 envs, that has an API that produces the SB3-style vectorized env. For example, in procgen, they used baselines to train and initialize the env in the following way

# https://github.com/openai/train-procgen/blob/1a2ae2194a61f76a733a39339530401c024c3ad8/train_procgen/train.py#L36-L43
venv = ProcgenEnv(num_envs=num_envs, env_name=env_name, num_levels=num_levels, start_level=start_level, distribution_mode=distribution_mode)
venv = VecExtractDictObs(venv, "rgb")
venv = VecMonitor(
    venv=venv, filename=None, keep_buf=100,
)
venv = VecNormalize(venv=venv, ob=False)

apply VecMonitor at that stage won't give the true reward if wrappers are used before.

That's a great catch. As shown by the snippet above, the users should be quite careful about the order in which the vec wrapper is applied (e.g. VecMonitor should be applied before VecNormalize); same as how regular Monitor wrapper should be applied with care.

[araffin]

I think this addition should at least push SB3 closer to be compatible with Gym3 envs, that has an API that produces the SB3-style vectorized env. For example, in procgen, they used baselines to train and initialize the env in the following way

Interesting. Then it should work (but the env must be always wrapped with a VecEnvWrapper (VecMonitor should do the job) as we are doing internal checks and wrapping it automatically (in a DummyVecEnv) if needed, but here if we do that, it will fail).

[araffin]

In case it was not clear, you can go ahead and implement the VecMonitor, make sure to read the contributing guide ;)

[vwxyzjn]

Hey sorry for the delay. I made a fork and added the changes but was having trouble setting up test cases; this could take some time. Additionally, I noticed the existing gym monitor class also has the feature to save a CSV; would this be a desired feature as well?

[araffin]

fixed in #311