[Feature Request] Gymnasium: Add options argument to VecEnv.reset()

[ReHoss]

🚀 Feature

Hello,

As gymnasium implements the very useful options argument in the .reset() method, it would be nice to propagate it to the VecEnv.reset() notably to reset the environment in various states (useful for instance during evaluation phases).

Same for the seed argument even though an other workaround has been chosen.

Would it cause issues if we just add option=None to the signatures ?

PS: When writing gymnasium environments now, IDE (e.g PyCharm) triggers warnings if the signature of the custom environments don't match with the abstract class, so it encourages defining options and seed arguments.

Have a good day,
Thanks,

Motivation

No response

Pitch

No response

Alternatives

No response

Additional context

No response

Checklist

• I have checked that there is no similar issue in the repo

[araffin]

Hello,
same as seed (#1486), VecEnv will not provide a seed argument (see discussion in #1481 and for a longer answer, you can read the full discussion in #780).

However, as for seed, I think it would be ok to provide a set_options() method that will be passed at the next reset.

Alternatives

In the meantime, you can always use the env_method() (cf. doc) to call the gym envs and prepare the options for the next reset.

PS: When writing gymnasium environments now, IDE (e.g PyCharm) triggers warnings if the signature of the custom environments don't match with the abstract class,

VecEnv is not a gym env.

All the differences: https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#vecenv-api-vs-gym-api

PS: please don't forget to fill the "alternative" part next time

[ReHoss]

Hello, same as seed (#1486), VecEnv will not provide a seed argument (see discussion in #1481 and for a longer answer, you can read the full discussion in #780).

However, as for seed, I think it would be ok to provide a set_options() method that will be passed at the next reset.

Alternatives

In the meantime, you can always use the env_method() (cf. doc) to call the gym envs and prepare the options for the next reset.

This is what I implemented so far:

observations = env.reset()
env.env_method("reset", **{"array_x0": array_x0})

When evaluating a trajectory I first call the VecEnv.reset method in order to comply with the interface.
Then I call the unwrapped reset method.

I would like to run only reset once.

If I remove the second line to keep the below,

env.env_method("reset", **{"array_x0": array_x0})

Do you agree I am going to mess something up with the Monitor wrapper, when logging reward for instance ?

PS: When writing gymnasium environments now, IDE (e.g PyCharm) triggers warnings if the signature of the custom environments don't match with the abstract class,

VecEnv is not a gym env.

I am not talking about VecEnv.
In other words, when working with custom environments, the stable-baselines3 users implement gymnasium environments. Indeed, those environments are later wrapped (e.g. as a DummyVecEnv).
However, when the user designs its custom gymnasium environment, warnings/code analysis suggest to add options and seed arguments to the signature in order to match gym>=0.28.
I am just giving another argument aside @Kallinteris-Andreas one, : the VecEnv signatures don't match with the gym>=0.28 signatures which is a bit misleading (at least I was fooled when I migrated).

All the differences: https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#vecenv-api-vs-gym-api

Thank you!

PS: please don't forget to fill the "alternative" part next time

[araffin]

Hello,
there might be some misunderstandings.

I meant that with both gym 0.21/0.26 or gymnasium, you can do:

class CustomEnv(gym.Env):
    def reset(self):
        # Use seed and options
        self._reset_state(self._seed, self._options)
        # Reset seed and options
        self._options = {}
        self._seed = None
        return self.get_obs()

    def set_seed(self, seed: int):
        self._seed = seed

    def set_options(self, options):
        # Set options for the next reset
        self._options = options



vec_env = DummyVecEnv([lambda: CustomEnv()])
vec_env.env_method("set_seed", 0)
vec_env.env_method("set_options", dict(a=1))
vec_env.reset()

and you don't need anything more to have the same features.
This doesn't mess up the logging.

As I said, I would welcome a PR that would implement something similar to #1486 for options.

: the VecEnv signatures don't match with the gym>=0.28 signatures which is a bit misleading (at least I was fooled when I migrated).

(One can also say that gym>=0.28 signatures don't match VecEnv ones, the main misleading part are gym breaking changes.)

Anyway, VecEnv interface is the same since Stable-Baselines v1.0 and we tried recently to make it clearer in the doc, with warnings and consistently using vec_env for variable names instead of env.

The consistency of VecEnv API is key in the stability of Stable-Baselines and related projects.
For instance, it allowed to switch seamlessly the SB3 jax version from gym to gymnasium, making sure no regression were introduced.