Discrete action space

[szkLaszlo]

Hi, I have a paper review where the reviewer asked for some SafeRL method to test against. Your framework looks just what I need, however, I can not see if any of the algos are working with discrete environments. Can you help me out in this? I would run the CPO or SAC on my custom env to test against my algorithm. I would try to build my custom env from your tutorials, however I see the assertion regarding the box action spaces. Here is my env: https://github.com/szkLaszlo/PrefVec/blob/main/other_envs/highway_cl.py. CumulantIntersectionEnv should be used, but we use discrete action spaces.

[Gaiejj]

We have implemented this feature in pull request, currently supporting discrete action spaces for on-policy algorithms. You are welcome to try it out and perhaps attempt an off-policy version based on this.

[szkLaszlo]

Thank you. I will try to apply this Discrete wrapper to my custom gym :)

[szkLaszlo]

I checked out the branch, created a virtual env and started to run the following code:
`import omnisafe

env_id = 'Taxi-v3'
custom_cfgs = {
'train_cfgs': {
'total_steps': 10000000,
'vector_env_nums': 1,
'parallel': 1,
},
'algo_cfgs': {
'steps_per_epoch': 20000,
},
'logger_cfgs': {
'use_wandb': False,
'use_tensorboard': True,
},
}

agent = omnisafe.Agent('CPO', env_id, custom_cfgs=custom_cfgs)
agent.learn()`

I get the following error:

Loading CPO.yaml from /home/szl95/workspace/omnisafe_zjy/omnisafe/utils/../configs/on-policy/CPO.yaml
Logging data to ./runs/CPO-{Taxi-v3}/seed-000-2024-01-22-16-14-32/progress.csv
Save with config in config.json
INFO: Start training
Processing rollout for epoch: 0... 0% -:--:--
Traceback (most recent call last):
File "/mnt/c/Program Files/JetBrains/PyCharm 2022.2/plugins/python/helpers/pydev/pydevd.py", line 1534, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/mnt/c/Program Files/JetBrains/PyCharm 2022.2/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/mnt/c/Users/zlo2bp/workspace/PrefVeC/safe_algo/safe_experiments.py", line 21, in
agent.learn()
File "/home/szl95/workspace/omnisafe_zjy/omnisafe/algorithms/algo_wrapper.py", line 179, in learn
ep_ret, ep_cost, ep_len = self.agent.learn()
File "/home/szl95/workspace/omnisafe_zjy/omnisafe/algorithms/on_policy/base/policy_gradient.py", line 246, in learn
self._env.rollout(
File "/home/szl95/workspace/omnisafe_zjy/omnisafe/adapter/onpolicy_adapter.py", line 86, in rollout
next_obs, reward, cost, terminated, truncated, info = self.step(act)
File "/home/szl95/workspace/omnisafe_zjy/omnisafe/adapter/online_adapter.py", line 170, in step
return self._env.step(action)
File "/home/szl95/workspace/omnisafe_zjy/omnisafe/envs/wrapper.py", line 620, in step
obs, reward, cost, terminated, truncated, info = super().step(action)
File "/home/szl95/workspace/omnisafe_zjy/omnisafe/envs/core.py", line 244, in step
return self._env.step(action)
File "/home/szl95/workspace/omnisafe_zjy/omnisafe/envs/wrapper.py", line 161, in step
obs, reward, cost, terminated, truncated, info = super().step(action)
File "/home/szl95/workspace/omnisafe_zjy/omnisafe/envs/core.py", line 244, in step
return self._env.step(action)
File "/home/szl95/workspace/omnisafe_zjy/omnisafe/envs/wrapper.py", line 103, in step
obs, reward, cost, terminated, truncated, info = super().step(action)
File "/home/szl95/workspace/omnisafe_zjy/omnisafe/envs/core.py", line 244, in step
return self._env.step(action)
File "/home/szl95/workspace/omnisafe_zjy/omnisafe/envs/discrete_env.py", line 130, in step
obs, reward, terminated, truncated, info = self._env.step(
File "/usr/local/lib/python3.10/dist-packages/gymnasium/wrappers/autoreset.py", line 55, in step
obs, reward, terminated, truncated, info = self.env.step(action)
File "/usr/local/lib/python3.10/dist-packages/gymnasium/wrappers/time_limit.py", line 57, in step
observation, reward, terminated, truncated, info = self.env.step(action)
File "/usr/local/lib/python3.10/dist-packages/gymnasium/wrappers/order_enforcing.py", line 56, in step
return self.env.step(action)
File "/usr/local/lib/python3.10/dist-packages/gymnasium/wrappers/env_checker.py", line 47, in step
return env_step_passive_checker(self.env, action)
File "/usr/local/lib/python3.10/dist-packages/gymnasium/utils/passive_env_checker.py", line 237, in env_step_passive_checker
result = env.step(action)
File "/usr/local/lib/python3.10/dist-packages/gymnasium/envs/toy_text/taxi.py", line 286, in step
transitions = self.P[self.s][a]
TypeError: unhashable type: 'numpy.ndarray'

Process finished with exit code 1

I tested it in windows, in linux and the result is the same. Is it possible that the current version of the code is not running? Could you help me out?

Thanks in advance.

[Gaiejj]

Unfortunately, I wasn't able to reproduce it locally. You might try using Python 3.8 or the same version of Gymnasium, 0.28.1 that I am using.