Improving offline RL scripts

[dosssman]

Problem Description // Current behavior

A reminder for the cleanrl/offline related issues mentioned in #130

1. SPS logging is missing (attempted to match Add SPS and q-values metrics for value-based methods #126 Tweaks for PR 'add-sps-qvalues' #130

2. Monitor cannot be imported anymore due to gym=0.23.0 update

3. Tests for those two scripts are missing

4. Unlike dqn_atari, the wrappers are not imported from SB3

5. The offline-env-id that is required to load the dataset does not seem to work anymore. Is there any dependency missing, such as d4rl or d4rl_atari for example ?

(cleanrl) d055@kara:~/random/rl/cleanrl/cleanrl/offline$ python offline_dqn_atari_visual.py 

A.L.E: Arcade Learning Environment (version 0.7.4+069f8bd)
[Powered by Stella]
Traceback (most recent call last):
  File "offline_dqn_atari_visual.py", line 559, in <module>
    data_loader = iter(torch.utils.data.DataLoader(ExperienceReplayDataset(), batch_size=args.batch_size, num_workers=2))
  File "offline_dqn_atari_visual.py", line 544, in __init__
    self.dataset_env = gym.make(args.offline_env_id)
  File "/home/d055/anaconda3/envs/cleanrl/lib/python3.8/site-packages/gym/envs/registration.py", line 676, in make
    return registry.make(id, **kwargs)
  File "/home/d055/anaconda3/envs/cleanrl/lib/python3.8/site-packages/gym/envs/registration.py", line 490, in make
    versions = self.env_specs.versions(namespace, name)
  File "/home/d055/anaconda3/envs/cleanrl/lib/python3.8/site-packages/gym/envs/registration.py", line 220, in versions
    self._assert_name_exists(namespace, name)
  File "/home/d055/anaconda3/envs/cleanrl/lib/python3.8/site-packages/gym/envs/registration.py", line 297, in _assert_name_exists
    raise error.NameNotFound(message)
gym.error.NameNotFound: Environment `breakout-expert` doesn't exist. Did you mean: `Breakout-ram`?

Possible Solution

1. Straight forward to add, but did not want to overload the Tweaks for PR 'add-sps-qvalues' #130

2. and 4. : Make do without Monitor, use SB3's wrapper instead

3. Straight forward to add

4. Investigate the missing dependencies, as well as the generation of the args.offline_env_id

[vwxyzjn]

Hey @dosssman thanks for bringing this up. I think maybe we should remove these scripts from the main repo and put them in a separate branch. After all, we haven't done a thorough evaluation of these scripts to make sure of their implementation quality.

[vwxyzjn]

It looks like there is a really good single-file implementation offline RL library! tinkoff-ai/CORL#1