[Bug Report] Possible bug with bootstrapping when environment is truncated in CleanRL mutli-agent Atari example

[KaleabTessera]

Describe the bug

I think there is possibly a bug with the clean rl examples or any examples using pettingzoo_env_to_vec_env_v1. Truncation should be treated differently to termination when bootstrapping - as done in stable baselines (DLR-RM/stable-baselines3#633).

I have also made comments on the clean rl PR - vwxyzjn/cleanrl#408.

In short, we should still bootstrap when our environment is truncated, where in the examples we treat truncation and termination the same

PettingZoo/tutorials/CleanRL/cleanrl_advanced.py

Line 265 in c65e894

next_done = torch.maximum(next_termination, next_truncation)

and

PettingZoo/tutorials/CleanRL/cleanrl_advanced.py

Line 266 in c65e894

dones = torch.maximum(terminations, truncations)

.

This also means we need this PR to be merged in to get the correct terminal obs and then implement something similar to how sb3 do it.

Code example

No response

System info

No response

Additional context

No response

Checklist

• I have checked that there is no similar issue in the repo

[elliottower]

I and Costa commented on that other issue and PR but I think we're going to be best off using his rewritten version of the tutorial when that is finished, but this issue can stay up until then so I don't forget.