I can't reproduce the Atari results reported in your paper. I directly use your code with your hyper parameters, with same package versions. However, I can't reproduce your results. The problem also holds for MuJoCo as I explained in another issue. Can you please provide correct hyperparameters to reproduce your results?