fix qvalue mask_action error for obs_next

[Trinkle23897]

No description provided.

[codecov-io]

Codecov Report

Merging #310 (dc3833f) into master (243ab43) will decrease coverage by 0.03%.
The diff coverage is 91.66%.

@@            Coverage Diff             @@
##           master     #310      +/-   ##
==========================================
- Coverage   93.94%   93.91%   -0.04%     
==========================================
  Files          51       51              
  Lines        3270     3270              
==========================================
- Hits         3072     3071       -1     
- Misses        198      199       +1     

Flag	Coverage Δ
unittests	93.91% <91.66%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
tianshou/env/venvs.py	90.71% <0.00%> (-0.72%)	⬇️
tianshou/policy/modelfree/dqn.py	98.78% <93.75%> (ø)
tianshou/policy/modelfree/c51.py	93.61% <100.00%> (ø)
tianshou/policy/modelfree/qrdqn.py	93.18% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 243ab43...dc3833f. Read the comment docs.

[danagi]

I think the problem in #309 is caused by the function _target_q which uses unmasked logits to get target max q values in dqn.py. Since self(...).act is the legal max q value action, my personal suggestion is as follow:

model = 'model_old' if self._target else 'model'
result = self(batch, model=model, input='obs_next')
a = result.act
target_q = result.logits[np.arange(len(a)), a]