[Fix] Fix issue #305

[yezhen17]

The bug mentioned in issue #305 is caused by a discordance between the current cuda device and the device of the input tensors. This may potentially cause unexpected behaviours in other cuda ops.

[codecov]

Codecov Report

Merging #311 (df57552) into master (ac9a3e8) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #311   +/-   ##
=======================================
  Coverage   50.23%   50.23%           
=======================================
  Files         173      173           
  Lines       11585    11585           
  Branches     1803     1803           
=======================================
  Hits         5820     5820           
  Misses       5404     5404           
  Partials      361      361           

Flag	Coverage Δ
unittests	50.23% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmdet3d/models/fusion_layers/point_fusion.py	37.62% <ø> (ø)
mmdet3d/models/dense_heads/centerpoint_head.py	47.01% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5592a50...df57552. Read the comment docs.

[ZwwWayne]

Can we use the pytorch API https://pytorch.org/docs/stable/cuda.html#torch.cuda.set_device rather than modifying the CUDA code?