Too high GRAM consumption for spconv indexing

[zhanggefan]

I am trying to reimplement the PVRCNN using this codebase and I found that the GRAM consumption during training on kitti is unacceptably high compare to OpenPCDet. There is the a surge in GRAM consumption for approximately 4GB for the first time when get_indice_pairs is called.

After some investigation I found that the spconv used by MMDet3D is a customized version, and I also found that the following 4096 seems to be the root cause of it. Changing them to 27 would solve the problem, but in an ugly way (all conv kernels larger than 3x3x3 would fail then).

mmdetection3d/mmdet3d/ops/spconv/src/indice_cuda.cu

Line 48 in e5a87f3

prepareDeConvIndicePairsKernel<Index, IndexGrid, NDim, 4096>

mmdetection3d/mmdet3d/ops/spconv/src/indice_cuda.cu

Line 54 in e5a87f3

prepareIndicePairsKernel<Index, IndexGrid, NDim, 4096>

mmdetection3d/mmdet3d/ops/spconv/src/indice_cuda.cu

Line 119 in e5a87f3

getSubMIndicePairsKernel<Index, IndexGrid, NDim, 4096>

[ZwwWayne]

The spconv in MMDet3D is adopted from an early version (around 1.0) of spconv. The settings of kernel are not changed.

Do we have a more flexible way to handle this case? Like CUDA operators in MMCV, these numbers are usually not hardcoded. If we have a flexible way to handle these number, we can create a PR to fix that.

BTW, we are migrating CUDA operators to MMCV, feel free to have a look or review it if you are interested.

[ZwwWayne]

The progress can be find here #994