Lack of memory on the GPU during training

[dwz138831]

Hi, I'm reproducing with the latest version of the code on 8 A30 GPUs (24GB of memory each) and it shows CUDA out of memory at the beginning of the 1st epoch, what is the cause of this.

[LinShan-Bin]

To reproduce the full results, you need 80G memory. The baseline method (no semantics, contracted_coord = False and auxiliary_frame = False) can be reproduced with 24G memory.

[ZiyangYan]

To reproduce the full results, you need 80G memory. The baseline method (no semantics, contracted_coord = False and auxiliary_frame = False) can be reproduced with 24G memory.

Hi, I change the config as you mentioned, and run it in 4 24G A30 GPU, but still has OOM problem

[LinShan-Bin]

To reproduce the full results, you need 80G memory. The baseline method (no semantics, contracted_coord = False and auxiliary_frame = False) can be reproduced with 24G memory.

Hi, I change the config as you mentioned, and run it in 4 24G A30 GPU, but still has OOM problem

For contracted_coord = False, we recommend you use a voxel size of [16, 200, 200]. This keeps the resolution of the inner part and removes the contracted outer part. Though we use [24, 300, 300] in our experiments for fair comparisons, to save memory you have to cut down the voxel size.