About the training speed on SemanticKITTI

[zzh-ecnu]

Hello, i implement your code on SemanticKITTI dataset, and keep the default config setting.

However, i have cost 3 days to train the network, and the current epoch is 30. The max_epoch is 100. Following this setting, i will cost 10 days to train the whole pipeline.

The GPU i used is Tesla-V100 16G.

The process is too slow to train, so any suggestions?

Thanks sincerely~

[edwardzhou130]

Sorry for the late reply. Here are some tips to speed up the training:

• In my experiment, the model needs only around 20/20(before and after SAP starts) epochs to reach the highest validation PQ. So probably you can use a smaller epoch number, like max 50 epoch and starting SAP at the 20th epoch).
• Another suggestion is to decrease the grid size. Grid size [320,240,32] has 1.5% PQ loss compared to the default grid size [480,360,32] while the training is much faster.
• The validation in training takes a lot of time. You can increase the interval of the validation at the beginning of the training to save some time.

[zzh-ecnu]

Thanks for your carefully reply~

Recently, i try to introduce pytorch multi-gpu support into this base, however i meet a problem about the dimension inconsistent.

According to the error infomation, it looks like the input visibilty feature is inconsistent with the output point feature of the network.

Currently i have no idea about this error, could you have any suggestions, thanks sincerely~

[Carl12138aka]

Sorry for the late reply. Here are some tips to speed up the training:

• In my experiment, the model needs only around 20/20(before and after SAP starts) epochs to reach the highest validation PQ. So probably you can use a smaller epoch number, like max 50 epoch and starting SAP at the 20th epoch).
• Another suggestion is to decrease the grid size. Grid size [320,240,32] has 1.5% PQ loss compared to the default grid size [480,360,32] while the training is much faster.
• The validation in training takes a lot of time. You can increase the interval of the validation at the beginning of the training to save some time.

Could you please tell me how to increase the interval of the validation，thanks a lot！