DepthFormer results on NYU dataset

[mshaheryarmalik]

Hello,

I trained DepthFormer model on NYU dataset after processing the raw data from other work and made few modifications to code such as:

• Added GradientCumulativeOptimizerHook in optimizer_config (DepthFormer config) to handle larger batch size I tired with 16)
• Batch size samples_per_gpu=4, workers_per_gpu=4 here
• Added this to checkpoint_config create_symlink=False here

However, when I try to predict depth with newly trained model, it just gives strange results that if you can explain. I have attached training logs for your reference and one prediction example from newly trained model below.

P.S: I was able to get good results with BTS with same code modifications.
20220708_175744.log

[zhyever]

This is related to the unsolved issue of convergence. Some discussions can be seen in Issue #20, #32. As you can see from the log files, the sigloss is not decreasing and retain the value of about 3.

While I have tried several methods to solve this issue, they are futile and the problem exists. Now, the simplest way is to run the training script several times. Whenever you see that the loss is less than 3 (can decrease to 0.x) in the first several printed log info, it means that the model can be successfully trained in the following training stage.

Refer to Issue #20, #32 for more details.

[zhyever]

Close for now.