The sigloss does not go down for custom dataset

[prateekstark]

Hi there, I tried using the DepthFormer model for my custom dataset but surprisingly the sigloss in the validation set does not reduce. I referred to #20 but I noticed this is not a random phenomenon, it happens everytime. I tried restarting many times. Could you point out some possible solution? I also tried the warming up of SIGLOSS, doesn't help. For my case, I chose:
depth_scaling: 1000 for 16 bit depth images.
Min, Max depth: 1e-3 and 10.
Image Size: (512, 384)
Configs I tried for: DepthFormer, BinsFormer

Rest all Parameters are same as respective NYU configs

[zhyever]

That sucks.

Please first ensure that the value of your depth maps lies in 0-10 after dividing it by the depth_scale=1000.

Then, try to set scale_up=True presented in issue#20 to modify the depth head into a sigmoid version.

Actually, in the experiments of my new paper LiteDepth, I still observe this convergence issue. It is so hard to find the reason and I also observe this issue in other repos like TransDepth and Adabins. Hence, it may be related to general experimental settings like crop size, color aug, sigloss itself, and so on. I also look for help from the community to fix this bug.

Anyway, a more direct way to solve this issue: loading our pre-trained model. You may see some warning about misalignment of param_dict, but that's ok and will not break down the training since I set the strict=False of load_checkpoint. Since the model is pre-trained on the depth task, it can avoid breaking down in the early stage of training. Hope it is useful for you.

Sometimes, I also think it is not a random phenomenon. But after repeating re-starting several times and setting different random seeds, it works... Gook luck! :D

[zhyever]

Close it for now.