Training baseline model doesn't converge

[shauloron]

Hi,
I'm trying to train the model with the provided config (R50 256x704) with code pulled on July 10, 2024.
I'm using 4 A100 GPU's with total batch size 48.
With the original LR 6e-4 the training diverges and grad_norm goes NaN after ~20 epochs.
When I lower the LR to 4e-4 the loss goes down and grad_norm is ok but the final model has AP=0 for all classes after 100 epochs.
I read through all the open and closed issue. Checked that the resnet50 pretrain is loading successfully.
using the default aug_config:
data_aug_conf = {
"resize_lim": (0.40, 0.47),
"final_dim": input_shape[::-1],
"bot_pct_lim": (0.0, 0.0),
"rot_lim": (-5.4, 5.4),
"H": 900,
"W": 1600,
"rand_flip": True,
"rot3d_range": [-0.3925, 0.3925],
}

Any ideas why my training doesn't converge?

[linxuewu]

Can you obtain correct results (metrics and visualizations) by running inference with the released checkpoint?

[shauloron]

I can't seem to find any pretrained model, only pretrained resnet50 weights.
Can you please point me to the pretrained weights you released?
Thanks

[shauloron]

ok found it now: ckpt
I'll check how it works out.

[shauloron]

I've tried using the checkpoint above and get different results than the ones in the tutorial notebook visualizations
To start the anchor map doesn't look the same:

and the detection results are partial:

any ideas?

[shauloron]

Hi @linxuewu I would really appreciate any idea you may have on this one?