training

[trungpham2606]

Dear @flyinglynx
When training on my custom dataset, after 2 epochs, I tested the trained model (my dataset is quite big with 50,000 images), the results are pretty good but after 4 epochs, the model outputs nothing.
Do you have any explanation about this situation ?

[flyinglynx]

Hi, I encountered similar situations where model outputs all zeros in the density maps. Some tricks and experience might be helpful: 1) try some other random seeds (weird but sometimes useful); 2) try to initialize model parameters with our pretrained model on FSC-147; 3) change batch size or learning rate. Training density map-based models can sometimes be trapped in local optima.

According to my experience, the training usually get back to track after some epochs in such occasion. You can also wait for maybe 10 epochs to see the results.

[trungpham2606]

Thank you @flyinglynx for your suggestion.
I will retrain it with your FSC-147 pretrained ^^ and follow the training for a longer time ^^. I was too rush.
I did train famnet and the problem even occurs sooner (after some iterations.)
In addition, your pretrained weight is 50Mb only, but my trained weight is 3 times bigger (~150mb) :-?

[flyinglynx]

Thank you @flyinglynx for your suggestion. I will retrain it with your FSC-147 pretrained ^^ and follow the training for a longer time ^^. I was too rush. I did train famnet and the problem even occurs sooner (after some iterations.)

You can also check the resolutions of images. Too high or too low may both be harmful. You can follow the data pro-processing of FSC-147 dataset to constrain the resolutions.

I only provide the model parameters in the checkpoint file for convenience. However, if you run our code, the checkpoint file will store the model parameters, optimizer parameters, etc, which is more larger.

[trungpham2606]

Yeah @flyinglynx , I just checked the code, during training it saved some other information beside the parameters.
Because you use resnet as the backbone, so I think it will work best with small resolution, Iam resizing the 1920x1200 to 384x384 for training.

[trungpham2606]

Hi @flyinglynx it's me again ^^.
I have a question that the examplar size is pretty big (128x128), you found it by experiments or does it have any insight ideas ?

[flyinglynx]

Hi, we choose the exemplar size following UP-DETR. We use 128 * 128 and the results turned out to be satisfying. So we did not further finetune this parameter. I tested a few smaller sizes these days (64 * 64 and 96 * 96), and the performance drops slightly. But if using a simple multi-scale strategy by resizing exemplar into different sizes (64, 96, 128), and matching the exemplars of different sizes with the query image, the performance will be slightly better.