Fine-tuning details on iNaturalist dataset

[Reinhard-Tichy]

In #45 , I have found some basic details about finetuning for CIFAR10 and Cars, but I could not find anything about other datasets.

So, could you please also provide the details about hyperparameters used in iNat2018 & 2019 for finetuning?

Thank you very much!

[TouvronHugo]

Hi @Reinhard-Tichy ,
Thanks for your question,
The hparams for iNaturalist with the base model are:

• Batch size: 768
• LR: 7.5e-5
• epochs: 360
• Weight decay 0.05
• Scheduler: cosine
• Repeated augmentation: True
• Smoothing: 0.1
• Warmup: 5 epochs
• RandAugment: rand-m9-mstd0.5-inc1
• Mixup: 0.8
• CutMix: 1.0
• Stochastic Depth:0.1
• Erasing: 0.1
• Optimizer: AdamW
  Don't hesitate if you have any other questions,
  Best,
  Hugo

[Reinhard-Tichy]

Thanks for your reply! Could you further explain what the param Stochastic Depth means here? Cause I seemly cannot find this arg in the main.py

And does the finetuning process really need a 360 epochs? This is even more than the one of the pretraining step [Lol]

[TouvronHugo]

Hi @Reinhard-Tichy ,
For Stochastic Depth this is the --drop-path argument.
Yes, this allows for better results but it is possible to fine-tune with much less epochs but the results will be slightly worse.
Best,
Hugo

[TouvronHugo]

FYI, I added the SGD config for iNat finetuning here