Validation while training doesn't seem to load current model's weights

[afm215]

I noticed this issue while training a iresnet. At the end of the training, the accuracy shown on the 5 validation dataset are dramatically close to 0.5. I thus added a print call at line 175 of train_val.py: print(f"{k} : {v}", flush=True). By doing so I realized that the model's performance is close to random at each epoch, during the validation step. But when running the evaluation after the training (i.e. by running main with --evaluate ), the accuracies are much more coherent (i.e. much better).
I do think there is an issue while loading the current model's weights at the validation steps during training.

[afm215]

Ok problem solved. This was a direct consequence of #125. Solved by using srun correctly.

[too-sheep]

Hi, I'm having the same problem and would like to ask exactly how to use srun correctly.

[afm215]

Sure! Have you updated the get_world_size and get_local_rank functions as shown in #125 ?

[too-sheep]

Thanks for the quick reply, the problem solved!