Keep same random seed when restarting a run

[anaprietonem]

What happened?

Current AnemoiTrainer defines the initial seed according to def initial_seed(self) -> int:. When resuming a run this seed is not reloaded as part of the mlflow hyperparameters and hence a new random seed is generated. For consistency and reproducibility of the results, we would need to keep the same seed.

What are the steps to reproduce the bug?

Train a model in Leonardo and resume that after 24 hrs - see that the resumed job has a different seed.

Version

Current

Platform (OS and architecture)

ATOS

Relevant log output

No response

Accompanying data

No response

Organisation

No response