Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

Trainstats Clobber Bug #4386

Merged
merged 1 commit into from
Mar 3, 2022
Merged

Trainstats Clobber Bug #4386

merged 1 commit into from
Mar 3, 2022

Conversation

spencerp
Copy link
Contributor

@spencerp spencerp commented Mar 1, 2022

Patch description
TrainLoop._save_train_stats is currently called in two places: TrainLoop.train and TrainLoop.save_model. TrainLoop.save_model returns early if it's not the primary worker, but TrainLoop.train does not. To make sure the .trainstats file isn't written to from multiple workers at once, this PR adds a return-early check directly in TrainLoop._save_train_stats.

Testing steps
Circle CI

@spencerp spencerp merged commit bb9513a into main Mar 3, 2022
@spencerp spencerp deleted the trainstats-clobber branch March 3, 2022 16:44
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants