Skip to content

Seed all workers when using DDP #7942

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tchaton merged 8 commits into from
Jun 14, 2021
Merged

Seed all workers when using DDP #7942

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
3c1fb2b
Seed all workers when using DDP
Jun 11, 2021
e7a9546
Fix to dataloader seeding
Jun 11, 2021
52b94b3
Make argument name explicit
Jun 13, 2021
2ffe83a
Use f-strings when logging
Jun 13, 2021
dcd2536
Merge branch 'bugfix/seed-ddp-workers' of github.com:groke-technologi…
Jun 13, 2021
737e2e9
Merge branch 'master' into bugfix/seed-ddp-workers
Jun 13, 2021
e3fc299
Removed a redundant log message
Jun 14, 2021
ee0a16b
Merge branch 'master' into bugfix/seed-ddp-workers
Jun 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).

### Fixed

- Fixed setting `worker_init_fn` to seed dataloaders correctly when using DDP ([#7942](https://github.com/PyTorchLightning/pytorch-lightning/pull/7942))


- Fixed `DataModule.prepare_data` could only be called on the global rank 0 process ([#7945](https://github.com/PyTorchLightning/pytorch-lightning/pull/7945))


Expand Down
6 changes: 5 additions & 1 deletion pytorch_lightning/utilities/seed.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,9 @@ def reset_seed() -> None:
If :func:`pytorch_lightning.utilities.seed.seed_everything` is unused, this function will do nothing.
"""
seed = os.environ.get("PL_GLOBAL_SEED", None)
workers = os.environ.get("PL_SEED_WORKERS", False)
if seed is not None:
seed_everything(int(seed))
seed_everything(int(seed), workers=bool(workers))


def pl_worker_init_function(worker_id: int, rank: Optional = None) -> None: # pragma: no cover
Expand All @@ -100,6 +101,9 @@ def pl_worker_init_function(worker_id: int, rank: Optional = None) -> None: # p
process_seed = torch.initial_seed()
# back out the base seed so we can use all the bits
base_seed = process_seed - worker_id
log.debug(
f'Initializing random number generators of process {global_rank} worker {worker_id} with base seed {base_seed}'
)
ss = np.random.SeedSequence([base_seed, worker_id, global_rank])
# use 128 bits (4 x 32-bit words)
np.random.seed(ss.generate_state(4))
Expand Down