How to get access to DDP functions before initializing trainer? #20001
Unanswered
seermer
asked this question in
DDP / multi-GPU / multi-node
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am currently writing a pipeline that
update settings (e.g., add fileHandler to command line logging) -> initialize config from a user input file -> dynamically set some parameters accordingly (e.g., set rank-specific seed) -> initialize lightning trainer and LightningModule and start training
. It needs to be done in this order (as many steps depend on previous steps), and must be synchronized across GPU after each step (otherwise some operations that only runs on rank 0 with decoratorrank_zero_only
may not be effective and ready for other GPUs yet). How can I achieve this?All the documentations I can find uses
trainer.global_rank
to get global rank number, and usestrainer.strategy.barrier
to synchronize at certain line, and I cannot find examples that perform communications (e.g., gather, broadcast) outside aLightningModule
. What should I use before trainer and LightningModule are initialized?Beta Was this translation helpful? Give feedback.
All reactions