How to reinit wanbd in a for loop with PL Trainer #8572
Answered
by
morganmcg1
Gladiator07
asked this question in
Lightning Trainer API: Trainer, LightningModule, LightningDataModule
-
I am training 5-fold CV with PyTorch Lightning in a for loop. I am also logging all the results to wandb. I want wanbd to reinitalize the run after each fold, but it seems to continue with the same run and it logs all the results to the same run. I also tried passing kwargs in the WandbLogger as mentioned in the docs here, with no luck. def run(fold):
kwargs = {
"reinit": True,
"group": f"{CFG['exp_name']}"
}
wandb_logger = WandbLogger(project='<name>',
entity='<entity>',
config = CFG,
name=f"fold_{fold}",
**kwargs
)
trainer = Trainer(
precision=16,
gpus=1,
fast_dev_run=False,
callbacks = [checkpoint_callback],
logger=wandb_logger,
progress_bar_refresh_rate=1,
max_epochs=2,
log_every_n_steps=1
)
trainer.fit(
lit_model,
data_module
)
if __name__ == "__main__":
for fold in range(5):
run(fold) |
Beta Was this translation helpful? Give feedback.
Answered by
morganmcg1
Nov 11, 2021
Replies: 1 comment
-
@Gladiator07 you could try call |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
Gladiator07
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@Gladiator07 you could try call
wandb.finish()
at the end of every run. This should close the wandb process. A new one will be started when you call the next run