-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
5-Fold with PyTorchLightning + Wandb seems to log to the same experiment #8614
Comments
Dear @Gladiator07, I converted this to an issue as it should be created several runs. Best, |
@Gladiator07 I think I have a workaround for you. Put this import wandb
wandb.finish() before instantiating WandbLogger. This will make sure that the experiment from the previous "fold" gets finished. For context, our WandbLogger is simply wrapping the wandb.Run object which is sort of a global variable in wandb according to my understanding. I will try to make this into a real fix for our WandbLogger. Any feedback appreciated. Maybe @borisdayma has another idea :) |
Actually, just found this PR which enforced the current behavior and also defines tests for this. #4648 |
Yes this is perfectly correct @awaelchli |
@borisdayma @awaelchli could we say that one run corresponds to one logger object? So when training multiple stages you just reuse the logger object and if recreating that, you just get a new run? |
So the main issue is that users could have a run already created before, even without the logger: using sweeps or for example if they use an artifact (like a previous checkpoint logged). What do you think about adding a warning when a run is existing saying that we will be using the same one and that they can manually call |
@borisdayma I like that. Perhaps my PR #8617 adding the finish() method should be closed as it is confusing to have finish() alongside finalize() and close() methods. |
Yes I think I would just close it. |
See the new PR attached. Let me know what you think of this |
I like it! |
I am training 5-fold CV with PyTorch Lightning in a for loop. I am also logging all the results to wandb. I want wanbd to reinitalize the run after each fold, but it seems to continue with the same run and it logs all the results to the same run. I also tried passing kwargs in the WandbLogger as mentioned in the docs here, with no luck.
Here's a pseudo code of it:
Originally posted by @Gladiator07 in #8572
The text was updated successfully, but these errors were encountered: