bug(logger): wandb fails on sweep

[AmitMY]

🐛 Bug

When using wandb sweeps for hyperparameters search, I get this error:

wandb: ERROR Attempted to change value of key "dropout_std" from 0.030424838979365657 to 0.030424838979365654

The reason is I ran:

wandb_logger.log_hyperparams(params)

Which I guess has some problem with floating-point numbers in high accuracy?

[amoudgl]

+1, I faced the same issue when using pytorch lightning with wandb sweeps. To summarize, wandb automatically logs hyperparams when we run the wandb sweep agent on a machine. Later, pytorch lightning again tries to log same hyperparams but due to precision error between lightning and wandb already logged hyperparams, wandb throws this error. Just a guess: wandb sweep agent might be using double format to generate new hyperparams and when lightning receives those args from command line, it converts them to float and tries to log it. I haven't digged in detail where these hyperparams get altered, it could be on wandb side or lightning side.

I reported this issue to wandb and got the following response:

It is preferred to either pass your config parameters all at once to: wandb.init(config=config_dict_that_could_have_params_set_by_sweep)
or:
experiment = wandb.init()
experiment.config.setdefaults(config_dict_that_could_have_params_set_by_sweep)

The advantage of doing this is that it will ignore setting any key that has already been set by the sweep.

We will look into the torch lightning integration and see if we can make this safer for those using sweeps.

[amoudgl]

cc: @neggert, @Borda, @williamFalcon

[Borda]

@borisdayma @calclavia pls ^^

[borisdayma]

The problem is that it tries to log this value twice and is probably called before automatically by pytorch-lightning.
The callback will automatically log every parameter which is in pl.LightningModule.params (where you probably already have the dropout).

See an example of using pytorch-lightning with wandb (including sweeps) here: https://github.com/borisdayma/lightning-kitti

I'll be adding it to the pytorch-lightning repo later but still need to push a PR related to the watch method for it to work properly.

[Borda]

@borisdayma is it fixed now?

[borisdayma]

I added a fix.
@AmitMY @amoudgl Feel free to test it with your sweeps and let me know if there's still an error.