Fix checkpoint warning for floats

[ZhaofengWu]

#1819 added this warning when saving checkpoints. https://github.com/PyTorchLightning/pytorch-lightning/blob/bd49b07fbba09b1e7d8851ee5a1ffce3d5925e9e/pytorch_lightning/callbacks/model_checkpoint.py#L259-L263

1. Is there a reason why native Python floats aren't supported? I think it's actually quite common especially when libraries e.g. scipy are involved when computing the metrics.
2. It says checkpoint not saved, but actually checkpoints are still saved. This check doesn't actually change the saving logic besides giving that warning.

[Borda]

@justusschock @SkafteNicki mind have a look :]

[SkafteNicki]

The reason it is still working is due to the fact that it is being converted to a torch.tensor later on:
https://github.com/PyTorchLightning/pytorch-lightning/blob/bd49b07fbba09b1e7d8851ee5a1ffce3d5925e9e/pytorch_lightning/callbacks/model_checkpoint.py#L176-L181
so in that sense they are supported. I guess the warning is there due to:

• If you are returning something other than a native float, like a dict, you will get a non-informative error (RuntimeError: Could not infer dtype of dict). The warning could hopefully help you figure out what is going on. So the only way we can guarantee that this will work, is if you return a torch.Tensor.
• The PR that implemented this, tried to fix some ddp bugs. I cannot completely figure out how this is related to ddp right now, but I guess that again torch.Tensor is the only type that guarantee that this works in ddp mode.

[ZhaofengWu]

If types such as floats are supported, should they be excluded from the check and don't show warnings? And IMHO the warning message saying checkpoints won't be saved while they actually are is not desirable.

[Borda]

sounds reasonable, mind draft a PR? 🐰

[edenlightning]

@ZhaofengWu Checking in :)

[ZhaofengWu]

Ah sorry I overlooked this. I might not have the time recently to do this. Sorry :(