Fix checkpoint pickle during ddp (on master only)

[williamFalcon]

@neggert looks like the new hparams saving fails with ddp/ddp2

File "/private/home/falc/.conda/envs/ddt2/lib/python3.7/site-packages/pytorch_lightning/callbacks/pt_callbacks.py", line 245, in on_epoch_end
self.save_model(filepath, overwrite=True)
File "/private/home/falc/.conda/envs/ddt2/lib/python3.7/site-packages/pytorch_lightning/callbacks/pt_callbacks.py", line 224, in save_model
self.save_function(filepath)
File "/private/home/falc/.conda/envs/ddt2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer_io.py", line 127, in save_checkpoint
torch.save(checkpoint, filepath)
File "/private/home/falc/.local/lib/python3.7/site-packages/torch/serialization.py", line 224, in save
return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
File "/private/home/falc/.local/lib/python3.7/site-packages/torch/serialization.py", line 149, in _with_file_like
return body(f)
File "/private/home/falc/.local/lib/python3.7/site-packages/torch/serialization.py", line 224, in
return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
File "/private/home/falc/.local/lib/python3.7/site-packages/torch/serialization.py", line 296, in _save
pickler.dump(obj)
AttributeError: Can't pickle local object 'ArgumentParser.init..identity'

[neggert]

Will take a look.

[neggert]

I'm having trouble reproducing this. I'm running

python gpu_template.py  --gpus 2 --distributed_backend ddp

on a single node with 2 GPUs (I've locally reverted your patch that catches the error). That runs fine, and I'm able to see that hparams are saved to the checkpoint by loading it manually.

I even tried replacing the ArgumentParser with a HyperOptArgumentParser, but everything still works. Can you share any more details about what you were doing to get this error?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Borda]

@neggert does it mean that the problem is gone? 🤖