trainer.test not working in ddp

[asrafulashiq]

🐛 Bug

Testing in ddp is not working in the latest master.

To Reproduce

I am using the gpu_template example from basic_examples in the repo : "python gpu_template.py --gpus 2 --distributed_backend ddp", where, instead of trainer.fit(model), I am using trainer.test(model).

I am getting "RuntimeError: connect() timed out".

Environment

• PyTorch Version 1.3.1:
• Ubuntu 18.04

[github-actions]

Hi! thanks for your contribution!, great first issue!

[jmarsil]

I'm facing a similar issue to @asrafulashiq . I trained a model using ddp across 4gpus on our institution's slurm cluster. When running trainer.test(model) it seems like the script freezes. Help on this would be greatly appreciated!! Here is a screenshot of the output of my slurm script for reference.

[rakhimovv]

the same issue

[awaelchli]

can confirm, it freezes, but only in ddp. For everyone who faces this issue, use ddp_spawn until this is fixed.

[awaelchli]

I've identified that the process hangs at this line:
https://github.com/PyTorchLightning/pytorch-lightning/blob/f9ccb0fd9b53973a87b18cf9126ad9454b328575/pytorch_lightning/core/lightning.py#L958

[awaelchli]

Ok, more progress. The issue is caused because the master port is different on each rank. If we set the port manually, problem is solved.