Sync working directory with all processes when using Hydra (#5629)

* Append current work dir (hydra dir) to ensure all processes reference the same directory

* Add changelog

* Add different job name for DDP child processes to log to

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

CHANGELOG.md

@@ -29,6 +29,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Fixed `toggle_optimizer` to reset `requieres_grad` state ([#5574](https://github.com/PyTorchLightning/pytorch-lightning/pull/5574))
 
 
+- Fixed FileNotFoundError for best checkpoint when using DDP with Hydra ([#5629](https://github.com/PyTorchLightning/pytorch-lightning/pull/5629))
+
+
 - Fixed an error when logging a progress bar metric with a reserved name ([#5620](https://github.com/PyTorchLightning/pytorch-lightning/pull/5620))
 
 

pytorch_lightning/accelerators/ddp_accelerator.py

@@ -134,11 +134,16 @@ def _call_children_scripts(self):
  del env_copy['PL_GLOBAL_SEED']
 
  # start process
- # if hydra is available and initialized, make sure to set the cwd correctly
+ # if hydra is available and initialized, make sure to set the original cwd correctly
+ # and pass current cwd for ddp processes (which hydra has overridden)
  cwd: Optional[str] = None
  if HYDRA_AVAILABLE:
  if HydraConfig.initialized():
  cwd = get_original_cwd()
+ command += [
+ f'hydra.run.dir={os.getcwd()}',
+ f'hydra.job.name=train_ddp_process_{local_rank}'
+ ]
  proc = subprocess.Popen(command, env=env_copy, cwd=cwd)
  self.interactive_ddp_procs.append(proc)