How to do a simple multirun?

[Yevgnen]

The hyperparameter-search section does help when doing hyperparameter-search. But if I want a simple multirun like python train.py -m trainer.gpus=4 +trainer.strategy=ddp trainer.max_epochs=1 seed=1,2,3, I got 4 folders

└── multiruns
    └── default
        ├── 2022-02-22_10-48-25
        │   ├── 0
        │   │   ├── checkpoints
        │   │   │   ├── epoch_000.ckpt
        │   │   │   └── last.ckpt
        │   │   ├── config_tree.log
        │   │   └── train.log
        │   ├── 1
        │   │   ├── checkpoints
        │   │   │   ├── epoch_000.ckpt
        │   │   │   └── last.ckpt
        │   │   ├── config_tree.log
        │   │   └── train.log
        │   ├── 2
        │   │   ├── checkpoints
        │   │   │   ├── epoch_000.ckpt
        │   │   │   └── last.ckpt
        │   │   ├── config_tree.log
        │   │   └── train.log
        │   └── multirun.yaml
        ├── 2022-02-22_10-48-28
        │   ├── 0
        │   │   └── train_ddp_process_1.log
        │   ├── 1
        │   │   └── train_ddp_process_1.log
        │   ├── 2
        │   │   └── train_ddp_process_1.log
        │   └── multirun.yaml
        ├── 2022-02-22_10-48-31
        │   ├── 0
        │   │   └── train_ddp_process_2.log
        │   ├── 1
        │   │   └── train_ddp_process_2.log
        │   ├── 2
        │   │   └── train_ddp_process_2.log
        │   └── multirun.yaml
        └── 2022-02-22_10-48-35
            ├── 0
            │   └── train_ddp_process_3.log
            ├── 1
            │   └── train_ddp_process_3.log
            ├── 2
            │   └── train_ddp_process_3.log
            └── multirun.yaml

21 directories, 25 files

The first one seems to be the right multirun folder, right? Is this the right way to do a simple multirun? Thanks.

[ashleve]

As you can see from many issues, multirun + DDP currently doesn't work correctly. This due to the way hydra creates a new work dir for every run, which is not compatible with DDP which runs new process for each GPU.

A while ago I made an issue about in lightning repo, read it if you want to learn more: Lightning-AI/pytorch-lightning#11300

I don't see anything we can do about it for now. It's being worked on, so the future release might fix it.

[ashleve]

The multirun works correctly in 1 GPU mode though.

[Yevgnen]

Thanks!