AssertionError: MPI world size 16 does not match torch world size 8

[szhengac]

When I use mpirun for Megatron-LM bert pretraining, I received
AssertionError: MPI world size 16 does not match torch world size 8"

For mpirun args, I use --allow-run-as-root -np 16 -N 8 --hostfile ${hostfile}. Is there something I miss for setting torch world size appropriately?

[tjruwase]

@szhengac, can you share the number of nodes, gpu per node, and hostfile?

[szhengac]

There are 2 nodes with 8 gpus per node. Hostfile is: ip1 slots=8 ip2 slots=8

…

On Sun, 25 Oct 2020 at 22:20, Olatunji Ruwase ***@***.***> wrote: @szhengac <https://github.com/szhengac>, can you share the number of nodes, gpu per node, and hostfile? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#461 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6GZVFKZZY57SPC7SOCBVTSMUBLLANCNFSM4SDPRLVA> .

[szhengac]

@tjruwase I wonder if there is any update on this issue?

[jeffra]

Hi @szhengac, I have an idea what might be going on however want a bit more info. There should be another print out before this assertion such as "Discovered MPI settings of world_rank=". This comes from our mpi discovery code here: https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/engine.py#L280-L291

Also how many prints like this do you see? Are you seeing 16 of them or only 8?

[jeffra]

@szhengac: I see the issue now. It's with our Megatron example, it does not support running with MPI because it's initializing torch distributed itself and not with deepspeed, but this is required since it uses model parallelism. Let me update this and I'll update this thread once it's fixed.

[szhengac]

Thanks for the quick fix! I wonder if I use deepspeed launcher, is there any easy way to enable numactl such as https://github.com/mlperf/training_results_v0.6/blob/master/NVIDIA/benchmarks/minigo/implementations/tensorflow/minigo/ompi_bind_DGX1.sh? It is convenient to attach this script with mpirun.

[szhengac]

@jeffra I just had a try with the pr. But I got one new error when I use mpirun on a single node:

Traceback (most recent call last):  
  File "deepspeed_train.py", line 539, in <module>
    main()
  File "deepspeed_train.py", line 529, in main
    model, optimizer = prepare_model_optimizer(args)
  File "deepspeed_train.py", line 442, in prepare_model_optimizer
    model_parameters=optimizer_grouped_parameters)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/deepspeed/__init__.py", line 119, in initialize
    config_params=config_params)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 146, in __init__
    self._configure_distributed_model(model)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 485, in _configure_distributed_model
    self._broadcast_model()
  File "/home/ec2-user/.local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 463, in _broadcast_model
    group=self.data_parallel_group) 
  File "/usr/local/lib64/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 963, in broadcast
    work = group.broadcast([tensor], opts)
RuntimeError: NCCL error in: /tmp/pip-xturxqjq-build/torch/lib/c10d/ProcessGroupNCCL.cpp:896, invalid usage, NCCL version 2.7.8

I also found that GPU 0 spawns 8 processes:

|    0   N/A  N/A     36692      C   python3                          1227MiB |
|    0   N/A  N/A     36693      C   python3                           333MiB |
|    0   N/A  N/A     36694      C   python3                           335MiB |
|    0   N/A  N/A     36695      C   python3                           333MiB |
|    0   N/A  N/A     36696      C   python3                           343MiB |
|    0   N/A  N/A     36697      C   python3                           337MiB |
|    0   N/A  N/A     36698      C   python3                           333MiB |
|    0   N/A  N/A     36699      C   python3                           333MiB |
|    1   N/A  N/A     36693      C   python3                          1065MiB |
|    2   N/A  N/A     36694      C   python3                          1065MiB |
|    3   N/A  N/A     36695      C   python3                          1045MiB |
|    4   N/A  N/A     36696      C   python3                          1045MiB |
|    5   N/A  N/A     36697      C   python3                          1085MiB |
|    6   N/A  N/A     36698      C   python3                          1085MiB |
|    7   N/A  N/A     36699      C   python3                          1085MiB 

I don't have these issues when I use the deepspeed launcher.

[szhengac]

I just found it is because it links to the wrong nccl lib. For the second process spawning issue, I have submitted a pr to fix it.