Vader in a Docker Container

[ax3l]

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

OpenMPI 3.0.0 (and 2.1.2 for comparisons)

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

From source (via spack) inside a docker container.

Please describe the system on which you are running

• Operating system/version: Ubuntu "xenial" 16.04.4 LTS with kernel 4.4.0-116 (host), Ubuntu "xenial" 16.04.4 LTS with kernel 4.4.0-116 (container)
• Computer hardware: two NUMA nodes with each an Intel Xeon E5-2698v4 (Nvidia DGX-1)
• Network type: only in-node relevant for now.

---

Details of the problem

Starting MPI with more than one rank will result in errors of the form

Read -1, expected <someNumber>, errno =1
Read -1, expected <someNumber>, errno =1
Read -1, expected <someNumber>, errno =1
Read -1, expected <someNumber>, errno =1
Read -1, expected <someNumber>, errno =1
...

as soon as communication is performed (send, receive, reduce, etc.). Simple programs that only contain MPI startup (Init, Get_Rank, Finalize) and shutdown run without issues.

The only way to work-around this issue was for me to down-grade to OpenMPI 2.X which still supports "sm" as a BTL and deactivating vader, e.g. with export OMPI_MCA_btl="^vader".

Is it possible the detection/test of a working CMA is not fully functional? This issue is likely caused by either a non-existent or not-fully forwarded CMA kernel support inside the docker container. Do you have any recommendations on how to use vader in such an environment as the in-node BTL?

[jsquyres]

What are those error messages from -- are they from your program? I.e., what exactly are those error messages indicating?

FWIW: I do not believe we have any tests -- in configure or otherwise -- to check for non-functional CMA. If we find CMA support, we assume it's working. Vader should work just fine if CMA support is not present -- it will fall back to regular copy-in-copy-out shared memory.

[hppritcha]

Are the processes within different docker containers? If so, then its likely CMA is failing because the containers may be in different name spaces. The workaround is to disable CMA on the mpirun command line:

--mca btl_vader_single_copy_mechanism=none

[ax3l]

Thank you for the details!

What are those error messages from -- are they from your program?

No they originate from OpenMPI. I guess from here:

https://github.com/open-mpi/ompi/blob/v3.0.0/opal/mca/btl/vader/btl_vader_get.c#L74-L78

Are the processes within different docker containers?

No it's the same container on a Nvidia DGX-1 (low detail datasheet, detailed guide) which has two Xeon packages in case that is relevant. Might be, I am not sure if they have regular QPI on it. QPI is there (last link, page 6)

We will try to debug it hands-on again with Nvidia engineers next week. I was just wondering if the error (see code lines above) already tells you something that could give me pointers on how to debug CMA (or if you had runtime CMA tests in place).

I see that you define OPAL_BTL_VADER_HAVE_CMA at compile time, which is of course a bit tricky for an image that is shipped around (might exist on the build machine but not on the run machine).

[ggouaillardet]

@ax3l the issue here is that process_vm_readv() fails with EPERM which is suspicious.

The root cause could be docker prevents this, and some sysadmin config might be required.

From the man page

In order to read from or write to another process, either the caller must have the capability CAP_SYS_PTRACE, or the real user ID, effective user ID, and saved set-user-ID of the remote process must match the real user ID of the caller and the real group ID, effective group ID, and saved set-group-ID of the remote process must match the real group ID of the caller. (The permission required is exactly the same as that required to perform a ptrace(2) PTRACE_ATTACH on the remote process.)

you might want to manually check those conditions are met as well.
there is a theoretical risk your kernel incorrectly supports this in the context of docker (e.g. namespaces).

I would suggest you first try to run your app on the host first, and then in the container.
As pointed earlier

mpirun --mca btl_vader_single_copy_mechanism none ...

might help you

Note the btl/sm is still available in recent Open MPI versions, so

mpirun --mca btl ^vader ...

might also help you here.

[hjelmn]

@ggouaillardet btl/sm is not needed. The way we are recommending to deal with docker if the ptrace permissions can't be fixed is to set OMPI_MCA_btl_vader_single_copy_mechanism=none. That will disable CMA.

[felker]

I recently upgraded the OpenMPI library in our project's Travis CI setup from 2.1.1 to 3.0.2 (and tried 3.1.0), and we were observing

Read -1, expected 5120, errno = 1
Read -1, expected 5120, errno = 1
Read -1, expected 5120, errno = 1
....

as soon as mpirun was launched with 2 or more ranks on the Ubuntu Trusty Docker image build environments.

Setting

export OMPI_MCA_btl_vader_single_copy_mechanism=none

before launching the jobs, as @hjelmn suggested, seems to have fixed the problem in 3.0.2 for us.

[hjelmn]

Note if you want better performance you want CMA to work. It will only work if all the local MPI processes are in the same namespace.

An alternative (I haven't tested this) would be to use xpmem: http://gitlab.com/hjelmn/xpmem (there is a version on github but it will no longer be maintained).

[blechta]

Note that one can allow ptrace permissions by docker run --cap-add=SYS_PTRACE ... which seems to get CMA working.

[JiaweiZhuang]

Setting

export OMPI_MCA_btl_vader_single_copy_mechanism=none

Got exactly the same issue with OpenMPI 3.1.3 in Docker. This fixes the problem!

[milthorpe]

Same problem on OpenMPI 4.0.0 in Docker; disabling vader single copy mechanism as suggested above fixes it.

[AdamSimpson]

FWIW the root cause of this is likely that process_vm_readv()/process_vm_writev() are disabled in the default Docker seccomp profile. A slightly less heavy handed option than --cap-add=SYS_PTRACE would be to modify the seccomp profile so that process_vm_readv and process_vm_writev are whitelisted, by adding them to the syscalls.names list. Having done that I was able to use vader and UCX with CMA without issue.

Edit:
It is also worth mentioning that if you're still running into CMA issues: openucx/ucx#3545

[ax3l]

Saw the problem again today with:

• Docker version 20.10.7, build 20.10.7-0ubuntu1~20.04.1
• OpenMPI 4.0.3
• Ubuntu 20.04 image (x86-64)

Work-around as before is still:

export OMPI_MCA_btl_vader_single_copy_mechanism=none

[gpaulsen]

master PR #6844 was merged into v4.0.x with PR #6997 in time for Open MPI v4.0.2, so perhaps that's not the only issue here, or something changed after that.

[jjhursey]

A few notes on this ticket:

• In main/v5.0.x the btl_vader_single_copy_mechanism MCA parameter does not exist. Use -mca smsc ^cma instead. Also vader has been renamed to sm
• In PR smsc/cma: Add a check for CAP_SYS_PTRACE between processes #10694 I added an additional check that should disqualify cma if CAP_SYS_PTRACE is not granted to the container. See example output in the PR.

I think this ticket can be closed given the workaround for the v4.x series. The change in PR #10694 should make it so that the workaround is not required and the cma will disqualify itself automatically.

As noted above cma does have some nice performance benefits so it is recommended that you pass --cap-add SYS_PTRACE to your favorite container runtime to make sure that CAP_SYS_PTRACE is granted to processes in the resulting namespace.

[vedantroy]

I'm running into this issue on https://modal.com, and it is causing cudaIpcOpenMemHandle to fail. Using --mca btl ^vader doesn't solve the problem, and I can't modify the container environment, is there any other way to make this work?

[rhc54]

What version of OMPI are you using? It's hard to help if you don't provide at least some basic info.

[vedantroy]

mpirun --version prints mpirun (Open MPI) 4.1.2.

The command I'm running is:

mpirun --hostfile hostfile.txt --mca btl ^vader --allow-run-as-root -np 2 csrc/reference_allreduce/fastallreduce_test.bin

where the hostfile is: localhost slots=2 max_slots=2

[rhc54]

@hppritcha @ggouaillardet You folks have any thoughts here? I don't know anything about v4.1, I'm afraid.

[ggouaillardet]

This looks like a gpu within docker issue.

@vedantroy please open a new issue and give a full description of your problem.

[bosilca]

vader does not use cudaIpcOpenMemHandle, the culprit might be smcuda. Try --mca btl ^smcuda.

[hppritcha]

I second @ggouaillardet on opening a separate issue if @bosilca 's suggestion doesn't work.