Segfault with Slurm (14.03.8), 2+ tasks, and OpenSees

[kcgthb]

Hi,

One of our users reported a segfault that appears under specific circumstances.

When using Slurm (14.03.8) and srun to launch an OpenSees job (http://opensees.berkeley.edu/index.php, compiled and executed with OpenMPI 1.8.2) with more than 2 tasks, the execution fails with a segmentation fault that seems to occur in the vader BTL. I have no idea why it's being used, we're running a x86 Linux cluster on Redhat 6.5.

Reproducer is a follows:

$ srun -n 2 --pty bash
cn01:~$ srun OpenSeesMP
srun: error: cn01: task 0: Segmentation fault

The stack is:

cn01:~$ srun gdb OpenSeesMP
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
[...]
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff1c2e8ea in mca_btl_vader_sendi () from /share/sw/free/openmpi/1.8.2/gcc/4.4/lib/openmpi/mca_btl_vader.so
[...]
bt
(gdb) (gdb) #0  0x00007ffff1c2e8ea in mca_btl_vader_sendi () from /share/sw/free/openmpi/1.8.2/gcc/4.4/lib/openmpi/mca_btl_vader.so
#1  0x00007ffff15f604f in mca_pml_ob1_send_inline () from /share/sw/free/openmpi/1.8.2/gcc/4.4/lib/openmpi/mca_pml_ob1.so
#2  0x00007ffff15f6fe1 in mca_pml_ob1_send () from /share/sw/free/openmpi/1.8.2/gcc/4.4/lib/openmpi/mca_pml_ob1.so
#3  0x00007ffff7b7868a in PMPI_Send () from /share/sw/free/openmpi/1.8.2/gcc/4.4/lib/libmpi.so.1
#4  0x00000000005a1315 in MPI_Channel::sendID(int, int, ID const&, ChannelAddress*) ()
#5  0x00000000005787c6 in main ()

If I run only 1 task (srun -n1 OpenSeesMP), no segfault occurs.
If I use mpirun instead of srun, no segfault.
If I move aside mca_btl_vader.so, no segfault either.

So I have 2 questions:

1. why is vader even used?
2. what causes the segfault?

Thanks!

[rhc54]

Vader is an alternative, potentially faster, shared memory BTL. I'm not sure why it would be segfaulting on your machine, but it is probably missing some startup info when direct launched by srun instead of mpirun. For those cases, you can just put OMPI_MCA_btl=^vader in your environment to turn it off.

We'll take a look and see if we can ID the problem. Have you tried 1.8.3 to see if the problem exists there too?

On Oct 14, 2014, at 3:34 PM, Kilian Cavalotti notifications@github.com wrote:

Hi,

One of our users reported a segfault that appears under specific circumstances.

When using Slurm (14.03.8) and srun to launch an OpenSees job (http://opensees.berkeley.edu/index.php, compiled and executed with OpenMPI 1.8.2) with more than 2 tasks, the execution fails with a segmentation fault that seems to occur in the vader BTL. I have no idea why it's being used, we're running a x86 Linux cluster on Redhat 6.5.

Reproducer is a follows:

$ srun -n 2 --pty bash
cn01:~$ srun OpenSeesMP
srun: error: cn01: task 0: Segmentation fault
The stack is:

cn01:~$ srun gdb OpenSeesMP
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
[...]
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff1c2e8ea in mca_btl_vader_sendi () from /share/sw/free/openmpi/1.8.2/gcc/4.4/lib/openmpi/mca_btl_vader.so
[...]
bt
(gdb) (gdb) #0 0x00007ffff1c2e8ea in mca_btl_vader_sendi () from /share/sw/free/openmpi/1.8.2/gcc/4.4/lib/openmpi/mca_btl_vader.so
#1 0x00007ffff15f604f in mca_pml_ob1_send_inline () from /share/sw/free/openmpi/1.8.2/gcc/4.4/lib/openmpi/mca_pml_ob1.so
#2 0x00007ffff15f6fe1 in mca_pml_ob1_send () from /share/sw/free/openmpi/1.8.2/gcc/4.4/lib/openmpi/mca_pml_ob1.so
#3 0x00007ffff7b7868a in PMPI_Send () from /share/sw/free/openmpi/1.8.2/gcc/4.4/lib/libmpi.so.1
#4 0x00000000005a1315 in MPI_Channel::sendID(int, int, ID const&, ChannelAddress*) ()
#5 0x00000000005787c6 in main ()
If I run only 1 task (srun -n1 OpenSeesMP), no segfault occurs.
If I use mpirun instead of srun, no segfault.
If I move aside mca_btl_vader.so, no segfault either.

So I have 2 questions:

1. why is vader even used?
2. what causes the segfault?

Thanks!

—
Reply to this email directly or view it on GitHub.

[kcgthb]

@rhc54: thanks for the explanation. Setting OMPI_MCA_btl=^vader indeed prevents the segfault from happening.

I've installed 1.8.3 and I'll work with the user to see if we can reproduce the problem with that version. I'll keep you posted.

[jsquyres]

@hjelmn Are you aware of any direct-launch problems with Vader?

[kcgthb]

Quick update: the segfault happens under the same circumstances with Open MPI 1.8.3, and produces the same exact backtrace.

[rhc54]

Are you using PMI-1 or PMI-2?

On Oct 15, 2014, at 11:29 AM, Kilian Cavalotti notifications@github.com wrote:

Quick update: the segfault happens under the same circumstances with Open MPI 1.8.3, and produces the same exact backtrace.

—
Reply to this email directly or view it on GitHub.

[rhc54]

Using the head of the 1.8 series (to be released soon as 1.8.4) and Slurm 2.5.4 (which means PMI-1), it works just fine.

On Oct 15, 2014, at 11:39 AM, Ralph Castain rhc@open-mpi.org wrote:

Are you using PMI-1 or PMI-2?

On Oct 15, 2014, at 11:29 AM, Kilian Cavalotti notifications@github.com wrote:

Quick update: the segfault happens under the same circumstances with Open MPI 1.8.3, and produces the same exact backtrace.

—
Reply to this email directly or view it on GitHub.

[kcgthb]

@rhc54: I'm actually not sure. Slurm 14.03.8 provides both PMI-1 and PMI-2, and I compiled Open MPI with --with-mpi, which seems to have picked up both. libmca_common_pmi.so ends up being linked against both libpmi.so.0 and libpmi2.so.0. From the Open MPI FAQ, it seems that if both are available, PMI-2 would be used, is that correct?

[kcgthb]

If I run srun --mpi=none OpenSeesMP, I got the following message, but no segfault.

--------------------------------------------------------------------------
PMI2 failed to initialize, returning an error code of 14.
We cannot use PMI2 at this time, and your job will
likely abort.
--------------------------------------------------------------------------

So it looks indeed that it occurs in conjunction with PMI-2

[rhc54]

Nope - Slurm defaults to using PMI-1 unless you explicitly tell it to use pmi2 on the srun cmd line. We just link so we have support for either one you choose to use.

FWIW: I can direct launch an MPI "hello" and an MPI ring example just fine using PMI-1 or PMI-2 under what was released as Slurm 14 without problem, including letting vader run.

On Oct 15, 2014, at 11:57 AM, Kilian Cavalotti notifications@github.com wrote:

@rhc54: I'm actually not sure. Slurm 14.03.8 provides both PMI-1 and PMI-2, and I compiled Open MPI with --with-mpi, which seems to have picked up both. libmca_common_pmi.so ends up being linked against both libpmi.so.0 and libpmi2.so.0. From the Open MPI FAQ, it seems that if both are available, PMI-2 would be used, is that correct?

—
Reply to this email directly or view it on GitHub.

[kcgthb]

@rhc54: Oh ok. I was under the impression that PMI-2 would be used over PMI-1 by reading the FAQ, especially this: "When the --with-pmi option is given, OMPI will automatically determine if PMI-2 support was built and use it in place of PMI-1" in https://www.open-mpi.org/faq/?category=slurm#slurm-direct-srun-mpi-apps

I was also able to run a simple MPI hello world program without a segfault, so it seems to be somewhat related to the OpenSees application, except the stack trace really seems to point at vader, which confuses me.

[rhc54]

On Oct 15, 2014, at 1:19 PM, Kilian Cavalotti notifications@github.com wrote:

@rhc54: Oh ok. I was under the impression that PMI-2 would be used over PMI-1 by reading the FAQ, especially this: "When the --with-pmi option is given, OMPI will automatically determine if PMI-2 support was built and use it in place of PMI-1" in https://www.open-mpi.org/faq/?category=slurm#slurm-direct-srun-mpi-apps

We can only decide that for build purposes - Slurm needs to know which one to actually use when it runs. So if you want Slurm to use PMI-2, you have to tell it to do so. Either on the srun cmd line, or you can set it as the default in the slurm.conf file.

I was also able to run a simple MPI hello world program without a segfault, so it seems to be somewhat related to the OpenSees application, except the stack trace really seems to point at vader, which confuses me.

Could be memory corruption? You could try running it thru valgrind. Add --enable-memchecker --enable-valgrind to the OMPI configure line so OMPI will be (relatively) clean.

—
Reply to this email directly or view it on GitHub.

[hjelmn]

@jsquyres vader runs cleanly for me launching directly with either srun and aprun

Can you provide more details? A line number at a minimum would help finger the culprit. It is very very unlikely a vader issue but rather an application issue. The sendi function does nothing more than copy from the user's pointer to a shared memory region. The region is allocated in exactly the same way it is in btl/sm (except each process owns its own segment).

[ggouaillardet]

i am able to reproduce the issue
under srun, when task 0 calls vader_add_procs, it ends up calling
segment_attach from shmem_mmap_module.c
but
ds_buf->seg_name is "/tmp/openmpi-sessi"
/* truncated string /
then
ds_buf->seg_id=open(ds_buf->seg_name, O_CREAT | O_RDWR, 0600)
creates an empty file.
/ i did not analyze this fully, but comments tell that segment_attach can only
be called after a successful call to segment_create, so do we really need the O_CREAT flag ? */
and then
mmap(...,ds_buf->seg_size, ..., ds_buf->seg_id,...);
does success though seg_size is 4M and the file is empty
which means we cannot read/write to ds_buf->seg_base_addr and that will lead do a crash later.

at this stage of the investigation :

• the truncated string is the cause of the crash (but not the root cause i am now looking for,
  string is not truncated when mpirun is used)
• is the O_CREAT flag required or a bug ?
• can someone explain why mmap did not fail ? that looks very counter intuitive to me ...

[ggouaillardet]

PR #238 makes the master work again, i am now investigating the latest v1.8