Treematch issues in master #4303

rhc54 · 2017-10-04T20:31:31Z

The treematch topology component is segfaulting in master when running MTT:

$ mpirun --oversubscribe --bind-to none   -np 16  topology/distgraph1 
using graph layout 'deterministic complete graph'
testing MPI_Dist_graph_create_adjacent
testing MPI_Dist_graph_create w/ outgoing only
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using defaultdistgraph1: topo_treematch_dist_graph_create.c:656:
mca_topo_treematch_dist_graph_create: Assertion `(int)sol->k_length == size' failed.
[rhc001:14020] *** Process received signal ***
[rhc001:14020] Signal: Aborted (6)
[rhc001:14020] Signal code:  (-6)[rhc001:14020] [ 0] /lib64/libpthread.so.0(+0xf370)[0x7f385b8e8370]
[rhc001:14020] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x7f385b54d1d7]
[rhc001:14020] [ 2] /lib64/libc.so.6(abort+0x148)[0x7f385b54e8c8]
[rhc001:14020] [ 3] /lib64/libc.so.6(+0x2e146)[0x7f385b546146]
[rhc001:14020] [ 4] /lib64/libc.so.6(+0x2e1f2)[0x7f385b5461f2]
[rhc001:14020] [ 5]
/home/common/openmpi/build/foobar/lib/openmpi/mca_topo_treematch.so(mca_topo_treematch_dist_graph_create+0x21e9)[0x7f384396e702]
[rhc001:14020] [ 6]
/home/common/openmpi/build/foobar/lib/libmpi.so.0(PMPI_Dist_graph_create+0x44d)[0x7f385bb7a83e]
[rhc001:14020] [ 7] topology/distgraph1[0x40219e]
[rhc001:14020] [ 8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f385b539b35]
[rhc001:14020] [ 9] topology/distgraph1[0x400fc9]
[rhc001:14020] *** End of error message ***-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
---------------------------------------------------------------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node rhc001 exited on signal 6 (Aborted).
--------------------------------------------------------------------------

The text was updated successfully, but these errors were encountered:

bosilca · 2017-10-05T04:59:56Z

We're looking into it, but I can't replicate it on any of my machines. The execution path in treematch depends on the local architecture of the platform. Can we have a description of the setting where this assert triggered ?

rhc54 · 2017-10-05T11:51:54Z

Here is the topology (local.txt is the xml version).

local.pdf

local.txt

Let me know if I can provide any debug output.

GuillaumeMercier · 2017-12-11T13:49:39Z

I'm unable to replicate this issue on our machines. The test program works fine. However, it fails when I try your topology (local.txt) with this error:

topology discovery failed
--> Returned value Not supported (-8) instead of ORTE_SUCCESS

I need to investigate this more.

rhc54 · 2017-12-12T16:48:53Z

Here's a little more info from a failure on AWS:

using graph layout 'deterministic complete graph'
testing MPI_Dist_graph_create_adjacenttesting MPI_Dist_graph_create w/ outgoing only
========== Centralized Reordering ========= 
*** Error: Core numbering not between 0 and 27: tab_node[18]=33    <<<<====== NOTE
nb_constraints = 0, N= 2; nb_processing units = 27                 <<<<====== NOTE
Error : More processes (2) than number of constraints              <<<<====== NOTE
(0)!-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
---------------------------------------------------------------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[40635,1],0]
  Exit code:    255

jsquyres · 2017-12-12T17:22:23Z

This same distgraph1 test is doing odd things for me at Cisco on a pair of 16-core nodes.

Here's a gist (https://gist.github.com/jsquyres/098f256cead9d20d2ad1c3aea0e6b0be) showing:

lstopo for the 2 nodes where the test was run
output from the test

The test doesn't actually fail for me, but it does give approximately ~21K lines like this:

[mpi025:15395] Unable to extract peer [[30746,1],2] nodeid from the modex.

And yes, I mean approximately twenty-one thousand lines like this.

Here's the exact mpirun command I used (inside a SLURM allocation containing these 2 nodes):

mpirun --mca btl tcp,vader,self distgraph1

GuillaumeMercier · 2017-12-12T17:45:49Z

@jsquyres: George and I are experiencing the same output also.
@rhc54: can you give me your mpiexec command line also? Thanks.

rhc54 · 2017-12-12T17:54:15Z

For that aws error report? It was from the v3.0 branch, and simply:

$ mpirun -n 2 topology/distgraph1

Here's another one, this from the v3.1.x branch:

$ mpirun -n 2 topology/distgraph1 
testing MPI_Dist_graph_create_adjacenttesting MPI_Dist_graph_create w/ outgoing only
Error: Cannot partition 2 elements in 3 parts
[ip-172-31-72-214:68988] *** Process received signal ***
[ip-172-31-72-214:68988] Signal: Segmentation fault (11)
[ip-172-31-72-214:68988] Signal code: Address not mapped (1)
[ip-172-31-72-214:68988] Failing at address: (nil)
Using default
Using default[ip-172-31-72-214:68988] [ 0] /lib64/libpthread.so.0(+0xf5a0)[0x7f06376965a0]
[ip-172-31-72-214:68988] [ 1]
/home/ec2-user/mtt-scratch/installs/xnS2/install/lib/openmpi/mca_topo_treematch.so(split_com_mat+0x10c)[0x7f0624c787c3]
[ip-172-31-72-214:68988] [ 2]
/home/ec2-user/mtt-scratch/installs/xnS2/install/lib/openmpi/mca_topo_treematch.so(kpartition_build_level_topology+0x141)[0x7f0624c78ed0]
[ip-172-31-72-214:68988] [ 3]
/home/ec2-user/mtt-scratch/installs/xnS2/install/lib/openmpi/mca_topo_treematch.so(kpartition_build_level_topology+0x2a7)[0x7f0624c79036]
[ip-172-31-72-214:68988] [ 4]
/home/ec2-user/mtt-scratch/installs/xnS2/install/lib/openmpi/mca_topo_treematch.so(kpartition_build_level_topology+0x2a7)[0x7f0624c79036]
[ip-172-31-72-214:68988] [ 5]
/home/ec2-user/mtt-scratch/installs/xnS2/install/lib/openmpi/mca_topo_treematch.so(kpartition_build_tree_from_topology+0x280)[0x7f0624c79360]
[ip-172-31-72-214:68988] [ 6]
/home/ec2-user/mtt-scratch/installs/xnS2/install/lib/openmpi/mca_topo_treematch.so(tm_build_tree_from_topology+0x1cb)[0x7f0624c75339]
[ip-172-31-72-214:68988] [ 7]
/home/ec2-user/mtt-scratch/installs/xnS2/install/lib/openmpi/mca_topo_treematch.so(mca_topo_treematch_dist_graph_create+0x213e)[0x7f0624c67526]
[ip-172-31-72-214:68988] [ 8]
/home/ec2-user/mtt-scratch/installs/xnS2/install/lib/libmpi.so.0(MPI_Dist_graph_create+0x45e)[0x7f063792ac20]
[ip-172-31-72-214:68988] [ 9] topology/distgraph1[0x40209a]
[ip-172-31-72-214:68988] [10] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f06372e4c05]
[ip-172-31-72-214:68988] [11] topology/distgraph1[0x400ec9]
[ip-172-31-72-214:68988] *** End of error message
***-------------------------------------------------------

And this is from master:

$ mpirun -n 2 topology/distgraph1 
using graph layout 'deterministic complete graph'
testing MPI_Dist_graph_create_adjacenttesting MPI_Dist_graph_create w/ outgoing only
Using default
Using default
Using default
Using default
Using default
Using default
Using default
Using defaultUsing default
distgraph1: topo_treematch_dist_graph_create.c:656: mca_topo_treematch_dist_graph_create: Assertion
`(int)sol->k_length == size' failed.
[ip-172-31-72-214:98463] *** Process received signal ***
[ip-172-31-72-214:98463] Signal: Aborted (6)
[ip-172-31-72-214:98463] Signal code:  (-6)
[ip-172-31-72-214:98463] [ 0] /lib64/libpthread.so.0(+0xf5a0)[0x7f65f19515a0]
[ip-172-31-72-214:98463] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x7f65f15b31f7]
[ip-172-31-72-214:98463] [ 2] /lib64/libc.so.6(abort+0x148)[0x7f65f15b48e8]
[ip-172-31-72-214:98463] [ 3] /lib64/libc.so.6(+0x2e266)[0x7f65f15ac266]
[ip-172-31-72-214:98463] [ 4] /lib64/libc.so.6(+0x2e312)[0x7f65f15ac312]
[ip-172-31-72-214:98463] [ 5]
/home/ec2-user/mtt-scratch/installs/zFDF/install/lib/openmpi/mca_topo_treematch.so(mca_topo_treematch_dist_graph_create+0x219e)[0x7f65def4e562]
[ip-172-31-72-214:98463] [ 6]
/home/ec2-user/mtt-scratch/installs/zFDF/install/lib/libmpi.so.0(MPI_Dist_graph_create+0x45e)[0x7f65f1be638d]
[ip-172-31-72-214:98463] [ 7] topology/distgraph1[0x40209a]
[ip-172-31-72-214:98463] [ 8] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f65f159fc05]
[ip-172-31-72-214:98463] [ 9] topology/distgraph1[0x400ec9]
[ip-172-31-72-214:98463] *** End of error message

GuillaumeMercier · 2017-12-12T18:36:30Z

@rhc54: the issue with the master is resolved : this assert should not be here in the first place.
For the 3.1.x, I need to check this.

bwbarrett · 2017-12-19T17:07:12Z

When I compile ompi with ./configure --prefix=<blah> from a git build, the error doesn't happen. When I compile ompi with ./configure CFLAGS=-pipe --enable-picky --enable-debug --prefix=$HOME/install, the error Ralph copy-and-pasted occurs.

bosilca · 2017-12-20T00:07:19Z

I have created PR #4644 o address the issue highlighted here.

rhc54 · 2017-12-20T00:20:51Z

Thanks George!

jsquyres · 2018-01-09T16:30:23Z

Per 2018-01-09 teleconf, @bwbarrett will follow up with @bosilca on this one.

bwbarrett · 2018-01-10T03:55:17Z

Reading the history in this ticket, clearly a fix is needed for v3.1.x, and it looks like it was never pulled. @bosilca, can you file a PR to merge the changes into v3.1.x? I'm not seeing the failure on v3.0.x, so I believe it's not needed there?

rhc54 added bug Target: main labels Oct 4, 2017

rhc54 assigned bosilca Oct 4, 2017

bosilca mentioned this issue Dec 20, 2017

Fix treematch topology assert #4644

Merged

bwbarrett added Severity: blocker Target: v3.1.x and removed Target: main labels Jan 10, 2018

gpaulsen mentioned this issue Jan 16, 2018

Topic/treematch v3.1.x #4720

Merged

bwbarrett closed this as completed Feb 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Treematch issues in master #4303

Treematch issues in master #4303

rhc54 commented Oct 4, 2017

bosilca commented Oct 5, 2017

rhc54 commented Oct 5, 2017

GuillaumeMercier commented Dec 11, 2017

rhc54 commented Dec 12, 2017 •

edited

Loading

jsquyres commented Dec 12, 2017

GuillaumeMercier commented Dec 12, 2017

rhc54 commented Dec 12, 2017 •

edited

Loading

GuillaumeMercier commented Dec 12, 2017

bwbarrett commented Dec 19, 2017 •

edited

Loading

bosilca commented Dec 20, 2017

rhc54 commented Dec 20, 2017

jsquyres commented Jan 9, 2018

bwbarrett commented Jan 10, 2018

Treematch issues in master #4303

Treematch issues in master #4303

Comments

rhc54 commented Oct 4, 2017

bosilca commented Oct 5, 2017

rhc54 commented Oct 5, 2017

GuillaumeMercier commented Dec 11, 2017

rhc54 commented Dec 12, 2017 • edited Loading

jsquyres commented Dec 12, 2017

GuillaumeMercier commented Dec 12, 2017

rhc54 commented Dec 12, 2017 • edited Loading

GuillaumeMercier commented Dec 12, 2017

bwbarrett commented Dec 19, 2017 • edited Loading

bosilca commented Dec 20, 2017

rhc54 commented Dec 20, 2017

jsquyres commented Jan 9, 2018

bwbarrett commented Jan 10, 2018

rhc54 commented Dec 12, 2017 •

edited

Loading

rhc54 commented Dec 12, 2017 •

edited

Loading

bwbarrett commented Dec 19, 2017 •

edited

Loading