including openib in btl list causes horrible vader or sm performance.

[gpaulsen]

On master branch

I observe a strange behavior. I think that openib may be using too large of a hammer for numa membinding, possibly setting the wrong memory binding policy for the vader and sm shared memory segments. I've only come to this conclusion empirically based on performance numbers.

For example, I have a RHEL 6.5 node with a single Mellanox Technologies MT25204 [InfiniHost III Lx HCA] ConnectX-3 card with a single port active.

Bad Latency run single host:

$  mpirun -host "mpi03" -np 4 --bind-to core --report-bindings --mca btl openib,vader,self ./ping_pong_ring.x2
[mpi03:12941] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..]
[mpi03:12941] MCW rank 1 bound to socket 1[core 8[hwt 0-1]]: [../../../../../../../..][BB/../../../../../../..]
[mpi03:12941] MCW rank 2 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../..][../../../../../../../..]
[mpi03:12941] MCW rank 3 bound to socket 1[core 9[hwt 0-1]]: [../../../../../../../..][../BB/../../../../../..]
[0:mpi03] ping-pong 0 bytes ...
0 bytes: 7.11 usec/msg
[1:mpi03] ping-pong 0 bytes ...
0 bytes: 7.10 usec/msg
[2:mpi03] ping-pong 0 bytes ...
0 bytes: 7.15 usec/msg
[3:mpi03] ping-pong 0 bytes ...
0 bytes: 7.17 usec/msg

Similar behavior with sm:

$ mpirun -host "mpi03" -np 4 --bind-to core --report-bindings --mca btl openib,sm,self ./ping_pong_ring.x2
[mpi03:14928] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..]
[mpi03:14928] MCW rank 1 bound to socket 1[core 8[hwt 0-1]]: [../../../../../../../..][BB/../../../../../../..]
[mpi03:14928] MCW rank 2 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../..][../../../../../../../..]
[mpi03:14928] MCW rank 3 bound to socket 1[core 9[hwt 0-1]]: [../../../../../../../..][../BB/../../../../../..]
[0:mpi03] ping-pong 0 bytes ...
0 bytes: 7.45 usec/msg
[1:mpi03] ping-pong 0 bytes ...
0 bytes: 7.38 usec/msg
[2:mpi03] ping-pong 0 bytes ...
0 bytes: 7.35 usec/msg
[3:mpi03] ping-pong 0 bytes ...
0 bytes: 7.38 usec/msg

When I remove openib results look much better:

$ mpirun -host "mpi03" -np 4 --bind-to core --report-bindings --mca btl vader,self ./ping_pong_ring.x2
[mpi03:15819] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..]
[mpi03:15819] MCW rank 1 bound to socket 1[core 8[hwt 0-1]]: [../../../../../../../..][BB/../../../../../../..]
[mpi03:15819] MCW rank 2 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../..][../../../../../../../..]
[mpi03:15819] MCW rank 3 bound to socket 1[core 9[hwt 0-1]]: [../../../../../../../..][../BB/../../../../../..]
[0:mpi03] ping-pong 0 bytes ...
0 bytes: 0.50 usec/msg
[1:mpi03] ping-pong 0 bytes ...
0 bytes: 0.50 usec/msg
[2:mpi03] ping-pong 0 bytes ...
0 bytes: 0.49 usec/msg
[3:mpi03] ping-pong 0 bytes ...
0 bytes: 0.51 usec/msg

Similar behavior with sm (though it's half as fast as vader):

$ mpirun -host "mpi03" -np 4 --bind-to core --report-bindings --mca btl sm,self ./ping_pong_ring.x2
[mpi03:16608] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..]
[mpi03:16608] MCW rank 1 bound to socket 1[core 8[hwt 0-1]]: [../../../../../../../..][BB/../../../../../../..]
[mpi03:16608] MCW rank 2 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../..][../../../../../../../..]
[mpi03:16608] MCW rank 3 bound to socket 1[core 9[hwt 0-1]]: [../../../../../../../..][../BB/../../../../../..]
[0:mpi03] ping-pong 0 bytes ...
0 bytes: 0.98 usec/msg
[1:mpi03] ping-pong 0 bytes ...
0 bytes: 1.00 usec/msg
[2:mpi03] ping-pong 0 bytes ...
0 bytes: 0.95 usec/msg
[3:mpi03] ping-pong 0 bytes ...
0 bytes: 0.93 usec/msg

If I disable binding explicitly with --bind-to none, even when specifying openib I see the expected results (with either vader or sm, but now sm is the same speed as vader... weird):

$ mpirun -host "mpi03" -np 4 --bind-to none --report-bindings --mca btl openib,vader,self ./ping_pong_ring.x2
[mpi03:20206] MCW rank 1 is not bound (or bound to all available processors)
[mpi03:20205] MCW rank 0 is not bound (or bound to all available processors)
[mpi03:20207] MCW rank 2 is not bound (or bound to all available processors)
[mpi03:20208] MCW rank 3 is not bound (or bound to all available processors)
[0:mpi03] ping-pong 0 bytes ...
0 bytes: 0.50 usec/msg
[1:mpi03] ping-pong 0 bytes ...
0 bytes: 0.50 usec/msg
[2:mpi03] ping-pong 0 bytes ...
0 bytes: 0.50 usec/msg
[3:mpi03] ping-pong 0 bytes ...
0 bytes: 0.49 usec/msg

$ mpirun -host "mpi03" -np 4 --bind-to none --report-bindings --mca btl openib,sm,self ./ping_pong_ring.x2
[mpi03:21058] MCW rank 0 is not bound (or bound to all available processors)
[mpi03:21059] MCW rank 1 is not bound (or bound to all available processors)
[mpi03:21060] MCW rank 2 is not bound (or bound to all available processors)
[mpi03:21061] MCW rank 3 is not bound (or bound to all available processors)
[0:mpi03] ping-pong 0 bytes ...
0 bytes: 0.50 usec/msg
[1:mpi03] ping-pong 0 bytes ...
0 bytes: 0.51 usec/msg
[2:mpi03] ping-pong 0 bytes ...
0 bytes: 0.51 usec/msg
[3:mpi03] ping-pong 0 bytes ...
0 bytes: 0.49 usec/msg

Finally just for completeness... the best 0 byte ping pong ring times I could get was with --bind-to core --map-by core:

$ mpirun -host "mpi03" -np 4 --bind-to core --map-by core --report-bindings --mca btl vader,self ./ping_pong_ring.x2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
[mpi03:32149] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..]
[mpi03:32149] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../..][../../../../../../../..]
[mpi03:32149] MCW rank 2 bound to socket 0[core 2[hwt 0-1]]: [../../BB/../../../../..][../../../../../../../..]
[mpi03:32149] MCW rank 3 bound to socket 0[core 3[hwt 0-1]]: [../../../BB/../../../..][../../../../../../../..]
[0:mpi03] ping-pong 0 bytes ...
0 bytes: 0.37 usec/msg
[1:mpi03] ping-pong 0 bytes ...
0 bytes: 0.37 usec/msg
[2:mpi03] ping-pong 0 bytes ...
0 bytes: 0.38 usec/msg
[3:mpi03] ping-pong 0 bytes ...
0 bytes: 0.38 usec/msg

I've attached my source for ping_pong_ring.c:

ping_pong_ring.txt

[gpaulsen]

@hjelmn - Jeff Squyres said I should tag you on this, that you might have some insight.

[hjelmn]

Yeah. This happens because we are polling the completion queue. I have been planning to work on this when i get a chance.

[hppritcha]

@gpaulsen which milestone would you like to give this. If it involves a lot of changes I'd prefer it to be in 2.1.0 or maybe 2.0.1? (I'll add the 2.0.1 milestone in a bit)

[gpaulsen]

I was hoping for 2.0, (I didn't see a 2.0 label), as this makes openib runs pretty useless.
This is NOT a blocker for IBM, but I assumed it was a blocker for Open MPI Community.

[jsquyres]

Hey @hjelmn, can we talk about this tomorrow on the call? I'm wondering:

• Should the openib BTL just be smart enough to not register its progress function when its add_procs() hasn't found any peers, and/or del_procs() has removed all peers?
  • As a side effect: is the openib BTL progress function required to process incoming connections? Or is this not even an issue if the BTL determines that it can't talk to any peers? More specifically: is there ever a race condition -- e.g., involving dynamic processes -- where an incoming connection request could arrive in a BTL (such as the openib BTL) before the BTL knew that an incoming connection could be coming from that peer?
• I'm guessing that other BTL will also have this (set of) issues. Do you know?
• If so, how much of a change does this represent to openib (and potentially others)? I.e., do we need this for 2.0.0, or can it wait until 2.1.0? I ask because Open MPI has had this behavior for a long time, so it's not technically a regression. But it would be a nice performance issue to fix.

[jsquyres]

Discussion on the call...

This seems to be because of how we now add all endpoints for all BTLs (because of the new add_procs() work). Hence, even though a process may be 100% on a single server, if network BTLs determine that they could run (e.g., openib), the openib BTL modules will be added in the PML. Therefore we queue up their progress functions and poll them -- even though they will never be used (unless a dynamic process comes along and uses them).

There's a secondary implication: one-sided atomics. It's unfortunate that CPU atomics != network hardware atomics, so we have to go 100% one way or the other. The current implementation uses the network stack atomics (for the same reason as above: a dynamic process may come along and therefore require the use of network atomics).

@hjelmn is going to look at a slightly different approach: have opal_progress() monitor the progress functions that it calls. Progress functions that don't return any events for a long time will get downgraded in priority and called less frequently.

[gpaulsen]

We discussed in today's call that this is a complex issue with the new dynamic add procs, and that we must include network endpoints for two reasons. One is that if we use any network atomics, all atomics must be network atomics. The other is to support spawning a new job on another node.

Nathan's proposed solution is to add a decay function to the progression loop, so that any components that are not actually progressing anything, won't get called as often.

[bosilca]

Talking specifically about IB, we should take advantage of the fact that it is a connection-based network and only register the progress callback when there are established connections. More generally, a more reasonable approach will be to delay the progress registration until there is something to progress (this is under the assumption that the connection establishment is handled by a separate thread).

[jsquyres]

Agreed: that, too.

[gpaulsen]

I should add that this was build Multi-threaded, for x86 with the GNU compilers.

[gpaulsen]

I'm ready to retest. Please comment in this issue with which PR to try.

[hppritcha]

@hjelmn do you have an eta for the backoff fix?