--report-bindings on whole machine, plus numa markers

[markalle]

I combined a few related features here:

1. --report-bindings on whole machine
2. mark unallowed (eg cgroup) parts of the whole machine with ~
3. numa markers
4. allow hwloc tree to not have sockets/cores

---

1. whole machine:

The incoming topology to the pretty-print functions probably didn't use the WHOLE_SYSTEM flag. In general we don't want WHOLE_SYSTEM, but for pretty printing I think it makes more sense. So I'm caching a WHOLE_SYSTEM load of the current machine and using it if the incoming topology also identifies as the current machine.

Examples of what pretty-printing looks like with/without whole system:

Suppose the machine is

    [..../..../..../....][..../..../..../....]
     0    4    8    12    16   20   24   28

and we run with

    cgset -r cpuset.cpus=24,25,28,29 mycgroup1
    cgset -r cpuset.cpus=26,27,30,31 mycgroup2
to leave only these hardware threads active:
    mycgroup1: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/..~~/..~~]
    mycgroup2: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/~~../~~..]

Without whole-system the printout (for both of the above) would be
(-np 2)

    MCW rank 0 bound to socket 1[core 0[hwt 0-1]]: [][BB/..]
    MCW rank 1 bound to socket 1[core 1[hwt 0-1]]: [][../BB]

With whole-system the output is this, which I think is more informative

mycgroup1 (-np 2):
    MCW rank 0 bound to socket 1[core 6[hwt 0-1]]: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/BB~~/..~~]
    MCW rank 1 bound to socket 1[core 7[hwt 0-1]]: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/..~~/BB~~]
mycgroup2 (-np 2):
    MCW rank 0 bound to socket 1[core 6[hwt 2-3]]: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/~~BB/~~..]
    MCW rank 1 bound to socket 1[core 7[hwt 2-3]]: [~~~~/~~~~/~~~~/~~~~][~~~~/~~~~/~~../~~BB]

2. mark unallowed (~)

When using the whole-machine option there's a bitmask available to identify
the allowed PUs, eg omitting PUs not in our cgroup. To distinguish those
PUs I'm using "~"

3. numa markers (<>)

I like having numa markers as well as the existing separators between sockets and cores. They're a little harder since the numas are more fluid, eg sockets always contain cores not vice versa, so you can hard code a loop over sockets follwed by a loop over cores. But numas might be be above or below sockets in the tree.

This code identifies which level should be considered the child of the numas, and has each of the hard coded loops capable of adding numa markers.

Currently I don't have any tunable to turn off the numa markers. A lot of machines have fairly simple numa output where each socket contains one numa, and that ends up looking like this:
[<..../..../..../....>][<..../..../..../....>]
If others feel that's too cluttered I'm okay with having some tunable so people would have to ask for numa markers.

4. allow hwloc tree to not have sockets/cores

I may be behind the times on hwloc development, but as far as I know hwloc trees aren't guaranteed to have sockets and cores, just a MACHINE at the top and PU at the bottom. So I added a little code to the loops so it would still print the PUs on a hypothetical machine that lacked any structuring of the PUs into cores/sockets.

Signed-off-by: Mark Allen markalle@us.ibm.com

[rhc54]

How do you know what the "whole machine" looks like on all the compute nodes? It appears that you are assuming all nodes are identical and so you can just mark any "disallowed" PUs on the remote machines based on this machine's local topology. That seems like an often inaccurate assumption as it depends on the cgroup settings, what processors are active on each node, etc.

It feels like you would have to get each daemon to report the "whole machine" to ensure you have a complete picture in order to get this right. We currently rely on our "topology signature" to detect whether or not a node is different from where mpirun is executing, but that might not be good enough to accurately detect a difference in this case (e.g., we only report the number of available CPUs, not which ones are available vs disallowed). Might take some playing around to determine what is required to meet this objective.

I'm also a bit disturbed about marking NUMA regions as there are some significant assumptions being made in the PR. The entire concept of "NUMA" is undergoing considerable change as memory layouts rapidly evolve. At the least, we should make it an option (not the default) as it may not even mean anything on upcoming systems.

I have no real objection to the idea of showing a broader picture of the machine - I'm just not sure you are accurately doing so with this PR.

[markalle]

Shoot, I didn't trace the lifecycle of the incoming topology for those calls. If it's not the local machine then I agree this PR doesn't get us very far.

Without having looked at the specifics yet the design I'm leaning toward is

• at topology_load time load without WHOLE_SYS as currently done, and load again with WHOLE_SYS and attach the WHOLE_SYS version into the userdata of the main version, that way the WHOLE_SYS is available but doesn't interfere because it's not being used for most things
• make the signature take into account both the main topology + any WHOLE_SYS attached to it
• wherever topologies are sent/recved double that code to handle the attached WHOLE_SYS (I haven't looked at it, does this use the hwloc XML import/export functions?)

It doesn't sound conceptually too hard but may be a bit of code

[rhc54]

You don't need to retain the WHOLE_SYS on the backend compute nodes since nothing back there can use it. What you need to do is:

• generate a signature that somehow exposes what the pattern of available/disallowed PUs looks like. This signature gets passed back to mpirun in the "phone home" message towards the end of "daemon init". Remember, which PUs are allowed/disallowed isn't constant across the nodes - thus, it isn't enough to know the number, you have to know precisely which PUs fall into each category.

• mpirun already compares its signature with the ones coming in from the remote nodes. When they don't match, it sends a message to that one node and asks it for a dump of its topology. This is where you could ask that it send the WHOLE_SYS version. The returned topology gets stored in mpirun and the node object's topology field is pointed to it.

When generating the binding output, we always use the topology for that specific node, and so you should then get the correct output.

[hppritcha]

bot:ompi:retest

[gpaulsen]

@hppritcha Looks like Cray aborted, due to build timeout:
https://jenkins.open-mpi.org/jenkins/job/open-mpi.build.platforms/Platform=cray_cle/4989/console

7. Running autotools on top-level tree

==> Remove stale files
==> Writing m4 file with autogen.pl results
==> Generating opal_get_version.sh
Running: autom4te --language=m4sh opal_get_version.m4sh -o opal_get_version.sh
==> Running autoreconf
Running: autoreconf -ivf --warnings=all,no-obsolete,no-override -I config
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I config --force --warnings=all,no-obsolete,no-override -I config
Build timed out (after 10 minutes). Marking the build as aborted.
Build was aborted
Finished: ABORTED

[markalle]

I took a different approach just adding a new rank-level option
-mca report_bindings_full 1

I agree with the mpirun-level stuff using the regular non-whole-system topologies and collecting that for its signatures and its info about the workers. Expanding those to include WHOLE_SYSTEM seemed like a lot of extra data and data collection for a small-ish feature.

So I added an alternate method of printing bindings that happens at the rank-level, where the needed topologies are easily available.

The only practical downside I see is the usual --report-bindings was capable of printing bindings even if your app was "hostname" while my -mca report_bindings_full 1 is only going to reach its print statements if the app calls MPI_Init.

[rhc54]

You are kidding, yes? You are really going to have 1000s of ranks each dumping their own binding string across stdout?? You do realize that the output will all be interleaved and hard to read, yes?

I realize the alternative is a bit of work, but surely that is better than something effectively unusable.

[markalle]

It's sent to rank 0 who does the printing. It's a big blob of stdout same as --report-bindings is already a big blob of output. It's more communication than the regular method because it has no compression to realize many ranks are sending the same data, but I don't think it's too excessive of a gatherv

[markalle]

I pushed a topology signature update (second commit to this PR) that adds a compressed list of os_indexes for the PUs. This way two different cgroups that look the same based on their socket/core/PU/etc counts come in with different signatures, and the compressed lists tend to be pretty small.

I'm still mulling over whether to change back to modifying the original --report-bindings from the mpirun level. Currently this PR drops in a new rank-level report-bindings option that's independent of the old, so it doesn't have to worry about what topologies are collected by mpirun.

[bgoglin]

The compressed list of PU indexes should be identical to hwloc_bitmap_list_asprintf(&string, hwloc_topology_get_topology_cpuset(topology)). There's also a snprintf() if you prefer static buffers.

[markalle]

That's a good looking function, but I don't think it does as much compression as mine. Eg noticing patterns analogous to MPI_Type_vector:

int a[] = {1,2,3,4,5, 11,12,13,14,15, 21,22,23,24,25};

Yours gives "1-5,11-15,21-25" where mine identifies the repetition as "1-5+10*3"

My code has an easier job though because I don't have to convert both ways, so I didn't bother writing a conversion function for the other direction

[markalle]

I'm about to push again just to change the code for the --report-bindings path to use a cached WHOLE_SYSTEM topology. The earlier comments had me believing those strings were produced at mpirun, but they're not, at least not in the code path I've been tracing. So a simple caching of one WHOLE_SYSTEM topology to be used inside the pretty-print functions is all it takes to make --report-bindings pretty again.

I'm still leaving in my allternate path rank-level pretty printer as a backup in case there are other codepaths I'm not aware of, eg if it turns out my --report-bindings WHOLE_SYSTEM fix is only a partial fix.

[jsquyres]

So a simple caching of one WHOLE_SYSTEM topology to be used inside the pretty-print functions is all it takes to make --report-bindings pretty again.

@markalle I'm a little confused -- isn't the premise that the nodes in the system can be heterogeneous (i.e., more than one topology)?

[markalle]

Yeah, but those --report-bindings output strings are being produced remotely, at the heterogeneous nodes. So each node only needs its own topology loaded WHOLE_SYSTEM to produce its pretty string, as opposed to the mpirun-level node needing everybody's topology in WHOLE_SYSTEM form.

Looking through all the "cset2mapstr" calls I think the ones that matter for --report-bindings output are all using opal_hwloc_topology. There are some cset2mapstr calls that operate on node->topology->topo for example and those calls would produce the less-pretty string, but I think those are things like affinity running at some verboseness setting, not the --report-bindings output which is where I think the pretty-print has the most value

[gpaulsen]

@jsquyres @rhc54 @bwbarrett, Is there anything else for this PR? If not, lets get this in.

[rhc54]

I'd rather do this right by taking into account the hetero nature of the system that is inherent in this use-case. I'll try to put something together.

[markalle]

I think it's handling at least one significant aspect of the heterogeneous nature in the second commit. That's the one that expands the signatures a bit so heterogeneous cgroups that might otherwise look the same will successfully produce different signatures and thus trigger mpirun to ask for topologies from those systems.

Or were you talking about some other aspect of the hetero cases that needs to be covered?

[rhc54]

This has been dealt with in PRRTE and is no longer applicable to OMPI master