Wrong binding to hwthreads

[igor-ivanov]

@rhc54 could you look at it.
I am observing this issue at least on 1.8.8 and trunk

I see something strange using --bind-to hwthread:
cpu_set 42,28,0,14 and 14,15,0,1 do the same???
Both report
Cpus_allowed_list: 0
Cpus_allowed_list: 14
Cpus_allowed_list: 28
Cpus_allowed_list: 42

• Expected:
  Cpus_allowed_list: 14
  Cpus_allowed_list: 1
  Cpus_allowed_list: 15
  Cpus_allowed_list: 0
• Existing configuration:

$numactl --hardware | grep cpus
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13                       28 29 30 31 32 33 34 35 36 37 38 39 40 41
node 1 cpus: 14 15 16 17 18 19 20 21 22 23 24 25 26 27 42 43 44 45 46 47 48 49 50 51 52 53 54 55

$hwloc-info
    Socket L#0 + L3 L#0 (35MB)
      L2 L#0 (256KB) + L1d L#0 (32KB) + Core L#0
        PU L#0 (P#0)
        PU L#1 (P#28)
      L2 L#1 (256KB) + L1d L#1 (32KB) + Core L#1
        PU L#2 (P#1)
        PU L#3 (P#29)
    Socket L#1 + L3 L#1 (35MB)
      L2 L#14 (256KB) + L1d L#14 (32KB) + Core L#14
        PU L#28 (P#14)
        PU L#29 (P#42)
      L2 L#15 (256KB) + L1d L#15 (32KB) + Core L#15
        PU L#30 (P#15)
        PU L#31 (P#43)

• I checked to ways:

1. using -mca hwloc variables:

$/usr/mpi/gcc/openmpi-1.8.8/bin/oshrun --report-bindings -n 4 --oversubscribe -mca hwloc_base_cpu_set 42,28,0,14 -mca hwloc_base_binding_policy hwthread -mca hwloc_base_use_hwthreads_as_cpus 1 bash -c "cat /proc/self/status |grep Cpus_allowed_list"
[clx-orion-001:13313] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001:13313] MCW rank 1 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
Cpus_allowed_list:      0
[clx-orion-001:13313] MCW rank 2 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
Cpus_allowed_list:      14
[clx-orion-001:13313] MCW rank 3 bound to socket 1[core 14[hwt 1]]: [../../../../../../../../../../../../../..][.B/../../../../../../../../../../../../..]
Cpus_allowed_list:      28
Cpus_allowed_list:      42

$/usr/mpi/gcc/openmpi-1.8.8/bin/oshrun --report-bindings -n 4 --oversubscribe -mca hwloc_base_cpu_set 14,15,0,1 -mca hwloc_base_binding_policy hwthread -mca hwloc_base_use_hwthreads_as_cpus 1 bash -c "cat /proc/self/status |grep Cpus_allowed_list"
[clx-orion-001:13346] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001:13346] MCW rank 1 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
Cpus_allowed_list:      0
[clx-orion-001:13346] MCW rank 2 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
Cpus_allowed_list:      14
[clx-orion-001:13346] MCW rank 3 bound to socket 1[core 14[hwt 1]]: [../../../../../../../../../../../../../..][.B/../../../../../../../../../../../../..]
Cpus_allowed_list:      28
Cpus_allowed_list:      42

2. using options:

$/usr/mpi/gcc/openmpi-1.8.8/bin/oshrun --report-bindings -n 4 --oversubscribe --cpu-set 14,42,0,28 --bind-to hwthread bash -c "cat /proc/self/status |grep Cpus_allowed_list"
[clx-orion-001:39050] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001:39050] MCW rank 1 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
Cpus_allowed_list:      0
[clx-orion-001:39050] MCW rank 2 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
Cpus_allowed_list:      14
[clx-orion-001:39050] MCW rank 3 bound to socket 1[core 14[hwt 1]]: [../../../../../../../../../../../../../..][.B/../../../../../../../../../../../../..]
Cpus_allowed_list:      28
Cpus_allowed_list:      42

$/usr/mpi/gcc/openmpi-1.8.8/bin/oshrun --report-bindings -n 4 --oversubscribe --cpu-set 14,15,0,1 --bind-to hwthread bash -c "cat /proc/self/status |grep Cpus_allowed_list"
[clx-orion-001:13495] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001:13495] MCW rank 1 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
Cpus_allowed_list:      0
[clx-orion-001:13495] MCW rank 2 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
Cpus_allowed_list:      14
[clx-orion-001:13495] MCW rank 3 bound to socket 1[core 14[hwt 1]]: [../../../../../../../../../../../../../..][.B/../../../../../../../../../../../../..]
Cpus_allowed_list:      28
Cpus_allowed_list:      42

[ggouaillardet]

out of curiosity, can you replace
grep Cpus_allowed_list /proc/self
with
hwloc-bind --get
and post the new output ?

also, did you use internal or external hwloc ?

[igor-ivanov]

by @ggouaillardet request
checked on v1.10
used internal hwloc

$/hpc/local/benchmarks/hpc-stack-gcc/install/ompi-v1.10/bin/oshrun  --report-bindings -n 4 --oversubscribe --cpu-set 14,42,0,28 --bind-to hwthread bash -c "cat /proc/self/status |grep Cpus_allowed_list"
[clx-orion-001.dmz.e2e.mlnx:51560] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001.dmz.e2e.mlnx:51560] MCW rank 1 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
[clx-orion-001.dmz.e2e.mlnx:51560] MCW rank 2 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001.dmz.e2e.mlnx:51560] MCW rank 3 bound to socket 1[core 14[hwt 1]]: [../../../../../../../../../../../../../..][.B/../../../../../../../../../../../../..]
Cpus_allowed_list:      0
Cpus_allowed_list:      28
Cpus_allowed_list:      14
Cpus_allowed_list:      42
$/hpc/local/benchmarks/hpc-stack-gcc/install/ompi-v1.10/bin/oshrun  --report-bindings -n 4 --oversubscribe --cpu-set 14,15,0,1 --bind-to hwthread bash -c "cat /proc/self/status |grep Cpus_allowed_list"
[clx-orion-001.dmz.e2e.mlnx:51586] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001.dmz.e2e.mlnx:51586] MCW rank 1 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
[clx-orion-001.dmz.e2e.mlnx:51586] MCW rank 2 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
Cpus_allowed_list:      0
[clx-orion-001.dmz.e2e.mlnx:51586] MCW rank 3 bound to socket 1[core 14[hwt 1]]: [../../../../../../../../../../../../../..][.B/../../../../../../../../../../../../..]
Cpus_allowed_list:      14
Cpus_allowed_list:      28
Cpus_allowed_list:      42

the same with hwloc-bind --get

$/hpc/local/benchmarks/hpc-stack-gcc/install/ompi-v1.10/bin/oshrun  --report-bindings -n 4 --oversubscribe --cpu-set 14,15,0,1 --bind-to hwthread bash -c "hwloc-bind --get"
[clx-orion-001.dmz.e2e.mlnx:51645] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001.dmz.e2e.mlnx:51645] MCW rank 1 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
[clx-orion-001.dmz.e2e.mlnx:51645] MCW rank 2 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001.dmz.e2e.mlnx:51645] MCW rank 3 bound to socket 1[core 14[hwt 1]]: [../../../../../../../../../../../../../..][.B/../../../../../../../../../../../../..]
0x10000000
0x00000001
0x00000400,0x0
0x00004000

$/hpc/local/benchmarks/hpc-stack-gcc/install/ompi-v1.10/bin/oshrun  --report-bindings -n 4 --oversubscribe --cpu-set 14,42,0,2 --bind-to hwthread bash -c "hwloc-bind --get"
[clx-orion-001.dmz.e2e.mlnx:51663] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001.dmz.e2e.mlnx:51663] MCW rank 1 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
[clx-orion-001.dmz.e2e.mlnx:51663] MCW rank 2 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001.dmz.e2e.mlnx:51663] MCW rank 3 bound to socket 1[core 14[hwt 1]]: [../../../../../../../../../../../../../..][.B/../../../../../../../../../../../../..]
0x00004000
0x00000400,0x0
0x00000001
0x10000000

[ggouaillardet]

@igor-ivanov
from mpirun --help

   -cpu-set|--cpu-set <arg0>  
                         Comma-separated list of ranges specifying logical
                         cpus allocated to this job [default: none]

--cpu-set expects a list of logical cpus.
/* did you really use the logical ids and not the physical ids ? */

one way to see things is cpu means core so what you are doing is wrong (e.g. you are passing non existing logical core id), and the issue is ompi did not fail with an understandable error message.

an other way to see things is cpu means PU (in hwloc jargon) which is an hwthread
(and this is equivalent to a core on a system where hyperthreading is disabled)
in this case, can you give a try to this inline patch, and add --map-by hwthread to your mpirun command line ?

/* i made zero non regression test, and this is unlikely wrong for primetime, also i did not assess yet whether having to --map-by hwthread is a bug or a feature */

diff --git a/opal/mca/hwloc/base/hwloc_base_util.c b/opal/mca/hwloc/base/hwloc_base_util.c
index cd429ee..52e8616 100644
--- a/opal/mca/hwloc/base/hwloc_base_util.c
+++ b/opal/mca/hwloc/base/hwloc_base_util.c
@@ -71,7 +71,7 @@ hwloc_obj_t opal_hwloc_base_get_pu(hwloc_topology_t topo,
        So first we have to see if we can find *any* cores by looking
        for the 0th core.  If we find it, then try to find the Nth
        core.  Otherwise, try to find the Nth PU. */
-    if (NULL == hwloc_get_obj_by_type(topo, HWLOC_OBJ_CORE, 0)) {
+    if (OPAL_HWLOC_LOGICAL == rtype || NULL == hwloc_get_obj_by_type(topo, HWLOC_OBJ_CORE, 0)) {
         obj_type = HWLOC_OBJ_PU;
     }

[igor-ivanov]

@ggouaillardet I have a node with 28 cores and 56cpus in case HT is ON.
I would like to map my processes to specific cpus of 56cpus. Detailed node configuration is shown in the issue description.
Do you know valid mpirun command I can do what I want?
Commands I am using report identical result for 42,28,0,14 and 14,15,0,1 cpu sets.

[ggouaillardet]

Did you try my patch and --map-by hwthread ?
I expect that would work

[igor-ivanov]

@ggouaillardet will check when get access to a node with enabled ht.

[igor-ivanov]

@ggouaillardet I can confirm

an other way to see things is cpu means PU (in hwloc jargon) which is an hwthread
(and this is equivalent to a core on a system where hyperthreading is disabled)
in this case, can you give a try to this inline patch, and add --map-by hwthread to your mpirun > > command line ?

after checking your change with observation that using --map-by hwthread does not impact result (using --map-by hwthread with original code force another mapping: bad but another)
based on #4f8c0c5 event/external: add missing include file

• w/ your change (no --map-by hwthread)

$$HOME/prj/ii-ompi/build/install/bin/oshrun --report-bindings -n 4 --oversubscribe --cpu-set 14,15,0,1 --bind-to hwthread bash -c "cat /proc/self/status |grep Cpus_allowed_list"
[clx-orion-065.dmz.e2e.mlnx:15998] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:15998] MCW rank 1 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:15998] MCW rank 2 bound to socket 0[core 7[hwt 0]]: [../../../../../../../B./../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:15998] MCW rank 3 bound to socket 0[core 7[hwt 1]]: [../../../../../../../.B/../../../../../..][../../../../../../../../../../../../../..]
Cpus_allowed_list:      0
Cpus_allowed_list:      28
Cpus_allowed_list:      7
Cpus_allowed_list:      35

$$HOME/prj/ii-ompi/build/install/bin/oshrun --report-bindings -n 4 --oversubscribe --cpu-set 14,42,0,28 --bind-to hwthread --map-by hwthread bash -c "cat /proc/self/status |grep Cpus_allowed_list"
[clx-orion-065.dmz.e2e.mlnx:16018] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:16018] MCW rank 1 bound to socket 0[core 7[hwt 0]]: [../../../../../../../B./../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:16018] MCW rank 2 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:16018] MCW rank 3 bound to socket 1[core 21[hwt 0]]: [../../../../../../../../../../../../../..][../../../../../../../B./../../../../../..]
Cpus_allowed_list:      0
Cpus_allowed_list:      7
Cpus_allowed_list:      14
Cpus_allowed_list:      21

$$HOME/prj/ii-ompi/build/install/bin/oshrun --report-bindings -n 4 --oversubscribe --cpu-set 14,15,0,1 --bind-to hwthread --map-by hwthread bash -c "hwloc-bind --get"
[clx-orion-065.dmz.e2e.mlnx:16034] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:16034] MCW rank 1 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:16034] MCW rank 2 bound to socket 0[core 7[hwt 0]]: [../../../../../../../B./../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:16034] MCW rank 3 bound to socket 0[core 7[hwt 1]]: [../../../../../../../.B/../../../../../..][../../../../../../../../../../../../../..]
0x00000008,0x0
0x00000080
0x00000001
0x10000000

$$HOME/prj/ii-ompi/build/install/bin/oshrun --report-bindings -n 4 --oversubscribe --cpu-set 14,42,0,28 --bind-to hwthread --map-by hwthread bash -c "hwloc-bind --get"
[clx-orion-065.dmz.e2e.mlnx:16043] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:16043] MCW rank 1 bound to socket 0[core 7[hwt 0]]: [../../../../../../../B./../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:16043] MCW rank 2 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:16043] MCW rank 3 bound to socket 1[core 21[hwt 0]]: [../../../../../../../../../../../../../..][../../../../../../../B./../../../../../..]
0x00000080
0x00000001
0x00200000
0x00004000

• w/ your change (using --map-by hwthread)

$$HOME/prj/ii-ompi/build/install/bin/oshrun --report-bindings -n 4 --oversubscribe --cpu-set 14,15,0,1 --bind-to hwthread --map-by hwthread bash -c "cat /proc/self/status |grep Cpus_allowed_list"
Cpus_allowed_list:      0
[clx-orion-065.dmz.e2e.mlnx:14338] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14338] MCW rank 1 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14338] MCW rank 2 bound to socket 0[core 7[hwt 0]]: [../../../../../../../B./../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14338] MCW rank 3 bound to socket 0[core 7[hwt 1]]: [../../../../../../../.B/../../../../../..][../../../../../../../../../../../../../..]
Cpus_allowed_list:      28
Cpus_allowed_list:      7
Cpus_allowed_list:      35

$$HOME/prj/ii-ompi/build/install/bin/oshrun --report-bindings -n 4 --oversubscribe --cpu-set 14,42,0,28 --bind-to hwthread --map-by hwthread bash -c "cat /proc/self/status |grep Cpus_allowed_list"
[clx-orion-065.dmz.e2e.mlnx:14357] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14357] MCW rank 1 bound to socket 0[core 7[hwt 0]]: [../../../../../../../B./../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14357] MCW rank 2 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14357] MCW rank 3 bound to socket 1[core 21[hwt 0]]: [../../../../../../../../../../../../../..][../../../../../../../B./../../../../../..]
Cpus_allowed_list:      0
Cpus_allowed_list:      7
Cpus_allowed_list:      14
Cpus_allowed_list:      21

$$HOME/prj/ii-ompi/build/install/bin/oshrun --report-bindings -n 4 --oversubscribe --cpu-set 14,15,0,1 --bind-to hwthread --map-by hwthread bash -c "hwloc-bind --get"
[clx-orion-065.dmz.e2e.mlnx:14374] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14374] MCW rank 1 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14374] MCW rank 2 bound to socket 0[core 7[hwt 0]]: [../../../../../../../B./../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14374] MCW rank 3 bound to socket 0[core 7[hwt 1]]: [../../../../../../../.B/../../../../../..][../../../../../../../../../../../../../..]
0x00000008,0x0
0x00000080
0x00000001
0x10000000

$$HOME/prj/ii-ompi/build/install/bin/oshrun --report-bindings -n 4 --oversubscribe --cpu-set 14,42,0,28 --bind-to hwthread --map-by hwthread bash -c "hwloc-bind --get"
[clx-orion-065.dmz.e2e.mlnx:14385] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14385] MCW rank 1 bound to socket 0[core 7[hwt 0]]: [../../../../../../../B./../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14385] MCW rank 2 bound to socket 1[core 14[hwt 0]]: [../../../../../../../../../../../../../..][B./../../../../../../../../../../../../..]
[clx-orion-065.dmz.e2e.mlnx:14385] MCW rank 3 bound to socket 1[core 21[hwt 0]]: [../../../../../../../../../../../../../..][../../../../../../../B./../../../../../..]
0x00000001
0x00000080
0x00004000
0x00200000

After your change --cpu-set x,y,z does mapping to PU L#x,yz in hwloc-ls notation
Example

--cpu-set 14,42,0,28

socket:core:hwt         hwloc          numactl (/proc)
0        :0     :0            PU L#0               0
0        :7     :0            PU L#14             7
1        :14   :0            PU L#28             14
1        :21   :0            PU L#42             21

--cpu-set 14,15,0,1

socket:core:hwt         hwloc          numactl (/proc)
0        :0     :0            PU L#0               0
0        :0     :1            PU L#1               28
1        :7     :0            PU L#14             7
1        :7     :1            PU L#15             35

[rhc54]

I'm not entirely sure I'm understanding this discussion, but I'll play with things this week now that I'm back and have access to my cluster again. One thing I will note: we do not deal with physical processor IDs at all any more in OMPI. We abandoned that a few years ago as the logical vs physical numbering was just confusing the heck out of users.

So OMPI now only deals with LOGICAL processor IDs. Please be sure that you are specifying logical IDs, and that your code is printing out logical IDs when checking for correctness.

[ggouaillardet]

@rhc54 i think we should first clearly define what is a logical cpu.
if there is no hyper threading, then it is a core.
but what if hyper threading is enabled ?
is it a core ? a hwthread ? a core unless --map-by hwthread is used ?

[rhc54]

Wow - we've been thru that discussion years ago. It is a core unless you specify --use-hwthread-cpus. We treat hwthreads as cpus ONLY if you specify that flag. Otherwise, you can map-by hwthread, bind-to hwthread, etc. - and we will still consider a core as being the cpu.

Only place that changes is on those systems that do not report cores - in that case, we use PU as the cpu, and it is up to the system to define what a "PU" means.

[igor-ivanov]

Sorry, I am tangled a bit so let me rephrase my question:
I can get information about cores, sockets, hwt etc for a node with enabled hyperthreading using hwloc tools.
Question:
How can I use/interpret this information (find ompi logical id) to set right --cpu-set values and get expected mapping.
For example: after @ggouaillardet change I see obvious link between hwloc`s PU L#z and --cpu-set=z (I do not know if this change correct and I am using one just to demonstrate)
But I do not see any clear link using current ompi.

$./install/bin/oshrun --report-bindings -n 2 --oversubscribe --cpu-set 17,29 --bind-to hwthread --map-by hwthread -mca hwloc_base_binding_policy hwthread -mca hwloc_base_use_hwthreads_as_cpus 1 bash -c "cat /proc/self/status |grep Cpus_allowed_list"
[clx-orion-065:18679] MCW rank 1 bound to socket 1[core 17[hwt 1]]: [../../../../../../../../../../../../../..][../../../.B/../../../../../../../../../..]
[clx-orion-065:18679] MCW rank 0 bound to socket 1[core 17[hwt 0]]: [../../../../../../../../../../../../../..][../../../B./../../../../../../../../../..]
Cpus_allowed_list:      17
Cpus_allowed_list:      45

where node information:

  NUMANode L#1 (P#1 128GB)
    Socket L#1 + L3 L#1 (35MB)
      L2 L#14 (256KB) + L1d L#14 (32KB) + Core L#14
        PU L#28 (P#14)
        PU L#29 (P#42)
      L2 L#15 (256KB) + L1d L#15 (32KB) + Core L#15
        PU L#30 (P#15)
        PU L#31 (P#43)
      L2 L#16 (256KB) + L1d L#16 (32KB) + Core L#16
        PU L#32 (P#16)
        PU L#33 (P#44)
      L2 L#17 (256KB) + L1d L#17 (32KB) + Core L#17
        PU L#34 (P#17)
        PU L#35 (P#45)

@rhc54 if there is clear method to get values for --cpu-set using information from hwloc?

[igor-ivanov]

@rhc54 sorry for bothering but may be you missed my question during holidays. Could you share a way I can set right logical values in --cpu-set option using hwloc information or may be other tools in case hyperthreading node.

[rhc54]

Sorry for delayed response - this did indeed slip thru the cracks. The id's you want are the numbers marked with an "L" in the hwloc output. However, it appears there is an error in the code base when using ht's as cpus, but I'm not sure if the fix from @ggouaillardet is correct, and I don't have time right now to address it.

[igor-ivanov]

@rhc54 I see, thank you.

[ggouaillardet]

@igor-ivanov @rhc54 i think this patch is correct (and replaces the previous one)

diff --git a/opal/mca/hwloc/base/hwloc_base_util.c b/opal/mca/hwloc/base/hwloc_base_util.c
index 6a030d5..cf95089 100644
--- a/opal/mca/hwloc/base/hwloc_base_util.c
+++ b/opal/mca/hwloc/base/hwloc_base_util.c
@@ -14,7 +14,7 @@
  * Copyright (c) 2012-2015 Los Alamos National Security, LLC.
  *                         All rights reserved.
  * Copyright (c) 2013-2014 Intel, Inc. All rights reserved.
- * Copyright (c) 2015      Research Organization for Information Science
+ * Copyright (c) 2015-2016 Research Organization for Information Science
  *                         and Technology (RIST). All rights reserved.
  * $COPYRIGHT$
  *
@@ -71,7 +71,7 @@ hwloc_obj_t opal_hwloc_base_get_pu(hwloc_topology_t topo,
        So first we have to see if we can find *any* cores by looking
        for the 0th core.  If we find it, then try to find the Nth
        core.  Otherwise, try to find the Nth PU. */
-    if (NULL == hwloc_get_obj_by_type(topo, HWLOC_OBJ_CORE, 0)) {
+    if (opal_hwloc_use_hwthreads_as_cpus || (NULL == hwloc_get_obj_by_type(topo, HWLOC_OBJ_CORE, 0))) {
         obj_type = HWLOC_OBJ_PU;
     }

and here is my mpirun command line

mpirun  --bind-to hwthread --use-hwthread-cpus ...

so far, i could not find a simple way to query the logical PU(s) (aka thread is this context) a process is bound to.

it seems both
hwloc-bind --get --logical and hwloc-bind --get --physical return the physical PU(s)

@bgoglin is this the expected behavior of hwloc-bind ? is there any way to query the logical PU ?