openib error: "ibv_exp_query_device: invalid comp_mask" -- reported by multiple users

[jsquyres]

There have been multiple reports of the openib BTL reporting variations this error:

ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x27800000002 valid_mask = 0x1)"

I know that openib is on its way out the door, but it's still supported in v2.x and 3.x. Is there a quick/easy fix for this issue?

I am unable to reproduce the issue with ConnectX 5's on Ethernet and the inbox verbs drivers on RHEL 7.2. Is there something that has changed in upstream OFED and/or MOFED that is causing this issue?

#5810 is the most recent activity where this came up, but it has come up in other issues, too (and possibly on mailing lists...?).

[hppritcha]

We are seeing this at LANL, esp. on our ARM machines when using many processes (64 or more)
per node. We see this using UCX as well.

[yosefe]

Seems like it could happen if openib BTL, or an older version of UCX, is compiled with MLNX_OFED 4.4.
@hppritcha it should work with UCX v1.4.x or UCX master

[abeltre1]

Seems like it could happen if openib BTL, or an older version of UCX, is compiled with MLNX_OFED 4.4.
@hppritcha it should work with UCX v1.4.x or UCX master

Can you point out the source code of UCX to include it in the MLNX_OFED 4.4 installation?

[yosefe]

@abeltre1 not sure i understand the question, anyway - UCX binary is included in MLNX_OFED 4.4 installation, source code is in https://github.com/openucx/ucx, and the particular code which avoids the error from ibv_exp_query_device() is here

[hppritcha]

I did some more checking on one of our clusters and actually do not see the issue when using
UCX. Our setup is MOFED 4.4 and UCX shipped as part of that release -

/home/hpp/ompi/examples@cn805:~/ompi/examples> (v4.0.x)ofed_info
MLNX_OFED_LINUX-4.4-2.0.7.0 (OFED-4.4-2.0.7):

/home/hpp/ompi/examples@cn805:~/ompi/examples> (v4.0.x)ucx_info -v
# UCT version=1.4.0 revision 739b569

I do see this with openib BTL initialization even though its not even being used:

/home/hpp/ompi/examples@cn805:~/srun -n 2 -N 2 ./ring_c
ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x4000031ba900 valid_mask = 0x1)
[cn805][[5425,6],0][btl_openib_component.c:1698:init_one_device] error obtaining device attributes for mlx5_1 errno says Invalid argument
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   cn805
  Local device: mlx5_1
--------------------------------------------------------------------------
ibv_exp_query_device: invalid comp_mask !!! (comp_mask = 0x27ddfac0 valid_mask = 0x1)
[cn806][[5425,6],1][btl_openib_component.c:1698:init_one_device] error obtaining device attributes for mlx5_1 errno says Invalid argument
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   cn806
  Local device: mlx5_1