rework argobots configury to be smarter

[hppritcha]

fixes #7578

Signed-off-by: Howard Pritchard howardp@lanl.gov

[hppritcha]

@shintaro-iwasaki please try on your system(s)

[shintaro-iwasaki]

@hppritcha Thank you very much! This patch perfectly worked on our two servers; the configuration option, with-argobots=$ARGOBOTS_PATH, automatically links MPI binaries (e.g., mpiexec and mpicc) and programs compiled by MPI compilers (e.g., mpicc) to Argobots.

[devreal]

I tried building the branch, it worked on one machine but failed on my local laptop:

  CCLD     ompi_info
/usr/bin/ld: ompi_info.o: undefined reference to symbol 'ABT_mutex_lock'
//home/joseph/opt/argobots-1.0rc2/lib/libabt.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

Configure line was:

../configure --with-ucx=$HOME/opt/ucx-1.8.0 --prefix=$HOME/opt/openmpi-master-abt/ --enable-grequest-extensions --disable-mpi-fortran --with-threads=argobots --with-argobots=$HOME/opt/argobots-1.0rc2

Not sure why it fails here while it succeeded elsewhere but it seems that libmpi.so links to libabt.so but that library is not passed to the linker:

make[2]: Entering directory '$HOME/src/openmpi/git/hppritcha/build/ompi/tools/ompi_info'
/bin/bash ../../../libtool  --tag=CC   --mode=link gcc -DOPAL_CONFIGURE_USER="\"joseph\"" -DOPAL_CONFIGURE_HOST="\"beryl\"" -DOPAL_CONFIGURE_DATE="\"Tue May  5 12:20:46 CEST 2020\"" -DOMPI_BUILD_USER="\"$USER\"" -DOMPI_BUILD_HOST="\"${HOSTNAME:-`(hostname || uname -n) 2> /dev/null | sed 1q`}\"" -DOMPI_BUILD_DATE="\"`date`\"" -DOMPI_BUILD_CFLAGS="\"-O3 -DNDEBUG -finline-functions -fno-strict-aliasing -mcx16\"" -DOMPI_BUILD_CPPFLAGS="\"-iquote../../../.. -iquote../../.. -iquote../../../../opal/include -iquote../../../../ompi/include -iquote../../../../oshmem/include -I$HOME/src/openmpi/git/hppritcha/build/opal/mca/pmix/pmix4x/openpmix/include -I$HOME/src/openmpi/git/hppritcha/opal/mca/pmix/pmix4x/openpmix/include -I$HOME/src/openmpi/git/hppritcha/build/opal/mca/event/libevent2022/libevent/include -I$HOME/src/openmpi/git/hppritcha/opal/mca/event/libevent2022/libevent -I$HOME/src/openmpi/git/hppritcha/opal/mca/event/libevent2022/libevent/include -I$HOME/src/openmpi/git/hppritcha/build/opal/mca/hwloc/hwloc2/hwloc/include -I$HOME/src/openmpi/git/hppritcha/opal/mca/hwloc/hwloc2/hwloc/include  -I/usr/local/include -I/usr/local/include\"" -DOMPI_BUILD_CXXFLAGS="\"-DNDEBUG \"" -DOMPI_BUILD_CXXCPPFLAGS="\"@CXXCPPFLAGS@\"" -DOMPI_BUILD_FFLAGS="\"\"" -DOMPI_BUILD_FCFLAGS="\"\"" -DOMPI_BUILD_LDFLAGS="\"\"" -DOMPI_BUILD_LIBS="\"-lrt -lm -lutil \"" -DOPAL_CC_ABSOLUTE="\"/usr/bin/gcc\"" -DOMPI_CXX_ABSOLUTE="\"/usr/bin/g++\"" -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -mcx16   -o ompi_info ompi_info.o param.o ../../../ompi/libmpi.la ../../../opal/libopen-pal.la -lrt -lm -lutil 
libtool: link: gcc "-DOPAL_CONFIGURE_DATE=\"Tue May  5 12:20:46 CEST 2020\"" -DOMPI_BUILD_USER=\"joseph\" -DOMPI_BUILD_HOST=\"beryl\" "-DOMPI_BUILD_DATE=\"Tue May  5 14:10:23 CEST 2020\"" "-DOMPI_BUILD_CFLAGS=\"-O3 -DNDEBUG -finline-functions -fno-strict-aliasing -mcx16\"" "-DOMPI_BUILD_CPPFLAGS=\"-iquote../../../.. -iquote../../.. -iquote../../../../opal/include -iquote../../../../ompi/include -iquote../../../../oshmem/include -I$HOME/src/openmpi/git/hppritcha/build/opal/mca/pmix/pmix4x/openpmix/include -I$HOME/src/openmpi/git/hppritcha/opal/mca/pmix/pmix4x/openpmix/include -I$HOME/src/openmpi/git/hppritcha/build/opal/mca/event/libevent2022/libevent/include -I$HOME/src/openmpi/git/hppritcha/opal/mca/event/libevent2022/libevent -I$HOME/src/openmpi/git/hppritcha/opal/mca/event/libevent2022/libevent/include -I$HOME/src/openmpi/git/hppritcha/build/opal/mca/hwloc/hwloc2/hwloc/include -I$HOME/src/openmpi/git/hppritcha/opal/mca/hwloc/hwloc2/hwloc/include  -I/usr/local/include -I/usr/local/include\"" "-DOMPI_BUILD_CXXFLAGS=\"-DNDEBUG \"" -DOMPI_BUILD_CXXCPPFLAGS=\"@CXXCPPFLAGS@\" -DOMPI_BUILD_FFLAGS=\"\" -DOMPI_BUILD_FCFLAGS=\"\" -DOMPI_BUILD_LDFLAGS=\"\" "-DOMPI_BUILD_LIBS=\"-lrt -lm -lutil \"" -DOPAL_CC_ABSOLUTE=\"/usr/bin/gcc\" -DOMPI_CXX_ABSOLUTE=\"/usr/bin/g++\" -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -mcx16 -o .libs/ompi_info ompi_info.o param.o  ../../../ompi/.libs/libmpi.so ../../../opal/.libs/libopen-pal.so -lrt -lm -lutil -pthread -Wl,-rpath -Wl,$HOME/opt/openmpi-master-abt/lib
/usr/bin/ld: ompi_info.o: undefined reference to symbol 'ABT_mutex_lock'
/$HOME/opt/argobots-1.0rc2/lib/libabt.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
Makefile:1514: recipe for target 'ompi_info' failed

The dependencies of libmpi.so include libabt.so.0 but that is not present on the linker line above:

$ ldd ompi/.libs/libmpi.so
	linux-vdso.so.1 (0x00007ffe28cf0000)
[...]
	libabt.so.0 => $HOME/opt/argobots-1.0rc2/lib/libabt.so.0 (0x00007f3845ff3000)
[...]

[devreal]

Update: I can pass LDFLAGS="-L$HOME/opt/argobots-1.0rc2/lib -labt" to configure as a workaround to successfully compile this branch on my local system.

[shintaro-iwasaki]

@devreal Thanks! I could reproduce the same issue on my Ubuntu 16 laptop.

In my case, link_all_deplibs (a flag in libtool) on my CentOS 7.6 server is set to unknown while no on my local laptop (Ubuntu 16.04).

## $(ompi_root_dir)/libtool
# Whether libtool must link a program against all its dependency libraries.
link_all_deplibs=no

When this is set to no, libtool does not expand libopen-pal.la in linking, and thus libabt.la added to libopen-pal.la is ignored. If I manually rewrite this value to unknown (or probably anything other than no), it works on my laptop.

I am not sure if this is caused by changes in libtool's default behavior. (2.4.2 on CentOS vs 2.4.6 on my laptop, so it is unlikely.) In any case, I need to understand why it happens and will find a workaround.

[hppritcha]

thanks looks like I need to add some wrapper flags to the configury

[shintaro-iwasaki]

@hppritcha Thanks! This is an update, but it seems a Debian/Ubuntu's libtool issue. The discussion in https://lists.gnu.org/archive/html/libtool/2007-09/msg00012.html was in 2007 but this behavior still continues as far as I see. link_all_deplibs is by default set to no on these machines.

[hppritcha]

@Shinji-Sumimoto try this PR out when you get a chance. I tested on ubuntu both with install of argo into a non-default location and into /usr/local and worked for me.

[shintaro-iwasaki]

@hppritcha Thank you very much! It worked on my Ubuntu laptop (Ubuntu 16.04 LTS) as well as a CentOS server (CentOS 7.6).

[hppritcha]

@tkordenbrock could you review?

[tkordenbrock]

@hppritcha I can review this afternoon.