MPI error building libmesh #26538

ke7kto · 2024-01-11T20:06:19Z

ke7kto
Jan 11, 2024

This question is related to discussion #18008. Building libmesh I get the following error:

Source/moose/petsc/include/petscsys.h:279:25: error: invalid conversion from 'MPI_Comm' {aka 'ompi_communicator_t*'} to 'TIMPI::communicator' {aka 'int'} [-fpermissive]

I have openmpi-4.1.6-1 installed through my system package manager, which also installed VTK (HDF5 is also installed via system package manager, but petsc requires a static library and my distribution only supplies dynamic libs).

I can successfully build libmesh with the --download-mpich option, but then building moose fails, as moose still builds with the system openmpi. I'm happy to try to hide the system environment and only pull in the necessary components, but I'm not sure how to do that in the build process.

The output of printenv is

SHELL=/bin/bash
WINDOWID=50331658
COLORTERM=truecolor
XDG_CONFIG_DIRS=/home/adriaan/.config/kdedefaults:/etc/xdg
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
LANGUAGE=
SHELL_SESSION_ID=76fb94c65d39499b912cd4cd0e332820
MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
DESKTOP_SESSION=plasma
GTK_RC_FILES=/etc/gtk/gtkrc:/home/adriaan/.gtkrc:/home/adriaan/.config/gtkrc
XCURSOR_SIZE=24
GTK_MODULES=canberra-gtk-module
XDG_SEAT=seat0
PWD=/home/adriaan/Source/moose/scripts
XDG_SESSION_DESKTOP=KDE
LOGNAME=adriaan
XDG_SESSION_TYPE=x11
SYSTEMD_EXEC_PID=829
XAUTHORITY=/tmp/xauth_mpBLKW
MOTD_SHOWN=pam
GTK2_RC_FILES=/etc/gtk-2.0/gtkrc:/home/adriaan/.gtkrc-2.0:/home/adriaan/.config/gtkrc-2.0
HOME=/home/adriaan
LANG=C.UTF-8
NVCC_PREPEND_FLAGS=-ccbin /opt/cuda/bin
XDG_CURRENT_DESKTOP=KDE
KONSOLE_DBUS_SERVICE=:1.59
MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/background.slice/plasma-kglobalaccel.service/memory.pressure
KONSOLE_DBUS_SESSION=/Sessions/1
PROFILEHOME=
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
INVOCATION_ID=2d879f8aa9fa4f55a7514fd672c41abe
KONSOLE_VERSION=230804
MANAGERPID=717
KDE_SESSION_UID=1000
XDG_SESSION_CLASS=user
TERM=xterm-256color
USER=adriaan
CUDA_PATH=/opt/cuda
COLORFGBG=15;0
KDE_SESSION_VERSION=5
DISPLAY=:0
SHLVL=1
XDG_VTNR=2
XDG_SESSION_ID=1
XDG_RUNTIME_DIR=/run/user/1000
MKLROOT=/opt/intel/oneapi/mkl/latest
DEBUGINFOD_URLS=https://debuginfod.archlinux.org
QT_AUTO_SCREEN_SCALE_FACTOR=0
JOURNAL_STREAM=8:8707
XCURSOR_THEME=breeze_cursors
KDE_FULL_SESSION=true
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/apache-spark/bin:/opt/apache-spark/sbin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/usr/lib/jvm/default/bin:/usr/local/bin:/home/adriaan/.local/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
KDE_APPLICATIONS_AS_SCOPE=1
MAIL=/var/spool/mail/adriaan
KONSOLE_DBUS_WINDOW=/Windows/1
OLDPWD=/home/adriaan/Source/moose

This is me playing around at home and partially to keep track of my thoughts, not an urgent support request.

Answered by ke7kto

Jan 17, 2024

Got the system OpenMPI to compile! The answer was in the output from the libmesh build script:

note: Checking /lib and /include for MPI
note: testing /lib/libmpi(.a/.so/.dylib)
checking for lam_show_version in -llam... no
checking for MPI_Init in -lmpi... yes
Found valid MPI installation...
Could not find MPI header <mpi.h>...

My mpi.h is at /usr/include/mpi.h, which was not checked. Simply adding --with-mpi-include=/usr/include let moose compile!

View full answer

milljm · 2024-01-11T20:53:43Z

milljm
Jan 11, 2024
Maintainer

Can you run the diagnostic script and report those results?

cd /home/adriaan/Source/moose/scripts
./diagnostics.sh

17 replies

milljm Jan 15, 2024
Maintainer

I suspect MOOSE is being asked to link to libraries among the system path. And in so doing, it is linking to the system OpenMPI library still.

Can you run the following script, and report the results?

/home/adriaan/Source/moose/libmesh/installed/bin/libmesh-config --libs

ke7kto Jan 16, 2024
Author

Did you git clean moose before recompiling?

Yes.

Can you run the following script, and report the results?

adriaan@Fermat:~/Source/moose/scripts$ /home/adriaan/Source/moose/libmesh/installed/bin/libmesh-config --libs
-Wl,-rpath,/home/adriaan/Source/moose/scripts/../libmesh/installed/lib 
-L/home/adriaan/Source/moose/scripts/../libmesh/installed/lib -lmesh_opt 
-ltimpi_opt -Wl,-rpath,/home/adriaan/Source/moose/petsc/arch-moose/lib 
-L/home/adriaan/Source/moose/petsc/arch-moose/lib -L/lib -Wl,-rpath,/lib 
-Wl,-rpath,/usr/lib -L/usr/lib -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1 
-L/usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1 -lvtkIOCore -lvtkCommonCore -lvtkCommonDataModel 
-lvtkFiltersCore -lvtkIOXML -lvtkImagingCore -lvtkIOImage -lvtkImagingMath -lvtkIOParallelXML 
-lvtkParallelMPI -lvtkParallelCore -lvtkCommonExecutionModel -lvtksys -lz -lslepc -lpetsc -lHYPRE 
-ldmumps -lmumps_common -lpord -lpthread -lstrumpack -lscalapack -lsuperlu_dist -lflapack 
-lfblas -lptesmumps -lptscotchparmetisv3 -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr 
-lparmetis -lmetis -lhdf5_hl -lhdf5 -lX11 -lmpifort -lmpi -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lrt -ltirpc

If I understand correctly, the system VTK is being linked, which uses the system openmpi.

I'll try building and linking VTK using PETSC's mpich tomorrow. Are there other libs I should be watching out for?

milljm Jan 16, 2024
Maintainer

I believe this will most likely be the culprit:

-L/lib -Wl,
-rpath,/lib
-Wl,-rpath,/usr/lib
-L/usr/lib
-Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1 
-L/usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1 

<trimmed>

-lmpifort -lmpi -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lrt -ltirpc

With the above -L pointing to those directories, and then later a -lmpi, it will link to system MPI libraries.

I think we need to figure out why System OpenMPI does not work as your MPI wrapper/stack...

ke7kto Jan 17, 2024
Author

Got the system OpenMPI to compile! The answer was in the output from the libmesh build script:

note: Checking /lib and /include for MPI
note: testing /lib/libmpi(.a/.so/.dylib)
checking for lam_show_version in -llam... no
checking for MPI_Init in -lmpi... yes
Found valid MPI installation...
Could not find MPI header <mpi.h>...

My mpi.h is at /usr/include/mpi.h, which was not checked. Simply adding --with-mpi-include=/usr/include let moose compile!

Answer selected by ke7kto

milljm Jan 18, 2024
Maintainer

Pinging @roystgnr because I think /usr/include is a standard search path. But I could be wrong!

Glad you got it working!

roystgnr Jan 22, 2024
Collaborator

/usr/include is a standard search path to essentially every Unix compiler (albeit not to the C++ standard, which has to support Windows too). Hence the trouble with trying to use an MPI elsewhere while there's an MPI there.

But we're not failing an AC_CHECK_HEADER (which uses the compiler to make sure a header exists and is usable) if we see that error message ... we're looking at the return value of test -e $MPI_INCLUDES/mpi.h, and test -e $MPI_INCLUDES/mpi/mpi.h. It looks like, if we don't have any $MPI environment variable (and haven't seen an MPI wrapper, and haven't seen explicit paths in configure arguments...) then $MPI_INCLUDES defaults to /include rather than /usr/include, because we just set it to $MPI/include.

That should be easy enough to fix; I've just never seen a need to fix it before. I loathe the "compiler wrapper" method for adding environment to a compile, and yet here we are, with the first MPI installation I've ever seen that doesn't include the wrappers, and instead of cheering libMesh doesn't recognize it out of the box. That's embarrassing.

roystgnr Jan 23, 2024
Collaborator

Any chance you could try configuring and building https://github.com/roystgnr/TIMPI/tree/usr_mpi and see if it works for you, specifying $PETSC_ARCH and $PETSC_DIR but not specifying --with-mpi-include? I'm having trouble setting up a decent environment to replicate the problem, but I also don't want to push a patch downstream without first knowing it's an improvement.

ke7kto Jan 23, 2024
Author

Sure. By the way, the system is Arch Linux.

adriaan@Fermat:~/Source/TIMPI$ ./configure
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether UID '1000' is supported by ustar format... yes
checking whether GID '1000' is supported by ustar format... yes
checking how to create a ustar tar archive... gnutar
checking whether make supports nested variables... (cached) yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for a sed that does not truncate output... /usr/bin/sed
checking Major version... 1
checking Minor version... 8
checking Point version... 5
note: MPI library path not given...
note: MPI library path not given...
trying prefix=/usr
checking whether make supports the include directive... yes (GNU style)
checking for mpicxx... mpicxx
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C++... yes
checking whether /usr/bin/mpicxx accepts -g... yes
checking for /usr/bin/mpicxx option to enable C++11 features... none needed
checking dependency style of /usr/bin/mpicxx... gcc3
<<< C++ compiler is unknown but accepted gcc version >>>
configure: Seeking a C++ standard between "2014" and "2017"
checking whether /usr/bin/mpicxx supports C++17 features by default... yes
configure: Found C++17 standard support
checking whether /usr/bin/mpicxx supports C++14 features by default... yes
configure: Found C++14 standard support
checking whether /usr/bin/mpicxx supports C++11 features by default... yes
configure: Found C++11 standard support
configure: Using support for C++17 standard
<<< C++ compiler is unknown but accepted gcc version >>>
checking for gcc... gcc
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
<<< Adding -ftrapping-math for FPE safety >>>
checking for gfortran... gfortran
checking whether the compiler supports GNU Fortran... yes
checking whether gfortran accepts -g... yes
No build methods specified, defaulting to "dbg devel opt"
<<< Configuring with methods "dbg devel opt" >>>
<<< Disabling extra paranoid compiler warnings >>>
<<< Compiler warnings are just warnings >>>
checking for stdio.h... no
checking for stdlib.h... no
checking for string.h... no
checking for inttypes.h... no
checking for stdint.h... no
checking for strings.h... no
checking for sys/stat.h... no
checking for sys/types.h... no
checking for unistd.h... no
---------------------------------------------
------- Configuring optional features -------
---------------------------------------------
<<< Configuring library with deprecated code support >>>
<<< Configuring library with exception throwing support >>>
<<< Default floating point is double precision (double) >>>
<<< Configuring library with compile timestamps >>>
<<< Configuring library with warnings >>>
configuring size of processor_id... 4
---------------------------------------------
----- Done configuring optional features ----
---------------------------------------------
checking how to print strings... printf
checking for a sed that does not truncate output... (cached) /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for file... file
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/bin/dd
checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... no
checking how to run the C++ preprocessor... /usr/bin/mpicxx -E
checking for ld used by /usr/bin/mpicxx... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the /usr/bin/mpicxx linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for /usr/bin/mpicxx option to produce PIC... -fPIC -DPIC
checking if /usr/bin/mpicxx PIC flag -fPIC -DPIC works... yes
checking if /usr/bin/mpicxx static flag -static works... no
checking if /usr/bin/mpicxx supports -c -o file.o... yes
checking if /usr/bin/mpicxx supports -c -o file.o... (cached) yes
checking whether the /usr/bin/mpicxx linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... no
checking for gfortran option to produce PIC... -fPIC
checking if gfortran PIC flag -fPIC works... yes
checking if gfortran static flag -static works... no
checking if gfortran supports -c -o file.o... yes
checking if gfortran supports -c -o file.o... (cached) yes
checking whether the gfortran linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking for git... /usr/bin/git
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating src/utilities/include/timpi/timpi_version.h
config.status: creating Makefile
config.status: creating timpi.pc
config.status: creating src/Makefile
config.status: creating test/Makefile
config.status: creating bin/timpi-config
config.status: creating test/run_unit_tests.sh
config.status: creating src/utilities/include/timpi/timpi_config.h.tmp
config.status: src/utilities/include/timpi/timpi_config.h.tmp is unchanged
config.status: executing src/utilities/include/timpi/timpi_config.h commands
config.status: creating src/utilities/include/timpi/timpi_config.h - prefix TIMPI for src/utilities/include/timpi/timpi_config.h.tmp defines
config.status: src/utilities/include/timpi/timpi_config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands

----------------------------------- SUMMARY -----------------------------------

Package version............... : timpi-1.8.5

C++ compiler.................. : /usr/bin/mpicxx
Build Methods...................... : dbg devel opt
CPPFLAGS...(dbg)................... : -DDEBUG  
CXXFLAGS...(dbg)................... :  -O0 -felide-constructors -g -pedantic -W -Wall -Wextra -Wno-long-long -Wunused -Wpointer-arith -Wformat -Wparentheses -Woverloaded-virtual -ftrapping-math    

CPPFLAGS...(devel)................. :   
CXXFLAGS...(devel)................. :  -O2 -felide-constructors -g -pedantic -W -Wall -Wextra -Wno-long-long -Wunused -Wpointer-arith -Wformat -Wparentheses -Wuninitialized -fstrict-aliasing -Woverloaded-virtual -Wdisabled-optimization -funroll-loops -ftrapping-math    

CPPFLAGS...(opt)................... : -DNDEBUG  
CXXFLAGS...(opt)................... :  -O2 -felide-constructors -fstrict-aliasing -Wdisabled-optimization -funroll-loops -ftrapping-math    

Any warnings-to-errors flags.. : 
Any paranoid warning flags.... : 
Install dir................... : /usr/local
Build user.................... : adriaan
Build host.................... : Fermat
Configure date................ : 2024-01-22 20:20
Build architecture............ : x86_64-pc-linux-gnu
Git revision number........... : 2c552dbe4650cdd8dc2af0a0339f0456e4bb374f

-------------------------------------------------------------------------------
Optional Packages for Testing:
  MPI......................... : yes
  MPI_IMPL.................... : mpi-wrapper-built-in

Configure complete, now type \'make\' and then \'make install\'.

adriaan@Fermat:~/Source/TIMPI$ make
Making all in src
make[1]: Entering directory '/home/adriaan/Source/TIMPI/src'
  CXX      apps/version.o
  CXX      utilities/src/libtimpi_dbg_la-timpi_version.lo
  CXXLD    libtimpi_dbg.la
  CXXLD    timpi_version-dbg
libtool: warning: '-version-info' is ignored for programs
  CXX      utilities/src/libtimpi_devel_la-timpi_version.lo
  CXXLD    libtimpi_devel.la
  CXXLD    timpi_version-devel
libtool: warning: '-version-info' is ignored for programs
  CXX      utilities/src/libtimpi_opt_la-timpi_version.lo
  CXXLD    libtimpi_opt.la
  CXXLD    timpi_version-opt
libtool: warning: '-version-info' is ignored for programs
make[1]: Leaving directory '/home/adriaan/Source/TIMPI/src'
Making all in test
make[1]: Entering directory '/home/adriaan/Source/TIMPI/test'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/adriaan/Source/TIMPI/test'
make[1]: Entering directory '/home/adriaan/Source/TIMPI'
make[1]: Nothing to be done for 'all-am'.
make[1]: Leaving directory '/home/adriaan/Source/TIMPI'

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MPI error building libmesh #26538

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 17 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

MPI error building libmesh #26538

ke7kto Jan 11, 2024

Replies: 1 comment · 17 replies

milljm Jan 11, 2024 Maintainer

milljm Jan 15, 2024 Maintainer

ke7kto Jan 16, 2024 Author

milljm Jan 16, 2024 Maintainer

ke7kto Jan 17, 2024 Author

milljm Jan 18, 2024 Maintainer

roystgnr Jan 22, 2024 Collaborator

roystgnr Jan 23, 2024 Collaborator

ke7kto Jan 23, 2024 Author

ke7kto
Jan 11, 2024

Replies: 1 comment 17 replies

milljm
Jan 11, 2024
Maintainer

milljm Jan 15, 2024
Maintainer

ke7kto Jan 16, 2024
Author

milljm Jan 16, 2024
Maintainer

ke7kto Jan 17, 2024
Author

milljm Jan 18, 2024
Maintainer

roystgnr Jan 22, 2024
Collaborator

roystgnr Jan 23, 2024
Collaborator

ke7kto Jan 23, 2024
Author