Undefined references when building jigsaw-lib on Perlmutter

[mcarlson801]

Albany's weekly Perlmutter compass build failed this week with the following errors:

/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `aligned_alloc@GLIBC_2.16'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `memcpy@GLIBC_2.14'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `__cxa_thread_atexit_impl@GLIBC_2.18'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `clock_gettime@GLIBC_2.17'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/dev_compass_1.4.0-alpha.2/lib/././libicuuc.so.73: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
gmake[2]: *** [src/CMakeFiles/marche-cmd.dir/build.make:100: src/marche] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:156: src/CMakeFiles/marche-cmd.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `aligned_alloc@GLIBC_2.16'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `memcpy@GLIBC_2.14'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `__cxa_thread_atexit_impl@GLIBC_2.18'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `clock_gettime@GLIBC_2.17'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/dev_compass_1.4.0-alpha.2/lib/././libicuuc.so.73: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
gmake[2]: *** [src/CMakeFiles/tripod-cmd.dir/build.make:100: src/tripod] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:130: src/CMakeFiles/tripod-cmd.dir/all] Error 2
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `aligned_alloc@GLIBC_2.16'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `memcpy@GLIBC_2.14'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `__cxa_thread_atexit_impl@GLIBC_2.18'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `clock_gettime@GLIBC_2.17'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/dev_compass_1.4.0-alpha.2/lib/././libicuuc.so.73: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
gmake[2]: *** [src/CMakeFiles/jigsaw-cmd.dir/build.make:100: src/jigsaw] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:104: src/CMakeFiles/jigsaw-cmd.dir/all] Error 2

Any idea what might be causing this? It looks like there was maybe a similar issue back in January.

[xylar]

@mcarlson801, thanks for reporting this.

I've seen something similar on the OpenMPI feedstock on conda-forge. Based on this comment:
conda-forge/openmpi-feedstock#143 (comment)
we might need to install sysroot_linux-64=2.17 explicitly or based on later comments, we might need to make sure $LDFLAGS are getting passed to the linker in the JIGSAW build. I probably won't have time to look into this further until Friday at the earliest.

[minrk]

Since that issue got pretty long with digressions into my own cmake confusion, I'm going to try to make a concise summary:

• openmpi is compiled with 2.17 and requires glibc >= 2.17. This runtime requirement is enforced properly.
• To link openmpi, you must either have sysroot_linux >=2.17 or add -Wl,--allow-shlib-undefined to your link flags (usually $LDFLAGS)
• The gcc package will get sysroot_linux 2.12 by default because newer versions are "weighed down" for maximum compatibility by default (the default will be 2.17 later this summer, I think)
• The c-compiler package (gcc_linux-64 aliased as c-compiler or {{ compiler("c"} }} in a recipe) sets -Wl,--allow-shlib-undefined in $LDFLAGS on activation, so most build systems will work by default. If this isn't happening for you, the main reasons would be:
  • $LDFLAGS isn't accepted by your build system (it's usually a good idea, I think), or
  • gcc is installed instead of c-compiler, so $LDFLAGS is not set. Whether this is right will depend on your circumstances.

So the simple version is to require sysroot_linux>=2.17 in your environment, because then there will be no undefined symbols. The alternative is to leave c_stdlib_version alone and make sure that either $LDFLAGS are passed through, or set -Wl,--allow-shlib-undefined explicitly.

This issue is not specific to openmpi, but actually most libraries built with newer-than-2.12 will encounter some version of this when downstream packages attempt to link them, either in runtime user environments or package building.

[xylar]

@minrk, thank for posting on this! I really appreciate it.

[xylar]

@mcarlson801, I wasn't able to reproduce the problem on Perlmutter. I see:

...
-- The C compiler identification is GNU 12.3.0                                  
-- The CXX compiler identification is GNU 12.3.0                                
-- Detecting C compiler ABI info                                                
-- Detecting C compiler ABI info - done                                         
-- Check for working C compiler: /global/u2/x/xylar/miniforge3/envs/compass_bootstrap/bin/cc - skipped                                                          
-- Detecting C compile features                                                 
-- Detecting C compile features - done                                          
-- Detecting CXX compiler ABI info                                              
-- Detecting CXX compiler ABI info - done                                       
-- Check for working CXX compiler: /global/u2/x/xylar/miniforge3/envs/compass_bootstrap/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- GCXX-like compiler detected
-- Installing locally
-- NetCDF library found
-- NetCDF inc. lib: /global/homes/x/xylar/miniforge3/envs/compass_test/lib/libnetcdf.so
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- OpenMP library found
-- OpenMP inc. lib: gomp;pthread
-- Configuring done (1.6s)
-- Generating done (0.1s)
-- Build files have been written to: /global/u2/x/xylar/compass/fix-jigsaw-build/jigsaw-python/external/jigsaw/tmp
[ 12%] Building CXX object src/CMakeFiles/jigsaw-cmd.dir/jigsaw.cpp.o
[ 25%] Building CXX object src/CMakeFiles/tripod-cmd.dir/jigsaw.cpp.o
[ 50%] Building CXX object src/CMakeFiles/marche-cmd.dir/jigsaw.cpp.o
[ 50%] Building CXX object src/CMakeFiles/jigsaw-lib.dir/jigsaw.cpp.o
[ 62%] Linking CXX executable marche
[ 62%] Built target marche-cmd
[ 75%] Linking CXX executable tripod
[ 75%] Built target tripod-cmd
[ 87%] Linking CXX shared library libjigsaw.so
[100%] Linking CXX executable jigsaw
[100%] Built target jigsaw-cmd
[100%] Built target jigsaw-lib
Install the project...
-- Install configuration: "Release"
-- Installing: /global/u2/x/xylar/compass/fix-jigsaw-build/jigsaw-python/external/jigsaw/bin/jigsaw
-- Set non-toolchain portion of runtime path of "/global/u2/x/xylar/compass/fix-jigsaw-build/jigsaw-python/external/jigsaw/bin/jigsaw" to "/global/homes/x/xylar/miniforge3/envs/compass_test/lib"
-- Installing: /global/u2/x/xylar/compass/fix-jigsaw-build/jigsaw-python/external/jigsaw/bin/tripod
-- Set non-toolchain portion of runtime path of "/global/u2/x/xylar/compass/fix-jigsaw-build/jigsaw-python/external/jigsaw/bin/tripod" to "/global/homes/x/xylar/miniforge3/envs/compass_test/lib"
-- Installing: /global/u2/x/xylar/compass/fix-jigsaw-build/jigsaw-python/external/jigsaw/bin/marche
-- Set non-toolchain portion of runtime path of "/global/u2/x/xylar/compass/fix-jigsaw-build/jigsaw-python/external/jigsaw/bin/marche" to "/global/homes/x/xylar/miniforge3/envs/compass_test/lib"
-- Installing: /global/u2/x/xylar/compass/fix-jigsaw-build/jigsaw-python/external/jigsaw/lib/libjigsaw.so
-- Set non-toolchain portion of runtime path of "/global/u2/x/xylar/compass/fix-jigsaw-build/jigsaw-python/external/jigsaw/lib/libjigsaw.so" to "/global/homes/x/xylar/miniforge3/envs/compass_test/lib"
Installing JIGSAW and JIGSAW-Python


 Running:
   source /global/homes/x/xylar/miniforge3/etc/profile.d/conda.sh &&conda activate compass_test
   cd /global/u2/x/xylar/compass/fix-jigsaw-build/jigsaw-python
   python -m pip install --no-deps -e .
   cp jigsawpy/_bin/* ${CONDA_PREFIX}/bin

Obtaining file:///global/u2/x/xylar/compass/fix-jigsaw-build/jigsaw-python
  Preparing metadata (setup.py) ... done
Installing collected packages: jigsawpy
  Running setup.py develop for jigsawpy
Successfully installed jigsawpy-1.0.0
JIGSAW install took 72.0 s.
...

[xylar]

@mcarlson801, could you let me know if your nightly builds are still failing? This may have been fixed on the conda-forge side somehow. If they are, can you point me to a more detailed log?

[mcarlson801]

@mcarlson801, could you let me know if your nightly builds are still failing? This may have been fixed on the conda-forge side somehow. If they are, can you point me to a more detailed log?

I just gave it a try again to see and it worked this time. I guess there must have been a system issue earlier this week that's already resolved. False alarm, I guess! Thanks for looking into it.

[xylar]

Okay, please reopen this if you see it again.

[xylar]

Also, I was amused to get GitHub emails that are from "Max" and "Min". Thanks @mcarlson801 and @minrk for being an unwitting duo.