Cannot turn off OpenMP if hpc-stack (NCEPLIBS) is built with OpenMP #410

climbfuji · 2021-02-07T13:13:31Z

Description

Turning off OpenMP for the ufs-weather-model has no effect when the NCEPLIBS are built with OpenMP enabled. In particular, flags like -fopenmp are still passed to the subcomponents (e.g. FMS as in PR #392) even if these turn off OpenMP in their CMakeLists.txt.

To Reproduce:

Tested on Cheyenne with GNU.

Check out the ufs-weather-model
Add set(OPENMP OFF CACHE BOOL "Enable OpenMP threading" FORCE) to cmake/configure_cheyenne.gnu.cmake
Turn on the BUILD_VERBOSE flag for build.sh to see the output of make
Compile with any suite in REPRO or PROD mode, for example

./compile.sh cheyenne.gnu 'SUITES=FV3_GFS_2017_gfdlmp REPRO=Y' '' NO NO 2>&1 | tee compile.log

Look at compile.log

Additional context

I added an INFO message to the two if tests related to OpenMP in the top-level CMakeLists.txt:

--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -90,7 +90,8 @@ message("")
 ###############################################################################
 find_package(MPI REQUIRED)
 if(OPENMP)
-  find_package(OpenMP REQUIRED)
+    message(INFO "OPENMP is true even though it shouldn't")
+#  find_package(OpenMP REQUIRED)
 endif()
 find_package(NetCDF REQUIRED C Fortran)
 find_package(ESMF MODULE REQUIRED)
@@ -158,7 +159,8 @@ target_include_directories(fms INTERFACE
 target_link_libraries(fms PUBLIC MPI::MPI_Fortran
                                  NetCDF::NetCDF_Fortran)
 if(OpenMP_Fortran_FOUND)
-  target_link_libraries(fms PRIVATE OpenMP::OpenMP_Fortran)
+    message(INFO "OpenMP_Fortran_FOUND is true even though it shouldn't")
+#  target_link_libraries(fms PRIVATE OpenMP::OpenMP_Fortran)
 endif()

Output

Setting configuration for cheyenne.gnu

32BIT ............ OFF
AVX2 ............. ON
SIMDMULTIARCH ...  OFF
DEBUG ............ OFF
DEBUG_LINKMPI .... ON
INLINE_POST ...... ON
MULTI_GASES ...... OFF
OPENMP ........... OFF
PARALLEL_NETCDF .. OFF
QUAD_PRECISION ... ON
REPRO ............ Y
WW3 .............. OFF
S2S .............. OFF
DATM ............. OFF

C       compiler: GNU 9.1.0 (mpicc)
CXX     compiler: GNU 9.1.0 (mpicxx)
Fortran compiler: GNU 9.1.0 (mpif90)


-- FindNetCDF defines targets:
--   - NetCDF_VERSION [4.7.4]
--   - NetCDF_PARALLEL [TRUE]
--   - NetCDF_C_CONFIG_EXECUTABLE [/glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/netcdf/4.7.4/bin/nc-config]
--   - NetCDF::NetCDF_C [STATIC] [Root: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/netcdf/4.7.4] Lib: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/netcdf/4.7.4/lib/libnetcdf.a
--   - NetCDF_Fortran_CONFIG_EXECUTABLE [/glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/netcdf/4.7.4/bin/nf-config]
--   - NetCDF::NetCDF_Fortran [STATIC] [Root: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/netcdf/4.7.4] Lib: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/netcdf/4.7.4/lib/libnetcdff.a
-- Found ESMF library: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/esmf/8_1_0_beta_snapshot_27/lib/libesmf.a
-- Found bacio: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/bacio/2.4.1/lib/libbacio_4.a (found version "2.4.0")
-- Found bacio: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/bacio/2.4.1/lib/libbacio_4.a (found version "2.4.0")
-- Found w3nco: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/w3nco/2.4.1/lib/libw3nco_4.a (found version "2.4.0")
-- Found MPI: TRUE (found version "3.1") found components: Fortran
-- Found nemsio: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/nemsio/2.5.2/lib/libnemsio.a (found version "2.5.1")
-- Found OpenMP_Fortran: -fopenmp
-- Found sp: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/sp/2.3.3/lib/libsp_4.a (found version "2.3.3")
-- Found bacio: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/bacio/2.4.1/lib/libbacio_4.a (found version "2.4.0")
-- Found w3nco: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/w3nco/2.4.1/lib/libw3nco_4.a (found version "2.4.0")
-- Found nemsio: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/nemsio/2.5.2/lib/libnemsio.a (found version "2.5.1")
-- Found sigio: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/sigio/2.3.2/lib/libsigio.a (found version "2.3.0")
-- Found MPI: TRUE (found version "3.1")
-- Found w3emc: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/w3emc/2.7.3/lib/libw3emc_4.a (found version "2.7.3")
-- Found w3nco: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/w3nco/2.4.1/lib/libw3nco_4.a (found version "2.4.0")
-- Found OpenMP_Fortran: -fopenmp
-- Found w3nco: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/w3nco/2.4.1/lib/libw3nco_4.a (found version "2.4.0")
-- Found g2: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/g2/3.4.1/lib/libg2_4.a (found version "3.4.0")
-- Found g2tmpl: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/g2tmpl/1.9.1/lib/libg2tmpl.a (found version "1.9.0")
-- Found bacio: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/bacio/2.4.1/lib/libbacio_4.a (found version "2.4.0")
-- Found OpenMP_Fortran: -fopenmp
-- Found ip: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/ip/3.3.3/lib/libip_4.a (found version "3.3.3")
-- Found OpenMP_Fortran: -fopenmp
-- Found sp: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/sp/2.3.3/lib/libsp_4.a (found version "2.3.3")
-- Found bacio: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/bacio/2.4.1/lib/libbacio_4.a (found version "2.4.0")
-- Found w3nco: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/w3nco/2.4.1/lib/libw3nco_4.a (found version "2.4.0")
-- Found MPI: TRUE (found version "3.1") found components: Fortran
-- Found nemsio: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/nemsio/2.5.2/lib/libnemsio.a (found version "2.5.1")
-- Found sigio: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/sigio/2.3.2/lib/libsigio.a (found version "2.3.0")
-- Found MPI: TRUE (found version "3.1")
-- Found w3emc: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/w3emc/2.7.3/lib/libw3emc_4.a (found version "2.7.3")
-- Found crtm: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/crtm/2.3.0/lib/libcrtm.a (found version "2.3.0")
-- Found upp: /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/gnu-9.1.0/mpt-2.22/upp/10.0.0/lib/libupp.a (found version "10.0.0")
Found Python: /usr/bin/python3.4
INFO OpenMP_Fortran_FOUND is true even though it shouldn't?
...

The text was updated successfully, but these errors were encountered:

junwang-noaa · 2021-04-12T03:23:58Z

@DusanJovic-NOAA @aerorahul Is this true we have to run the model with openmp if hpc_stack is built with OpenMP?

DusanJovic-NOAA · 2021-04-12T14:08:38Z

@DusanJovic-NOAA @aerorahul Is this true we have to run the model with openmp if hpc_stack is built with OpenMP?

Correct.

aerorahul · 2021-04-12T14:16:40Z

@DusanJovic-NOAA

This needs a clarification. NCEPlibs are currently being built with OpenMP ON and the OpenMP is linked PUBLIC-ly like so:

target_link_libraries(libName PUBLIC OpenMP::OpenMP_Fortran)

This needs to be turned to:

target_link_libraries(libName PRIVATE OpenMP::OpenMP_Fortran)

Doing so, will prevent OpenMP to be transiently propagated to the application linking against libName.
If the application is compiled without OpenMP, export OMP_NUM_THREADS=nthreads) will have no impact on the application threads, but libName will run with nthreads.

Someone needs to independently verify this. I had myself convinced with a simple test of my own.
I read about this somewhere and they recommended building the entire tree with and without OpenMP to clearly isolate these type of issues.

aerorahul · 2021-04-12T14:20:27Z

@climbfuji
The Found OpenMP messages you see are in the dependency libraries. That does not necessarily mean that OpenMP was found in the Project scope.

Try:

message(STATUS "OpenMP FOUND at UFS level: ${OpenMP_Fortran_FOUND}")

after all the find_packages. It should be OFF, but it is not.

DusanJovic-NOAA · 2021-04-12T19:46:44Z

@climbfuji
The Found OpenMP messages you see are in the dependency libraries. That does not necessarily mean that OpenMP was found in the Project scope.

Try:
message(STATUS "OpenMP FOUND at UFS level: ${OpenMP_Fortran_FOUND}")
after all the find_packages. It should be OFF, but it is not.

Then, specifying dependency on OpenMP should be conditional on OPENMP AND OpenMP_Fortran_FOUND. For example in

FV3/CMakeLists.txt:if(OpenMP_Fortran_FOUND)
FV3/CMakeLists.txt:  target_link_libraries(fv3dycore PUBLIC OpenMP::OpenMP_Fortran)

should be:

FV3/CMakeLists.txt:if(OPENMP AND OpenMP_Fortran_FOUND)
FV3/CMakeLists.txt:  target_link_libraries(fv3dycore PUBLIC OpenMP::OpenMP_Fortran)

In fact only if(OPENMP) should be enough. The fact that OpenMP is found, should not necessarily mean that it should be used as a dependency.

junwang-noaa · 2021-04-23T12:20:01Z

Code updates were committed, close the issue

climbfuji added the bug Something isn't working label Feb 7, 2021

climbfuji assigned climbfuji, aerorahul and DusanJovic-NOAA Feb 7, 2021

This was referenced Apr 16, 2021

Use OPENMP instead of OpenMP_Fortran_FOUND, cleanup compiler flags #531

Merged

Use OPENMP instead of OpenMP_Fortran_FOUND, cleanup compiler flags NOAA-PSL/stochastic_physics#39

Merged

junwang-noaa closed this as completed Apr 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot turn off OpenMP if hpc-stack (NCEPLIBS) is built with OpenMP #410

Cannot turn off OpenMP if hpc-stack (NCEPLIBS) is built with OpenMP #410

climbfuji commented Feb 7, 2021

junwang-noaa commented Apr 12, 2021 •

edited

Loading

DusanJovic-NOAA commented Apr 12, 2021

aerorahul commented Apr 12, 2021 •

edited

Loading

aerorahul commented Apr 12, 2021 •

edited

Loading

DusanJovic-NOAA commented Apr 12, 2021 •

edited

Loading

junwang-noaa commented Apr 23, 2021

Cannot turn off OpenMP if hpc-stack (NCEPLIBS) is built with OpenMP #410

Cannot turn off OpenMP if hpc-stack (NCEPLIBS) is built with OpenMP #410

Comments

climbfuji commented Feb 7, 2021

Description

To Reproduce:

Additional context

Output

junwang-noaa commented Apr 12, 2021 • edited Loading

DusanJovic-NOAA commented Apr 12, 2021

aerorahul commented Apr 12, 2021 • edited Loading

aerorahul commented Apr 12, 2021 • edited Loading

DusanJovic-NOAA commented Apr 12, 2021 • edited Loading

junwang-noaa commented Apr 23, 2021

junwang-noaa commented Apr 12, 2021 •

edited

Loading

aerorahul commented Apr 12, 2021 •

edited

Loading

aerorahul commented Apr 12, 2021 •

edited

Loading

DusanJovic-NOAA commented Apr 12, 2021 •

edited

Loading