Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[INSTALL] Install ESMF 8.3.0bs09 #407

Closed
junwang-noaa opened this issue Mar 15, 2022 · 46 comments
Closed

[INSTALL] Install ESMF 8.3.0bs09 #407

junwang-noaa opened this issue Mar 15, 2022 · 46 comments
Assignees
@kgerheiser @Hang-Lei-NOAA

Comments

@junwang-noaa
Copy link

Note: Please search to see if an issue already exists for the software you are requesting to be installed.

Which software in the stack would you like installed?
ESMF
What is the version/tag of the software?
8.3.0bs09
What compilation options would you like set?
Same as previous version
Which machines would you like to have the software installed?
Dell, cray, hera, jet , gaea, orion and cheyenne

Any other relevant information that we should know to correctly install the software??
Please describe in as much detail as possible.

Additional context
This library is required by the ufs-weather-model PR#1104, which needs to be commit by the end of March. We need to get the library installed before that in order to do some testing, see ufs-weather-model issue #1107.
@junwang-noaa
Copy link
Author

@jkbk2004 FYI.

@jkbk2004
Copy link

@junwang-noaa https://github.com/esmf-org/esmf/releases/tag/v8.3.0b09

@junwang-noaa
Copy link
Author

@rsdunlapiv @theurich FYI.

@junwang-noaa
Copy link
Author

@kgerheiser @Hang-Lei-NOAA May I ask if you can install the library on hera first so that we can start testing in ufs-weather-model? Thanks

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 18, 2022 via email

@jkbk2004
Copy link

@Hang-Lei-NOAA note that the current ufs-wm develop branch is using esmf v8.2.1b04 tag

@junwang-noaa
Copy link
Author

junwang-noaa commented Mar 18, 2022 •
edited
Loading

@Hang-Lei-NOAA We can't ESMF 830bs09 on hera, currently we are using the following modules in ufs-weather-model develop branch

module use /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/modulefiles/stack
module load hpc/1.1.0
module load hpc-intel/18.0.5.274
module load hpc-impi/2018.0.4

@DusanJovic-NOAA FYI,

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 18, 2022 via email

@rsdunlapiv
Copy link

@Hang-Lei-NOAA if you cannot figure out the issue, please put the error log here so we can take a look. @theurich

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 18, 2022 via email

@rsdunlapiv
Copy link

@Hang-Lei-NOAA here is an example of a recent Hera build that worked. Maybe you can use this to see if your configuration is different:

Modules
Build log

@kgerheiser
Copy link
Contributor

@Hang-Lei-NOAA's error seems to be related to a core ifort library not linking correctly.

@kgerheiser
Copy link
Contributor

I'll give it a try and see if I can spot anything.

@BinLiu-NOAA
Copy link

In case it might be useful as a reference, I build this esmf/8.3.0b09 tag on Hera with hpc-stack about two weeks ago. And you can find the build log and directory here:
/scratch1/NCEPDEV/hwrf/save/Bin.Liu/hpc_stack_202203/build_stack.log

Bin

@kgerheiser
Copy link
Contributor

@Hang-Lei-NOAA I had no issue building it with intel/18.0.5.274. I recommend making sure you have a clean environment when you install. It's possible something in your environment was mixed up.

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 18, 2022 via email

@DusanJovic-NOAA
Copy link
Contributor

Loading mapl failed after I switched esmf to 8.3.0bs09

$ module load ufs_hera.intel
Lmod has detected the following error:  Cannot load module "mapl/2.11.0-esmf-8.2.1b04". At least one of these module(s) must be loaded:
   esmf/8.2.1b04 esmf/8.2.1b04-debug

While processing the following module(s):
    Module fullname            Module Filename
    ---------------            ---------------
    mapl/2.11.0-esmf-8.2.1b04  /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/modulefiles/mpi/intel/18.0.5.274/impi/2018.0.4/mapl/2.11.0-esmf-8.2.1b04.lua
    ufs_common                 /scratch2/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/esmf830bs09/ufs-weather-model/modulefiles/ufs_common
    ufs_hera.intel             /scratch2/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/esmf830bs09/ufs-weather-model/modulefiles/ufs_hera.intel

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 21, 2022 via email

@DusanJovic-NOAA
Copy link
Contributor

ufs-weather-model regression tests passed on Hera/Intel. Please install these libraries on all other platforms. including GNU versions.

@junwang-noaa
Copy link
Author

@kgerheiser @Hang-Lei-NOAA May I ask if the esmf 830b09 and mapl/2.11.0--esmf-8.3.0b09 are installed on other platforms (orion, jet, gaea, dell, cray)?
@jkbk2004 Would you please install the two libraries on Cheyenne too? Thanks

@kgerheiser kgerheiser mentioned this issue Mar 25, 2022
Merged
@kgerheiser
Copy link
Contributor

Sorry, just saw that you completed the testing. Will install.

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 25, 2022 via email

@climbfuji
Copy link
Contributor

@jkbk2004 @junwang-noaa Can you clarify please if this version is to be installed in the old/existing tree (i.e. do I need to do that) or will this go only into the updated path (i.e. will EPIC do that)?

@jkbk2004 @mark-a-potts What is your take on changing from SGI MPT to Intel IMPI for Intel? Have you tried that? Timing/performance differences?
And if you want to make the change for Intel, do you want to keep GNU with SGI MPT? The reason I am asking is that Intel iMPI has been a better choice for very large configurations (half the machine, entire machine) when SGI MPT segfaulted in the past. Switching the path would be a good occasion to cover Intel iMPI on Cheyenne with the Intel compiler.

At any rate, thanks very much to the EPIC team for taking over the responsibility to maintain the stack on Cheyenne!

@jkbk2004
Copy link

@climbfuji @junwang-noaa @mark-a-potts I am working on it:mapl-2.11.0/emsf-8.3-0-b09/intel-2022. I will catch up with these updates on gnu10.1 that we are using on Cheyenne by Monday. After then, we will put mpt/impi issue on the list. gnu > 10.1 as well.

@climbfuji
Copy link
Contributor

@climbfuji @junwang-noaa @mark-a-potts I am working on it:mapl-2.11.0/emsf-8.3-0-b09/intel-2022. I will catch up with these updates on gnu10.1 that we are using on Cheyenne by Monday. After then, we will put mpt/impi issue on the list. gnu > 10.1 as well.

That's great, thanks for your hard work on this.

@kgerheiser
Copy link
Contributor

Complete on Jet, Orion, and Gaea.

@DusanJovic-NOAA
Copy link
Contributor

It has been installed on dell and cary.

Looks like installation on WCOSS Cray is broken:

-- Detected ESMF Beta snapshot v8.3.0b09
-- Static ESMF library not found, searching for dynamic library instead
-- Neither the dynamic nor the static ESMF library was found
-- Configuring incomplete, errors occurred!
See also "/gpfs/hps3/stmp/Dusan.Jovic/FV3_RT/rt_5782/compile_006/build_fv3_006/CMakeFiles/CMakeOutput.log".
Application 1307068 exit codes: 1

This is from a build log of a ufs-weather-model regression test that uses a debug version of the esmf library.

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 28, 2022 via email

@DusanJovic-NOAA
Copy link
Contributor

I did not see the detailed loading of esmf in "/gpfs/hps3/stmp/Dusan.Jovic/ FV3_RT/rt_5782/compile_006/build_fv3_006/CMakeFiles/CMakeOutput.log" SURGE-slogin1 > module show esmf/8.3.0b09-debug ------------------------------------------------------------------- /gpfs/hps/usrx/local/nceplibs/NCEPLIBS/cmake/install/NCEPLIBS-v1.3.0/modules/esmf/8.3.0b09-debug: conflict ESMF/4_0_0rp2 conflict ESMF/7_0_0 conflict ESMF/7_1_0r conflict ESMF/8.0.0 conflict ESMF/8.1.0 conflict ESMF/8.1.1 conflict ESMF/8.2.0 setenv ESMF_SRC /usrx/local/nceplibs/NCEPLIBS/cmake/src/esmf/pkg/v8.3.0b09-debug setenv ESMF_LIBSDIR /gpfs/hps/usrx/local/nceplibs/NCEPLIBS/cmake/install/NCEPLIBS-v1.3.0/esmf/esmf-8.3.0b09-debug/lib setenv esmf_lib /gpfs/hps/usrx/local/nceplibs/NCEPLIBS/cmake/install/NCEPLIBS-v1.3.0/esmf/esmf-8.3.0b09-debug/lib -lesmf -lstdc++ setenv ESMF_MOD /gpfs/hps/usrx/local/nceplibs/NCEPLIBS/cmake/install/NCEPLIBS-v1.3.0/esmf/esmf-8.3.0b09-debug/mod setenv ESMF_VER 8.3.0 setenv ESMFMKFILE /gpfs/hps/usrx/local/nceplibs/NCEPLIBS/cmake/install/NCEPLIBS-v1.3.0/esmf/esmf-8.3.0b09-debug/lib/esmf.mk prepend-path LD_LIBRARY_PATH /gpfs/hps/usrx/local/nceplibs/NCEPLIBS/cmake/install/NCEPLIBS-v1.3.0/esmf/esmf-8.3.0b09-debug/lib setenv HDF5_USE_FILE_LOCKING FALSE ------------------------------------------------------------------- ls /gpfs/hps/usrx/local/nceplibs/NCEPLIBS/cmake/install/NCEPLIBS-v1.3.0/esmf/esmf-8.3.0b09-debug/lib esmf.mk libesmf.a libesmf.so libesmftrace_preload.so libesmftrace_static.a libg preload.sh
…
On Mon, Mar 28, 2022 at 9:53 AM Dusan Jovic @.> wrote: It has been installed on dell and cary. Looks like installation on WCOSS Cray is broken: -- Detected ESMF Beta snapshot v8.3.0b09 -- Static ESMF library not found, searching for dynamic library instead -- Neither the dynamic nor the static ESMF library was found -- Configuring incomplete, errors occurred! See also "/gpfs/hps3/stmp/Dusan.Jovic/FV3_RT/rt_5782/compile_006/build_fv3_006/CMakeFiles/CMakeOutput.log". Application 1307068 exit codes: 1 This is from a build log of a ufs-weather-model regression test that uses a debug version of the esmf library. — Reply to this email directly, view it on GitHub <#407 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKWSMFHXXZKWEUY567LQRB3VCG2VDANCNFSM5QXIPIAQ . You are receiving this because you were mentioned.Message ID: @.>

It works now. Thanks.

@jkbk2004
Copy link

@junwang-noaa @DusanJovic-NOAA esmf/v8.3.0b09 update passes all RT tests on Cheyenne. New hpc-stack installation is module use /glade/scratch/epicufsrt/hpc-stack-v1.2.0_6eb6/modulefiles/stack. @Hang-Lei-NOAA @kgerheiser I have a question about module names. Right now, new esmf and mapl updates are installed on Cheyenne as module load esmf/v8.3.0b09 and module load mapl/v2.11.0. How do I change the version names like esmf/8.3.0b09 or mapl/2.11.0-esmf-8.3.0b09?

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 28, 2022 via email

@climbfuji
Copy link
Contributor

@junwang-noaa @DusanJovic-NOAA esmf/v8.3.0b09 update passes all RT tests on Cheyenne. New hpc-stack installation is module use /glade/scratch/epicufsrt/hpc-stack-v1.2.0_6eb6/modulefiles/stack. @Hang-Lei-NOAA @kgerheiser I have a question about module names. Right now, new esmf and mapl updates are installed on Cheyenne as module load esmf/v8.3.0b09 and module load mapl/v2.11.0. How do I change the version names like esmf/8.3.0b09 or mapl/2.11.0-esmf-8.3.0b09?

Are you sure you want to install in /glade/scratch/? There is an automatic purger that deletes files older than a certain number of days. It's pretty generous, 120 days at the moment, but CISL adjusts this based on usage: https://arc.ucar.edu/knowledge_base/68878466

The stack should go to the work space imo.

@jkbk2004
Copy link

@climbfuji @mark-a-potts the esmf update works well on Cheyenne. And we have the new hpc-stack installation on the epicufsrt directory.

@climbfuji climbfuji removed their assignment Mar 28, 2022
@jkbk2004
Copy link

@climbfuji Thanks for the note. They set 20TB under scratch/epicufsrt. I will double check.

@kgerheiser
Copy link
Contributor

@jkbk2004 check out PR #417 which gives an install_as option to ESMF, and MAPL automatically strips the v.

@kgerheiser
Copy link
Contributor

Closing this because ESMF has been successfully installed. Please open a new issue if you have other questions.

@DusanJovic-NOAA
Copy link
Contributor 

Lmod has detected the following error: Cannot load module                                                                            
"mapl/2.11.0-esmf-8.3.0b09-debug". At least one of these module(s) must be                                                           
loaded:                                                                                                                              
   esmf/8.3.0b09-debug esmf/8.3.0b09-debug-debug                                                                                     
                                                                                                                                     
While processing the following module(s):                                                                                            
    Module fullname                  Module Filename                                                                                 
    ---------------                  ---------------                                                                                 
    mapl/2.11.0-esmf-8.3.0b09-debug  /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/modulefiles/mpi/gnu/9.2.0/mpich/3.3.2/mapl/2.11.0-esmf-8.3.0b09-debug.lua
    ufs_common                       /scratch1/NCEPDEV/nems/emc.nemspara/autort/pr/892049795/20220330184513/ufs-weather-model/modulefiles/ufs_common
    ufs_hera.gnu                     /scratch1/NCEPDEV/nems/emc.nemspara/autort/pr/892049795/20220330184513/ufs-weather-model/modulefiles/ufs_hera.gnu

@junwang-noaa
Copy link
Author

@kgerheiser @Hang-Lei-NOAA We got following error on jet with ufs-weather-model PR#1146:

Found Python: /apps/spack/linux-centos7-x86_64/gcc-9.2.0/python-3.7.6-gi3efxgcxqilpjehkqnxrriedsuedoqu/bin/python3.7
Calling CCPP code generator (ccpp_prebuild.py) for suites --suites=FV3_GFS_v16_coupled_nsstNoahmpUGWPv1,FV3_GFS_v16_coupled_p7_rrtmgp,FV3_GFS_v17_coupled_p8 ...
Force 32-bit build for GOCART
CMake Error at GOCART/CMakeLists.txt:69 (include):
include could not find requested file:

mapl_acg

CMake Error at GOCART/ESMF/Aerosol_GridComp/CMakeLists.txt:9 (mapl_acg):
Unknown CMake command "mapl_acg".

We had same problem on jet with PR #1071 when updating MAPL/2.11.0, I saw Kyle's comments there, could this also be the issue of lib vs lib64? Thanks.

MAPL is installed properly, but the lib directory is lib64. It's possible MAPL changed their install directory to CMAKE_INSTALL_LIBDIR in this newer version.

ls /lfs4/HFIP/hfv3gfs/nwprod/hpc-stack/libs/intel-18.0.5.274/impi-2018.4.274/mapl/2.11.0-esmf-8.2.1b04/lib64/

cmake		   libMAPL.a	   libMAPL.cap.a	libMAPL.generic.a    libMAPL.griddedio.a  libMAPL.oomph.a  libMAPL.pfio.a      libMAPL.shared.a   libMAPL_cfio_r8.a
libMAPL.ExtData.a  libMAPL.base.a  libMAPL.constants.a	libMAPL.gridcomps.a  libMAPL.history.a	  libMAPL.orbit.a  libMAPL.profiler.a  libMAPL_cfio_r4.a  libpflogger.a

@DusanJovic-NOAA FYI.

@junwang-noaa junwang-noaa mentioned this issue Mar 31, 2022
Merged
16 tasks
@kgerheiser kgerheiser reopened this Mar 31, 2022
@Hang-Lei-NOAA Hang-Lei-NOAA reopened this Mar 31, 2022
@Hang-Lei-NOAA
Copy link
Contributor 

Lmod has detected the following error: Cannot load module                                                                            
"mapl/2.11.0-esmf-8.3.0b09-debug". At least one of these module(s) must be                                                           
loaded:                                                                                                                              
   esmf/8.3.0b09-debug esmf/8.3.0b09-debug-debug                                                                                     
                                                                                                                                     
While processing the following module(s):                                                                                            
    Module fullname                  Module Filename                                                                                 
    ---------------                  ---------------                                                                                 
    mapl/2.11.0-esmf-8.3.0b09-debug  /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/modulefiles/mpi/gnu/9.2.0/mpich/3.3.2/mapl/2.11.0-esmf-8.3.0b09-debug.lua
    ufs_common                       /scratch1/NCEPDEV/nems/emc.nemspara/autort/pr/892049795/20220330184513/ufs-weather-model/modulefiles/ufs_common
    ufs_hera.gnu                     /scratch1/NCEPDEV/nems/emc.nemspara/autort/pr/892049795/20220330184513/ufs-weather-model/modulefiles/ufs_hera.gnu

This is corrected with NON-debug esmf on Hera for GNU installation.

@kgerheiser
Copy link
Contributor

I'm looking at the issue on Jet, and I'm not sure what's going on. It's nothing to do with lib64 vs lib.

This is the contents of mapl-import.cmake which sets CMAKE_MODULE_PATH so that GOCART can find mapl_acg.

list (APPEND CMAKE_MODULE_PATH /lfs4/HFIP/hfv3gfs/nwprod/hpc-stack/libs/intel-18.0.5.274/impi-2018.4.274/mapl/2.11.0-esmf-8.2.1b04/share/MAPL/cmake)
include (mapl_create_stub_component)
include (mapl_acg)

It sets lfs4/HFIP/hfv3gfs/nwprod/hpc-stack/libs/intel-18.0.5.274/impi-2018.4.274/mapl/2.11.0-esmf-8.2.1b04/share/MAPL/cmake

Which contains:

$ ls /lfs4/HFIP/hfv3gfs/nwprod/hpc-stack/libs/intel-18.0.5.274/impi-2018.4.274/mapl/2.11.0-esmf-8.2.1b04/share/MAPL/cmake

mapl_acg.cmake

@kgerheiser
Copy link
Contributor

kgerheiser commented Mar 31, 2022 •
edited
Loading

Was looking in the wrong directory. mapl-import.cmake is missing, again.

I'm pretty sure this is a bug in MAPL somehow, or the filesystem, or CMake. I don't know what it only happens on Jet.

@junwang-noaa
Copy link
Author

@kgerheiser @Hang-Lei-NOAA Thank you for fixing the issues!

@kgerheiser
Copy link
Contributor

I think everything is squared away now. If you have anymore issues, you can re-open this.

@MicroTed
Copy link

MicroTed commented Apr 14, 2022 •
edited
Loading

I suppose the stack *.yaml files will be updated for this? The latest develop branch appears to need the newer esmf now. On mac (intel, Catalina), this change to stack/stack_mac.yaml works:

 esmf:
   build: YES
-  version: ESMF_8_2_0
-  install_as: 8.2.0
+  version: v8.3.0b09
+  install_as: 8.3.0b9

@mkavulich mkavulich mentioned this issue Apr 20, 2022
Open
@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Oct 11, 2022 via email

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Oct 11, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kgerheiser kgerheiser

@Hang-Lei-NOAA Hang-Lei-NOAA

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@kgerheiser @rsdunlapiv @climbfuji @BinLiu-NOAA @junwang-noaa @MicroTed @jkbk2004 @Hang-Lei-NOAA @DusanJovic-NOAA