Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test MAPL v2.53.0 in UFS weather model #2346

Open
Tracked by #2984
junwang-noaa opened this issue Jun 28, 2024 · 92 comments
Open
Tracked by #2984

Test MAPL v2.53.0 in UFS weather model #2346

junwang-noaa opened this issue Jun 28, 2024 · 92 comments
Assignees
@RatkoVasic-NOAA @ulmononian @natalie-perlin
Labels
enhancement New feature or request

Comments

@junwang-noaa
Copy link
Collaborator

junwang-noaa commented Jun 28, 2024 •
edited
Loading

Description

MAPL 2.46.2 has fixes for issue #2162. UFS weather model needs to be tested and updated with this version.

20240830:
MAPL 2.46.2 has a bug. MAPL 2.46.3 should be installed and tested in UFS weather model. The issue title is updated.

Solution

Alternatives

Related to
@junwang-noaa junwang-noaa added the enhancement New feature or request label Jun 28, 2024
@junwang-noaa
Copy link
Collaborator Author

@lipan-NOAA Can you confirm this version of MAPL works in GOCART for GEFSv13? Thanks

@BrianCurtis-NOAA
Copy link
Collaborator

@Hang-Lei-NOAA is MAPL 2.46.2 installed on Acorn/WCOSS2?

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Jul 1, 2024 via email

@lipan-NOAA
Copy link
Collaborator

@Hang-Lei-NOAA Can you tell me where you installed it?

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Jul 2, 2024 via email

@ulmononian
Copy link
Collaborator

do you want this installed in spack-stack/1.6.0? and on which machine?

@ulmononian ulmononian mentioned this issue Jul 2, 2024
Merged
@junwang-noaa junwang-noaa mentioned this issue Aug 12, 2024
Open
@junwang-noaa junwang-noaa changed the title Test MAPL v2.46.2 in UFS weather model Test MAPL v2.46.3 in UFS weather model Aug 30, 2024
@jkbk2004
Copy link
Collaborator

@DusanJovic-NOAA @junwang-noaa new maple and esmf version are available on hercules and orion for the test and debug activities. @RatkoVasic-NOAA thanks for the installation!

Hercules: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/ue-esmf-8.6.1-mapl-2.46.3/install/modulefiles/Core
Orion: /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/ue-esmf-8.6.1-mapl-2.46.3/install/modulefiles/Core

@jkbk2004 jkbk2004 mentioned this issue Sep 24, 2024
Open
14 tasks
@mathomp4
Copy link

As an FYI, the MAPL 2.46.3 fix was when using externally initialized MPI. (Which is something you all do but we don't do internally).

We also had this notice to users:

Notice to users: External code should initialize MPI with MPI_THREAD_MULTIPLE as that is what MAPL expects. Also, code might need to call:

call ESMF_InitializePreMPI()

for all features of MAPL to be supported, namely if ESMF Single System Image (SSI) code is enabled. If users do not use this (or know what it is), it most likely is not needed.

My guess is you do not need to call ESMF_InitializePreMPI() as I don't think even we internally quite use all the SSI code yet.

@DavidHuber-NOAA DavidHuber-NOAA mentioned this issue Oct 4, 2024
Open
36 tasks
@DavidHuber-NOAA DavidHuber-NOAA mentioned this issue Nov 19, 2024
Open
@junwang-noaa junwang-noaa mentioned this issue Dec 2, 2024
Draft
14 tasks
@junwang-noaa
Copy link
Collaborator Author

@weiyuan-jiang may I ask what compiler/mpi versions you want to test? Thanks

@mathomp4
Copy link

mathomp4 commented Dec 2, 2024

@weiyuan-jiang may I ask what compiler/mpi versions you want to test? Thanks

@junwang-noaa Well for that version of MAPL, we would have been testing with:

  • Intel ifort 2021.6
  • Intel ifort 2021.12 (or .13)
  • GCC 13.2.0

We never had Intel ifort 2021.9 on any machines we have. Are any of these possible?

@junwang-noaa
Copy link
Collaborator Author

@AlexanderRichert-NOAA do we have these intel and GCC version on Hera or Hercules orGaea?

@AlexanderRichert-NOAA
Copy link
Collaborator

I can't get onto Hera at the moment but for the others:

  • Gaea (C5 & C6):
    • ifort 2021.6 (module load intel-classic/2022.1.0)
    • GCC 13.2 (module load gcc-native/13.2)
  • Hercules & Orion:
    • ifort 2021.12.0 (module load intel-oneapi-compilers/2024.1.0)

@mathomp4
Copy link

mathomp4 commented Dec 3, 2024

As we have access to Hercules/Orion, I guess ifort 2021.12 is our best bet at the moment (though I'd have to imagine there must be a recent-ish gcc on there).

I'm fairly certain we can run with 2021.12. There were some bugfixes needed in places of GEOS (I don't think MAPL, but I can find out), so by the time we got them all in, 2021.13 was out, but I think 2021.12 works.

Plus, we have 2021.12 on discover, so we can do some matching if need be.

@junwang-noaa
Copy link
Collaborator Author

@weiyuan-jiang @mathomp4 Which MAPL version would you like us to test in UFS? I assume the ESMF version is 8.6.1

@weiyuan-jiang
Copy link
Collaborator

@junwang-noaa It doesn't matter. I can always replace the MAPL as long as it can be built and run under new compiler on Hercules

@DusanJovic-NOAA
Copy link
Collaborator

@DusanJovic-NOAA Do you want mapl to be updated to 2.52.0 for spack-stack-1.9.0 (we already have esmf 8.8.0)? We are hoping to cut the release branch tomorrow, thus this would be just in time. Thanks!

Note: We do think we have the fix for the GOCART OpenMP issue here:

GEOS-ESM/MAPL#3350

I believe it worked for @weiyuan-jiang on Hercules, but if it works for you, I could get a MAPL version out tomorrow.

Also, MAPL 2.52 doesn't (yet) require ESMF 8.8 as we don't need any features from that yet in MAPL2. MAPL3 does currently require it.

Thanks for the update @mathomp4. I am using a shell script provided by @AlexanderRichert-NOAA to build chained spack environment with updated ESMF and MAPL to run test on Hercules. If you can make a temporary tag or branch or something that spack can point to (I'm not sure if it has to be release), I should be able to try that version that includes OpenMP fix from 3350 PR you mentioned above. And if we can get that tested before @climbfuji needs to cut release for 1.9.0 spack-stack that would be perfect.

@climbfuji
Copy link
Collaborator

Other components/models need esmf 8.8.0, might as well use that across the board if it works for Dusan. Thanks Matt!

@mathomp4
Copy link

@DusanJovic-NOAA I've tagged with temp-tag-for-dusan if that's easy to use. It's also at commit GEOS-ESM/MAPL@260d563

@DusanJovic-NOAA
Copy link
Collaborator

@DusanJovic-NOAA I've tagged with temp-tag-for-dusan if that's easy to use. It's also at commit GEOS-ESM/MAPL@260d563

Thanks. @AlexanderRichert-NOAA is there a way to use this tag (specific commit hash) in your env chainer script to build MAPL.

@AlexanderRichert-NOAA
Copy link
Collaborator

Thanks. @AlexanderRichert-NOAA is there a way to use this tag (specific commit hash) in your env chainer script to build MAPL.

@DusanJovic-NOAA in theory yes-- I just made a small tweak in the env chainer script to allow it to use Spack's built-in support for using git references. For the mapl spec, it would be something like mapl@git.260d563d08ff0d3e8436ec33ed98a362110e731b=develop where 'develop' is the version that Spack is going to treat it as (you can set this to a numbered version for the purpose of determining dependencies etc.). The only trick is that in that commit, MAPL's CMakeLists.txt requires CMake 3.24 or higher, which is not what spack-stack-1.6.0 uses, so you have to work around that. mapl@git.260d563d08ff0d3e8436ec33ed98a362110e731b=develop cmake@3.24.4 for the specs worked for me on hercules just now (with another 'git pull' of the env chainer script, and not forgetting about the depends_on("udunits")).

@DusanJovic-NOAA
Copy link
Collaborator

Thanks. @AlexanderRichert-NOAA is there a way to use this tag (specific commit hash) in your env chainer script to build MAPL.

@DusanJovic-NOAA in theory yes-- I just made a small tweak in the env chainer script to allow it to use Spack's built-in support for using git references. For the mapl spec, it would be something like mapl@git.260d563d08ff0d3e8436ec33ed98a362110e731b=develop where 'develop' is the version that Spack is going to treat it as (you can set this to a numbered version for the purpose of determining dependencies etc.). The only trick is that in that commit, MAPL's CMakeLists.txt requires CMake 3.24 or higher, which is not what spack-stack-1.6.0 uses, so you have to work around that. mapl@git.260d563d08ff0d3e8436ec33ed98a362110e731b=develop cmake@3.24.4 for the specs worked for me on hercules just now (with another 'git pull' of the env chainer script, and not forgetting about the depends_on("udunits")).

Thanks. Let me try that.

@mathomp4
Copy link

I just confirmed with @weiyuan-jiang that he didn't do anything else after using that new MAPL code. He kept OpenMP on in his test.

He did mention you'll need to make sure you have udunits loaded, but @AlexanderRichert-NOAA indicated that as well.

@DusanJovic-NOAA
Copy link
Collaborator

I successfully compiled MAPL tag (commit 260d563d08ff0d3e8436ec33ed98a362110e731b) using @AlexanderRichert-NOAA instructions, and I am running all intel ufs-wm regression tests. So far all tests passed (most importantly, cpld_control_p8_intel passed, which is the test that was failing previously). This is the configuration with:

$ grep use_threads parm/gocart/GOCART2G_GridComp.rc
use_threads: .TRUE.

So looks like we finally have a version of MAPL that fixes both issues, cubed_sphere grid with threading and masking of ESMF_grid due to mask value conflict between fv3atm and mapl.

In the meantime I will try to build spack-stack using GNU, and run gnu ufs-wm regression tests, just to be sure.

@DusanJovic-NOAA
Copy link
Collaborator

DusanJovic-NOAA commented Jan 24, 2025 •
edited
Loading

I should also mention that in order to use ufs-wm with ESMF v8.8.0 an update in GOCART was needed. In my tests, I used a branch @theurich created that has some updates for esmf 8.8.0. This one. So before we can actually switch to 8.8.0 this change will need to be merged in gocart. The only issue is that we (ufs-wm) are still using a gocart commit from Sep 2023. And Gerhard's changes are on top of that commit. Somebody will need to port Gerhard's changes to newer gocart and then update ufs-wm to use that version.

@climbfuji
Copy link
Collaborator

@mathomp4 or @DusanJovic-NOAA Would you mind creating an issue in github.com/jcsda/spack-stack to update MAPL to version x.y.z for spack-stack-1.9.0, and list changes to the build options (if applicable)? Thanks!

@mathomp4
Copy link

mathomp4 commented Jan 24, 2025 •
edited
Loading

I'm going to work on getting MAPL 2.53 out now. Shouldn't take me too long. Then you'll have a semver tag rather than a crazy git-hash.

ETA: Well, might take about 30-40 minutes. My final test is a full run on discover to make sure it is zero-diff, etc. and good ol' CMake has decided to rebuild the full model. I really hope ifx doesn't have this same CMake+ifort issue with recompiling!

@mathomp4 mathomp4 mentioned this issue Jan 24, 2025
Closed
@mathomp4
Copy link

MAPL 2.53.0 has been released:

https://github.com/GEOS-ESM/MAPL/releases/tag/v2.53.0

And I have a PR to spack:

spack/spack#48712

and an issue for JCSDA/spack-stack:

JCSDA/spack-stack#1469

Though that last one might not be "complete" as I'm not sure if any updates are needed for how MAPL is built, etc.

@DusanJovic-NOAA DusanJovic-NOAA changed the title Test MAPL v2.46.3 in UFS weather model Test MAPL v2.53.0 in UFS weather model Jan 24, 2025
@climbfuji
Copy link
Collaborator

Thanks @mathomp4 !

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Jan 27, 2025 via email

@BrianCurtis-NOAA
Copy link
Collaborator

The mapl/2.53.0 requires the cmake min_version as 3.24.0, which is not on
wcoss2.
…

Can you let me know if you install that cmake version on Acorn so I can test it with the current develop version of the UFSWM.

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Jan 27, 2025 via email

@BrianCurtis-NOAA
Copy link
Collaborator

This is system lib, we cannot install it. It has to go through system
management.
…

OK, then we should get the process started to install it.

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Jan 27, 2025 via email

@mathomp4
Copy link

mathomp4 commented Jan 27, 2025 •
edited
Loading

I have modified the CMakelist in mapl/2.53.0, by setting cmake min version to 2.20.0. The build is successful.

Ugh. I guess I figured CMake 3.24 was not new enough to cause issues as it came out in 2022! And, I guess, that maybe spack-stack folks would be building cmake via spack.

I moved MAPL to 3.24 because we needed it for Python support. That was the oldest version of CMake I could get to work with Python 3.12 f2py which moved to meson. Plus Python FIND_STRATEGY LOCATION also seemed to need 3.24 in testing.

That said, I have no idea if MAPL can use CMake 2. That is ANCIENT. Did you mean CMake 3.20?

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Jan 27, 2025 via email

@mathomp4
Copy link

Yes, I used 2.20.3 installed on wcoss2

That seems...weird. I'm not even sure there was a 2.20.3. I mean, there was a CMake 2.8.12.2 in 2018, but I'm not sure the Kitware folks had more in the 2-series before CMake 3.0.0 came out.

@DavidHuber-NOAA
Copy link
Collaborator

Looking on Cactus, the system version of CMake is 3.20.4. There are also modules for 3.20.2 and 3.18.4.

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Jan 28, 2025 via email

@mathomp4
Copy link

Ah. Okay. That makes more sense. Still old, but not quite as old. :)

@DusanJovic-NOAA
Copy link
Collaborator

I have modified the CMakelist in mapl/2.53.0, by setting cmake min version
to 2.20.0. The build is successful.

module use
/lfs/h2/emc/eib/save/hang.lei/forgdit/nco_wcoss2/install2/modulefiles/mpi/intel/19.1.3.304/cray-mpich/8.1.12
module load esmf/8.8.0
module show mapl/2.53.0-esmf-8.8.0

/lfs/h2/emc/eib/save/hang.lei/forgdit/nco_wcoss2/install2/modulefiles/mpi/intel/19.1.3.304/cray-mpich/8.1.12/mapl/2.53.0-esmf-8.8.0.lua:

help([[]])
conflict("mapl")
setenv("MAPL_ROOT","/lfs/h2/emc/eib/save/hang.lei/forgdit/nco_wcoss2/install2/intel-19.1.3.304/cray-mpich-8.1.12/mapl/2.53.0-esmf-8.8.0")
whatis("Name: mapl")
whatis("Version: 2.53.0-esmf-8.8.0")
whatis("Category: library")
whatis("Description: MAPL is a foundation layer of the GEOS architecture")
…

Regression test passed on WCOSS2 (Cactus) using esmf/8.8.0 and mapl/2.53.0-esmf-8.8.0 modules installed in this location.

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Jan 29, 2025 via email

@ulmononian
Copy link
Collaborator

@DusanJovic-NOAA just curious what your ufs_common.lua lib versions were for these runs, or if you have a branch i could check out? (going to test w/ this esmf+mapl pair w/ spack-stack/1.9.0 on ursa).

@DusanJovic-NOAA
Copy link
Collaborator

@DusanJovic-NOAA just curious what your ufs_common.lua lib versions were for these runs, or if you have a branch i could check out? (going to test w/ this esmf+mapl pair w/ spack-stack/1.9.0 on ursa).

I used this branch for testing: https://github.com/DusanJovic-NOAA/ufs-weather-model/tree/esmf880_mapl2530

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RatkoVasic-NOAA RatkoVasic-NOAA

@ulmononian ulmononian

@natalie-perlin natalie-perlin

Labels
enhancement New feature or request
Projects
Model Infrastructure development FY25Q2
Status: In Progress
Model infrastructure development FY25Q1
Status: In Progress
ufs-community User Support
Status: No status
Model infrastructure development FY2024Q4
Status: In Progress
Milestone
No milestone
Development

No branches or pull requests