Test ESMF 8.6.1 beta in UFS weather model #2230

junwang-noaa · 2024-04-08T14:37:47Z

Description

The ESMF 8.6.1 beta has a fix for issue #1121, 1024 character limit is removed. ESMF team is asking the confirmation that this fix resolved the issue reported in UFS weather model.

Solution

install a test version of ESMF 8.6.1 beta (https://github.com/esmf-org/esmf/releases/tag/v8.6.1b03) on hera.
Run atm only test (e.g. control_p8) test for longer forecast time with output time specified in the output_fh.

Alternatives

Related to

junwang-noaa · 2024-04-08T14:38:14Z

@jkbk2004 May I ask if EPIC team can install the test library on hera? Thanks

jkbk2004 · 2024-04-09T12:38:53Z

@RatkoVasic-NOAA can you install https://github.com/esmf-org/esmf/releases/tag/v8.6.1b04 to spack stack 1.6.0 location? Hecules or Hera might be a good starting point for this beta test.

RatkoVasic-NOAA · 2024-04-09T15:34:45Z

@jkbk2004 new version of esmf is still not in spack:

    # generate chksum with 'spack checksum esmf@x.y.z'
    version("8.6.0", sha256="ed057eaddb158a3cce2afc0712b49353b7038b45b29aee86180f381457c0ebe7")
    version("8.5.0", sha256="acd0b2641587007cc3ca318427f47b9cae5bfd2da8d2a16ea778f637107c29c4")
    version("8.4.2", sha256="969304efa518c7859567fa6e65efd960df2b4f6d72dbf2c3f29e39e4ab5ae594")

It cannot be installed as part of spack-stack.

uturuncoglu · 2024-04-09T22:11:22Z

@RatkoVasic-NOAA I think you could still install with esmf@=8.6.1b04 syntax in spack side even if it is not in the package.py.

uturuncoglu · 2024-04-17T16:11:54Z

@RatkoVasic-NOAA @junwang-noaa @climbfuji If you need any help about installing beta snapshot from our side, just let us know.

RatkoVasic-NOAA · 2024-04-17T16:14:19Z

@uturuncoglu thanks! I installed esmf-8.6.1b04 on Hercules, under spack-stack-1.6.0
It was without problems. Now we have to install MAPL with new ESMF version.

uturuncoglu · 2024-04-17T16:28:55Z

@RatkoVasic-NOAA That is great. Thanks for the update.

DusanJovic-NOAA · 2024-04-18T14:17:07Z

I'm testing the esmf 8.6.1b04 on Hercules. cpld_cpntrol_p8 fails with this error:

 57: pe=00057 FAIL at line=00510    MAPL_CapGridComp.F90                     <status=41>
121: pe=00121 FAIL at line=00510    MAPL_CapGridComp.F90                     <status=41>
  8: pe=00008 FAIL at line=00510    MAPL_CapGridComp.F90                     <status=41>
  8: pe=00008 FAIL at line=00956    MAPL_CapGridComp.F90                     <status=41>
139: pe=00139 FAIL at line=00510    MAPL_CapGridComp.F90                     <status=41>
139: pe=00139 FAIL at line=00956    MAPL_CapGridComp.F90                     <status=41>
 48: pe=00048 FAIL at line=00510    MAPL_CapGridComp.F90                     <status=41>

DusanJovic-NOAA · 2024-04-18T14:18:56Z

These are the changes I made to current develop branch:

$ git diff 
diff --git a/modulefiles/ufs_common.lua b/modulefiles/ufs_common.lua
index 1f395d97..f05ff8d4 100644
--- a/modulefiles/ufs_common.lua
+++ b/modulefiles/ufs_common.lua
@@ -10,17 +10,17 @@ local ufs_modules = {
   {["netcdf-c"]        = "4.9.2"},
   {["netcdf-fortran"]  = "4.6.0"},
   {["parallelio"]      = "2.5.10"},
-  {["esmf"]            = "8.5.0"},
+  {["esmf"]            = "8.6.1bs4"},
   {["fms"]             = "2023.02.01"},
   {["bacio"]           = "2.4.1"},
   {["crtm"]            = "2.4.0"},
   {["g2"]              = "3.4.5"},
   {["g2tmpl"]          = "1.10.2"},
   {["ip"]              = "4.3.0"},
-  {["sp"]              = "2.3.3"},
+  {["sp"]              = "2.5.0"},
   {["w3emc"]           = "2.10.0"},
   {["gftl-shared"]     = "1.6.1"},
-  {["mapl"]            = "2.40.3-esmf-8.5.0"},
+  {["mapl"]            = "2.40.3-esmf-8.6.1b04"},
   {["scotch"]          = "7.0.4"},
 }
 
diff --git a/modulefiles/ufs_hercules.intel.lua b/modulefiles/ufs_hercules.intel.lua
index 605fe579..63cfaa98 100644
--- a/modulefiles/ufs_hercules.intel.lua
+++ b/modulefiles/ufs_hercules.intel.lua
@@ -2,7 +2,7 @@ help([[
 loads UFS Model prerequisites for Hercules/Intel
 ]])
 
-prepend_path("MODULEPATH", "/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/envs/unified-env/install/modulefiles/Core")
+prepend_path("MODULEPATH", "/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core")
 
 stack_intel_ver=os.getenv("stack_intel_ver") or "2021.9.0"
 load(pathJoin("stack-intel", stack_intel_ver))

jkbk2004 · 2024-04-18T14:24:28Z

@DusanJovic-NOAA can you check /work2/noaa/stmp/jongkim/stmp/jongkim/FV3_RT/rt_3264281/cpld_control_p8_intel ? intel runs ok. Setup is at /work/noaa/epic/jongkim/UFS-RT/hercules/pr-2093/modulefiles. Luckily mapl 2.40.3-esmf-8.5.0 was used ok somehow but mapl should be built with the esmf beta snapshot.

RatkoVasic-NOAA · 2024-04-18T14:25:02Z

@DusanJovic-NOAA there is typo in your ufs_common:
should be 8.6.1b04, not 8.6.1bs4

RatkoVasic-NOAA · 2024-04-18T14:25:48Z

Here is my ufs_common:

  {["jasper"]          = "2.0.32"},
  {["zlib"]            = "1.2.13"},
  {["libpng"]          = "1.6.37"},
  {["hdf5"]            = "1.14.0"},
  {["netcdf-c"]        = "4.9.2"},
  {["netcdf-fortran"]  = "4.6.0"},
  {["parallelio"]      = "2.5.10"},
  {["esmf"]            = "8.6.1b04"},
  {["fms"]             = "2023.04"},
  {["bacio"]           = "2.4.1"},
  {["crtm"]            = "2.4.0"},
  {["g2"]              = "3.4.5"},
  {["g2tmpl"]          = "1.10.2"},
  {["ip"]              = "4.3.0"},
  {["sp"]              = "2.5.0"},
  {["w3emc"]           = "2.10.0"},
  {["gftl-shared"]     = "1.6.1"},
  {["mapl"]            = "2.40.3-esmf-8.6.1b04"},
  {["scotch"]          = "7.0.4"},

DusanJovic-NOAA · 2024-04-18T14:29:25Z

Thanks. I fixed the typo and I'm rerunning the test. But correct modules were loaded despite the typo, which is weird.

RatkoVasic-NOAA · 2024-04-18T14:36:22Z

@DusanJovic-NOAA , NOTE , mapl in this configuration is compiled with esmf@8.6.0 (although name suggests differently). This was just test if esmf@8.6.1b04 is working. mapl@2.40.3 didn't compile with new esmf. I'm looking into this with Matt.
@jkbk2004 ran tests on Hercules successfully, check with him if he ran same tests as you are running.

DusanJovic-NOAA · 2024-04-18T15:10:37Z

cpld_control_p8 is still crashing.

RatkoVasic-NOAA · 2024-04-18T15:12:20Z

@DusanJovic-NOAA What is difference between your and Jong's run?

DusanJovic-NOAA · 2024-04-18T15:14:09Z

I don't know. My working directory is here: /work/noaa/fv3-cam/djovic/ufs/e861/ufs-weather-model.

Maybe the fact that @jkbk2004 used mapl 2.40.3-esmf-8.5.0? Not mapl 2.40.3-esmf-8.6.1b04

RatkoVasic-NOAA · 2024-04-18T15:21:37Z

You can try with this ufs_common:

  {["jasper"]          = "2.0.32"},
  {["zlib"]            = "1.2.13"},
  {["libpng"]          = "1.6.37"},
  {["hdf5"]            = "1.14.0"},
  {["netcdf-c"]        = "4.9.2"},
  {["netcdf-fortran"]  = "4.6.1"},
  {["parallelio"]      = "2.5.10"},
  {["esmf"]            = "8.6.1b04"},
  {["fms"]             = "2023.04"},
  {["bacio"]           = "2.4.1"},
  {["crtm"]            = "2.4.0"},
  {["g2"]              = "3.4.5"},
  {["g2tmpl"]          = "1.10.2"},
  {["ip"]              = "4.3.0"},
  {["sp"]              = "2.5.0"},
  {["w3emc"]           = "2.10.0"},
  {["gftl-shared"]     = "1.6.1"},
  {["mapl"]            = "2.40.3-esmf-8.5.0"},
  {["scotch"]          = "7.0.4"},

Also:

hercules: /work/noaa/epic/jongkim/UFS-RT/hercules/pr-2093/tests> git remote -v
origin  https://github.com/RatkoVasic-NOAA/ufs-weather-model (fetch)
origin  https://github.com/RatkoVasic-NOAA/ufs-weather-model (push)
hercules: /work/noaa/epic/jongkim/UFS-RT/hercules/pr-2093/tests> git branch
* ss-160

DusanJovic-NOAA · 2024-04-18T15:23:19Z

I can, but that will load esmf-8.5.0 not 8.6.1. And we should test 8.6.1, we know 8.5.0 works fine.

DusanJovic-NOAA · 2024-04-18T15:24:10Z

What's the issue with compiling mapl using esmf 8.6.1?

DusanJovic-NOAA · 2024-04-18T15:26:54Z

Isn't ESMF 8.6.x backward compatible with the previous release, ESMF 8.5.0?

climbfuji · 2024-04-18T15:31:46Z

I think the ESMF target definition has changed, which requires updates to the mapl cmake config (lowercase vs uppercase or something like that).

DusanJovic-NOAA · 2024-04-18T15:51:12Z

If update in the ESMF requires update in the MAPL then we should also update it. We should not be compiling 2.40.3 if it doesn't work with the latest ESMF.

climbfuji · 2024-04-18T15:53:10Z

If update in the ESMF requires update in the MAPL then we should also update it. We should not be compiling 2.40.3 if it doesn't work with the latest ESMF.

We are waiting for that tag (and they were hoping to wait for an official release of esmf as opposed to a beta snapshot ...)

I'll ping our NASA colleagues and ask them to create the tag. Will let you know when I hear back.

DusanJovic-NOAA · 2024-04-18T16:03:06Z

Hmm. MAPL people are waiting on the official ESMF release. And ESMF people are waiting on us to test a beta snapshot before they make a release, and we are waiting on MAPL tag, that is waiting on ESMF. circulus vitiosus.

climbfuji · 2024-04-18T16:04:24Z

Hmm. MAPL people are waiting on the official ESMF release. And ESMF people are waiting on us to test a beta snapshot before they make a release, and we are waiting on MAPL tag, that is waiting on ESMF. circulus vitiosus.

GMAO says the tag will be ready next week

mathomp4 · 2024-04-18T16:12:10Z

I hope I can make a new release of MAPL next week. But because of the requirement for ESMF 8.6.1 (beta or not), I'll need to build new libraries on all our clusters, etc. so that our devs don't have issues building with it (since MAPL will require ESMF 8.6.1).

Now, if you are wanting to test, you could try out this commit GEOS-ESM/MAPL@5f91a5c. This is my current draft PR (see GEOS-ESM/MAPL#2682). This is where I'm tracking the changes to MAPL.

You can also see my spack testing changes here: spack/spack@develop...mathomp4:spack:feature/mathomp4/test-mapl-build

You'll note it says MAPL v5 only because I wanted to make sure I was "safe" when doing the testing. And you'll note the commit for v5 is different since this was a couple weeks ago.

DusanJovic-NOAA · 2024-04-19T16:40:10Z

I ran the control_p8 test with esmf 8.6.1b04 for 240 hours creating outputs every hour. output_fh line for that configuration is longer than 1024 characters, which means version 8.6.1 will fix #1121.

junwang-noaa · 2024-04-26T14:22:41Z

@uturuncoglu @danrosen25 The ESMF 8.6.1b04 testing is done in UFS and the fix works. Thanks.

mathomp4 · 2024-04-26T17:48:15Z

Note: We are close to getting ESMF 8.6.1b04 in GEOS land. I encountered a fun bug with GCC 13 that we had to figure out today, but hopefully next week I can release MAPL 2.46 and all is well.

junwang-noaa added the enhancement New feature or request label Apr 8, 2024

ulmononian mentioned this issue Apr 9, 2024

[INSTALL]: mapl@2.46.2 with esmf@8.6.1 (spack-stack-1.8.0) JCSDA/spack-stack#1062

Closed

14 tasks

junwang-noaa added this to Model infrastructure development FY2024Q3 Apr 12, 2024

junwang-noaa assigned DusanJovic-NOAA Apr 12, 2024

mathomp4 mentioned this issue Apr 18, 2024

Update to ESMF 8.6.1b04 GEOS-ESM/ESMA-Baselibs#181

Closed

jkbk2004 added the EPIC Support Requested label Apr 18, 2024

junwang-noaa closed this as completed Apr 26, 2024

github-project-automation bot moved this to Done in Model infrastructure development FY2024Q3 Apr 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test ESMF 8.6.1 beta in UFS weather model #2230

Test ESMF 8.6.1 beta in UFS weather model #2230

junwang-noaa commented Apr 8, 2024

junwang-noaa commented Apr 8, 2024

jkbk2004 commented Apr 9, 2024

RatkoVasic-NOAA commented Apr 9, 2024

uturuncoglu commented Apr 9, 2024

uturuncoglu commented Apr 17, 2024

RatkoVasic-NOAA commented Apr 17, 2024

uturuncoglu commented Apr 17, 2024

DusanJovic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

jkbk2004 commented Apr 18, 2024 •

edited

Loading

RatkoVasic-NOAA commented Apr 18, 2024

RatkoVasic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

RatkoVasic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

RatkoVasic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024 •

edited

Loading

RatkoVasic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

climbfuji commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

climbfuji commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

climbfuji commented Apr 18, 2024

mathomp4 commented Apr 18, 2024

DusanJovic-NOAA commented Apr 19, 2024

junwang-noaa commented Apr 26, 2024

mathomp4 commented Apr 26, 2024

Test ESMF 8.6.1 beta in UFS weather model #2230

Test ESMF 8.6.1 beta in UFS weather model #2230

Comments

junwang-noaa commented Apr 8, 2024

Description

Solution

Alternatives

Related to

junwang-noaa commented Apr 8, 2024

jkbk2004 commented Apr 9, 2024

RatkoVasic-NOAA commented Apr 9, 2024

uturuncoglu commented Apr 9, 2024

uturuncoglu commented Apr 17, 2024

RatkoVasic-NOAA commented Apr 17, 2024

uturuncoglu commented Apr 17, 2024

DusanJovic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

jkbk2004 commented Apr 18, 2024 • edited Loading

RatkoVasic-NOAA commented Apr 18, 2024

RatkoVasic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

RatkoVasic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

RatkoVasic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024 • edited Loading

RatkoVasic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

climbfuji commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

climbfuji commented Apr 18, 2024

DusanJovic-NOAA commented Apr 18, 2024

climbfuji commented Apr 18, 2024

mathomp4 commented Apr 18, 2024

DusanJovic-NOAA commented Apr 19, 2024

junwang-noaa commented Apr 26, 2024

mathomp4 commented Apr 26, 2024

jkbk2004 commented Apr 18, 2024 •

edited

Loading

DusanJovic-NOAA commented Apr 18, 2024 •

edited

Loading