update CMEPS to allow bilinear ATM<->WAV mapping for global coupled application; utilize custom restart names for WW3 (was #1684)

[DeniseWorthen]

Description

Changes the mapping of state fields between ATM and WAV for the coupled model to use bilinear with nearest-source-to-destination filling.

A test case was run using the cpld_control_p8 test for 24 hours with and w/o this change. The following figure shows the difference in U10M along ~62S, from 90W:40W imported by the WAV model with the current mapping (mapnstod_consf, black line) vs the change in this PR (mapbilnr_nstod, red line).

The impact on the Z0 imported by the ATM at ~0.5S,6E (on the coast of Brazil, where a large difference in Z0 is seen) for the current mapping (red) vs this PR (black) is shown below

In the bmark test, the difference in Z0 imported by the ATM on tile 1 (scaled by 1.0e4) after 6 hours is shown below

Top of commit queue on: TBD

Input data additions/changes

• No changes are expected to input data.
• There will be new input data.
• Input data will be updated.

Anticipated changes to regression tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

This will change baselines for all coupled tests using ATM-WAV coupling. HAFS wave tests do not change since their mapping is mapfillv_bilnr and does not change.

Full RTs on cheyenne show the following:

GNU:

FAILED TESTS:
Test cpld_control_p8 047 failed in check_result failed
Test cpld_control_p8 047 failed in run_test failed
Test cpld_debug_p8 049 failed in check_result failed
Test cpld_debug_p8 049 failed in run_test failed

INTEL:

FAILED TESTS:
Test cpld_control_p8_mixedmode 001 failed in check_result failed
Test cpld_control_p8_mixedmode 001 failed in run_test failed
Test cpld_control_gfsv17 002 failed in check_result failed
Test cpld_control_gfsv17 002 failed in run_test failed
Test cpld_control_p8 003 failed in check_result failed
Test cpld_control_p8 003 failed in run_test failed
Test cpld_control_qr_p8 005 failed in check_result failed
Test cpld_control_qr_p8 005 failed in run_test failed
Test cpld_2threads_p8 007 failed in check_result failed
Test cpld_2threads_p8 007 failed in run_test failed
Test cpld_decomp_p8 008 failed in check_result failed
Test cpld_decomp_p8 008 failed in run_test failed
Test cpld_mpi_p8 009 failed in check_result failed
Test cpld_mpi_p8 009 failed in run_test failed
Test cpld_control_ciceC_p8 010 failed in check_result failed
Test cpld_control_ciceC_p8 010 failed in run_test failed
Test cpld_control_c192_p8 011 failed in check_result failed
Test cpld_control_c192_p8 011 failed in run_test failed
Test cpld_control_noaero_p8 013 failed in check_result failed
Test cpld_control_noaero_p8 013 failed in run_test failed
Test cpld_debug_p8 015 failed in check_result failed
Test cpld_debug_p8 015 failed in run_test failed
Test cpld_debug_noaero_p8 016 failed in check_result failed
Test cpld_debug_noaero_p8 016 failed in run_test failed

RegressionTests_cheyenne.gnu.log
RegressionTests_cheyenne.intel.log

Subcomponents involved:

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Combined with PR's (If Applicable):

• PR utilize custom restart names for WW3 #1684

Commit Queue Checklist:

• Link PR's from all sub-components involved
• Confirm reviews completed in sub-component PR's
• Add all appropriate labels to this PR.
• Run full RT suite on either Hera/Cheyenne with both Intel/GNU compilers
• Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

• Fixes map using bilinear + nearest source-to-destination for ATM-WAV NOAA-EMC/CMEPS#85
• Depends on use bilinear mapping for ATM-WAV fields NOAA-EMC/CMEPS#86
• Associated with Issue Add capability to run WW3 on unstructured mesh with domain decomposition #1556
• Fixes For a "continue" run, WW3 should not start unless a restart is present #1590

Testing Day Checklist:

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
• Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

• RDHPCS
  • Intel
    • Hera
    • Orion
    • Jet
    • Gaea
    • Cheyenne
  • GNU
    • Hera
    • Cheyenne
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[DeniseWorthen]

@jkbk2004 I had to recreate my CMEPS update PR and then re-merged the ww3 restart name change. This PR is equivalent to the previous PR #1685

[jkbk2004]

Sure! @zach1221 @BrianCurtis-NOAA this pr replaces #1685. I am adding new bl date.

[jkbk2004]

@DeniseWorthen two cases fail on jet: hafs_regional_datm_cdeps and regional_noquilt. I think regional_noquilt time out issue. But hafs_regional_datm_cdeps shows something hycom side in out file: error in zaiopf - can't open unit 13. can you check /lfs4/HFIP/h-nems/Jong.Kim/RT_RUNDIRS/Jong.Kim/FV3_RT/rt_52646/hafs_regional_datm_cdeps/out ? All ran ok on hera though.

[zach1221]

Please see jenkins-ci logs attached. ORTs passed.
ufs-weather-model » ort-docker-pipeline » PR-1692 #1 Console [Jenkins].pdf

[DeniseWorthen]

@jkbk2004 I will look, but I suspect a jet system issue. This PR does not touch anything in hycom or cdeps.

[DeniseWorthen]

There is a timeout message in the hafs_regional_datm_cdeps err log also. It seems to have hung. But I will run the test independently to check.

[jkbk2004]

There is a timeout message in the hafs_regional_datm_cdeps err log also. It seems to have hung. But I will run the test independently to check.

I agree the time out case usually ended up with that hycom error message. Same thing occasionally happens even with develop branch. I think we need to turn those cases off on jet.

[BrianCurtis-NOAA]

We can't keep turning off tests that don't work. The issues need to be addressed with the machine admins and/or fixes in the UFSWM.

[DeniseWorthen]

@BrianCurtis-NOAA It does seem that jet is a flakey platform in general and it is difficult for either us or sys admins to debug intermittent issues. Generally we have the "2 tries" but even that doesn't seem to be enough for jet.

[DeniseWorthen]

I don't think this causes the time-outs, but we should reduce the debug flags here. There is no reason for those to be anything other than 0 as a default. The higher settings simply report min/max values for import/export states for example. They do not actually turn on any sort of compiler-debug options.

nems.configure.hafs_atm_docn.IN:93:  dbug_flag = 20
nems.configure.hafs_atm_ocn.IN:122:  dbug_flag = 20
nems.configure.hafs_atm_ocn_wav.IN:134:  dbug_flag = 6
nems.configure.hafs_atm_wav.IN:96:  dbug_flag = 6

[DeniseWorthen]

The current jet log shows this test taking almost 24 minutes. On Cheyenne.intel, it takes a 21 minutes. I would suggest we reduce the fhmax for this test to 12 or even 6. It is currently 24 but all the other HAFS tests are only 6 hours.

[jkbk2004]

The current jet log shows this test taking almost 24 minutes. On Cheyenne.intel, it takes a 21 minutes. I would suggest we reduce the fhmax for this test to 12 or even 6. It is currently 24 but all the other HAFS tests are only 6 hours.

Sounds like consistent 6 hour across hafs tests makes a sense. same way for regional_noquilt, right?

[DeniseWorthen]

I think reducing fhmax may work sometimes, but I just ran a the hafs_regional_datm_cdeps case on Jet w/ fhmax=12 and it didn't even finish 12 hours. It was a half-hour short when it ran out of wall clock. I just think there are just system issues w/ Jet.

[jkbk2004]

On jet, these 5 cases are continuously facing the time-out issue: regional_noquilt hafs_regional_datm_cdeps regional_wofs regional_atmaq regional_atmaq_faster. @DeniseWorthen @BrianCurtis-NOAA sounds like turning off all regional aqm cases on jet, too much? Other than that, Jet RT log is available to push.

[DeniseWorthen]

I already pushed a commit to remove regional_noquilt and hafs_regional_datm_cdeps for jet.intel. Now the other regional tests are timing-out?

[jkbk2004]

I already pushed a commit to remove regional_noquilt and hafs_regional_datm_cdeps for jet.intel. Now the other regional tests are timing-out?

Yes, I keep seeing those 3 other cases continue to hit the time limit. I vote to turn those off if possible. We will create an issue and revisit the issue along with that.

[DeniseWorthen]

I will push a commit to remove regional_wofs, regional_atmaq, regional_atmaq_faster from jet.intel.

[jkbk2004]

I am writing an issue to address the jet time-out cases. We can start merging process. @DeniseWorthen @BrianCurtis-NOAA Can you go ahead to merge in CMEPS PR?

[jkbk2004]

issue was created #1695

[jkbk2004]

All set! @BrianCurtis-NOAA @SadeghTabas-NOAA please, go ahead to approve the pr.