Send Evap rate from ATM to OCN

[DeniseWorthen]

Description

Adds the evaporative flux to the export fields available from ATM; CMEPS uses this flux instead of internally converting from the latent heat flux.

This is an incremental update related to #1807. Adding the evaporative flux change created non-B4B answers so I would like to commit this PR first since the other changes in #1807 result only in changes to the mediator restart files but otherwise reproduce.

A test case was run using the cpld_control_nowave_noaero_p8 test. Instantaneous mediator history files were written for the OCN component. At the first pass through the coupling sequence, the difference between the evaporative flux sent to the ocean is ~O-15. This is due to the difference in converting from latent->evap in the atmosphere vs doing the conversion in CMEPS. The difference between the current code and this change at the end of the first coupling loop, scaled by 1.0e14 shows random differences:

At the end of 24 hour, the evaporative flux sent to the OCN and the difference between the develop branch and the feature branch is shown below. The differences are small compared to the field value.

Input data additions/changes

• No changes are expected to input data.
• Changes are expected to input data:
  • New input data.
  • Updated input data.

Anticipated changes to regression tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

RegressionTests_hera.log

Coupled tests will change baselines due to round-off level differences associated moving the latent->evap conversion from CMEPS to the ATM. No baselines are expected to change DATM tests since they the evaporative flux calculated in the mediator's aoflux routine.

The following test fail on Hera, as expected.

1:138:Test 001 cpld_control_p8_mixedmode_intel FAIL
2:209:Test 002 cpld_control_gfsv17_intel FAIL
3:281:Test 003 cpld_control_p8_intel FAIL
4:353:Test 005 cpld_control_qr_p8_intel FAIL
5:413:Test 007 cpld_2threads_p8_intel FAIL
6:473:Test 008 cpld_decomp_p8_intel FAIL
7:533:Test 009 cpld_mpi_p8_intel FAIL
8:605:Test 010 cpld_control_ciceC_p8_intel FAIL
9:665:Test 011 cpld_control_c192_p8_intel FAIL
10:720:Test 013 cpld_bmark_p8_intel FAIL
11:791:Test 015 cpld_control_noaero_p8_intel FAIL
12:860:Test 016 cpld_control_nowave_noaero_p8_intel FAIL
13:920:Test 017 cpld_debug_p8_intel FAIL
14:979:Test 018 cpld_debug_noaero_p8_intel FAIL
15:1105:Test 020 cpld_control_c48_intel FAIL
16:1107:Test 021 cpld_warmstart_c48_intel FAIL
17:1179:Test 023 cpld_control_p8_faster_intel FAIL
18:1250:Test 024 cpld_control_pdlib_p8_intel FAIL
19:1309:Test 027 cpld_debug_pdlib_p8_intel FAIL
24:7117:Test 234 cpld_control_p8_gnu FAIL
25:7186:Test 235 cpld_control_nowave_noaero_p8_gnu FAIL
26:7246:Test 236 cpld_debug_p8_gnu FAIL
27:7317:Test 237 cpld_control_pdlib_p8_gnu FAIL
28:7376:Test 238 cpld_debug_pdlib_p8_gnu FAIL

Four hafs tests fail because the special (fill) value in the mediator files is no longer being scaled by rtime prior to export by ATM

20:4203:Test 134 hafs_regional_atm_ocn_intel FAIL
21:4237:Test 136 hafs_regional_atm_ocn_wav_intel FAIL
22:4739:Test 151 hafs_regional_docn_intel FAIL
23:4754:Test 152 hafs_regional_docn_oisst_intel FAIL

For the hafs tests, all files are B4B except for

957:rt_134_hafs_regional_atm_ocn_intel.log:9: Comparing ufs.hafs.cpl.hi.2019-08-29-21600.nc ............ALT CHECK......NOT OK
958:rt_134_hafs_regional_atm_ocn_intel.log:10: Comparing ufs.hafs.cpl.r.2019-08-29-21600.nc ............ALT CHECK......NOT OK
959:rt_136_hafs_regional_atm_ocn_wav_intel.log:12: Comparing ufs.hafs.cpl.r.2019-08-29-21600.nc ............ALT CHECK......NOT OK
960:rt_151_hafs_regional_docn_intel.log:7: Comparing ufs.hafs.cpl.hi.2019-08-29-21600.nc ............ALT CHECK......NOT OK
961:rt_151_hafs_regional_docn_intel.log:8: Comparing ufs.hafs.cpl.r.2019-08-29-21600.nc ............ALT CHECK......NOT OK
962:rt_152_hafs_regional_docn_oisst_intel.log:7: Comparing ufs.hafs.cpl.hi.2019-08-29-21600.nc ............ALT CHECK......NOT OK
963:rt_152_hafs_regional_docn_oisst_intel.log:8: Comparing ufs.hafs.cpl.r.2019-08-29-21600.nc ............ALT CHECK......NOT OK

Subcomponents involved:

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Library Updates/Changes

• Not Needed
• Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
• Add issue link from JCSDA/spack-stack following this item

Combined with PR's (If Applicable):

Commit Queue Checklist:

• Link PR's from all sub-components involved in section below
• Confirm reviews completed in ALL sub-component PR's
• Add all appropriate labels to this PR.
• Run full RT suite on either Hera/Cheyenne AND attach log to a PR comment.
• Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

• partial fix for CMEPS is forced to use custom merges because sign of field imported from ATM is wrong or unit conversion is required NOAA-EMC/CMEPS#93
• depends on use evap from atm NOAA-EMC/CMEPS#96
• depends on Export evaporative flux from ATM NOAA-EMC/fv3atm#672
• creates re-enable c48-5deg warmstart test #1822

Testing Day Checklist:

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
• Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

• RDHPCS
  • Hera
  • Orion
  • Jet
  • Gaea
  • Cheyenne
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[jkbk2004]

It looks like cpld_warmstart_c48_intel crashed (not baseline check failure). @DeniseWorthen can you confirm?

[DeniseWorthen]

@jkbk2004 I suspect I know the issue. Please provide the run directory location so I can check.

[jkbk2004]

/scratch1/NCEPDEV/stmp2/Jong.Kim/FV3_RT/rt_278330/cpld_warmstart_c48_intel

[DeniseWorthen]

Checking either the mediator.log or PET log would have shown the reason:

mediator.log:

(med.F90:DataInitialize) read_restart =  T
 (med_io_read_init_iodesc) ERROR: atmImp_Faxa_evap is not present, aborting

PET00.ESMF_LogFile

20230704 125749.682 INFO             PET00 (med_io_read_init_iodesc) ERROR: atmImp_Faxa_evap is not present, aborting

This is the only coupled test which includes a warmstart test (meaning it starts up from staged ICs generated by an actual model run), which was a request of the G-W developers when the super-low res tests were committed.

Fixing this test failure will require a new set of updated ICs for this test. However, since the pre-staged files for the warm-start test will need to be updated again for PR #1807, I would propose we turn this test off until #1807 is up for commit.

[DeniseWorthen]

@jkbk2004 I just realized I need to turn off the c48-5deg restart test also. Making change now.

[jkbk2004]

@BrianCurtis-NOAA just to confirm. wcoss2 test will come, right? We can still try to combine in #1810 with a quick follow-up when hera comes back online this evening. And still we can merge in this pr first thing tomorrow morning.

[BrianCurtis-NOAA]

@BrianCurtis-NOAA just to confirm. wcoss2 test will come, right? We can still try to combine in #1810 with a quick follow-up when hera comes back online this evening. And still we can merge in this pr first thing tomorrow morning.

WCOSS2 still has an issue. I'll get back to it in the AM with hopes to finish early on.

[jkbk2004]

@BrianCurtis-NOAA I tested #1810 on hera. No problem on hera. Once you confirm with wcoss2, then we can combine #1810 back into this pr. @DeniseWorthen FYI

[DeniseWorthen]

Since testing is complete, it would be better to add #1810 to the next PR.

[BrianCurtis-NOAA]

On WCOSS2, the baselines generated for hrrr_control that 4 tests are comparing against, are failing comparisons for different restart tiles each time a new baseline is generated. I'm assuming at this point it's a guessing game when a new baseline for that test will not break at one tile in comparison. I do not know why WCOSS2 is the only platform having this issue.

@DeniseWorthen have we seen this issue before? I recall something along these lines.

[jkbk2004]

@BrianCurtis-NOAA it might be worth to make a quick sanity check for hrrr_control with develop branch on wcoss2.

[BrianCurtis-NOAA]

@BrianCurtis-NOAA it might be worth to make a quick sanity check for hrrr_control with develop branch on wcoss2.

Sure give me a little bit to run against current baselines, and then to create new baselines and compare against those.

[jkbk2004]

@DeniseWorthen fv3 pr was merged.

[SadeghTabas-NOAA]

Could you please update the gitmodules and pointer?