Update MOM6 templates for coordination with global-workflow updates #1979

JessicaMeixner-NOAA · 2023-11-06T13:16:54Z

PR Author Checklist:

I have linked PR's from all sub-components involved in section below.
I am confirming reviews are completed in ALL sub-component PR's.
I have run the full RT suite on either Hera/Cheyenne AND have attached the log to this PR below this line:
- LOG:
  RegressionTests_hera.log
I have added the list of all failed regression tests to "Anticipated changes" section.
I have filled out all sections of the template.

Description

Updates to MOM6 templates for coordination with recent changes in global-workflow for NOAA-EMC/global-workflow#1944 and for the 5 deg template file to include wave-coupling variables, update the EPBL_LANGMUIR_SCHEME value and update the output to match workflow.

Linked Issues and Pull Requests

Associated UFSWM Issue to close

Closes Updating MOM_input_template_500 with wave coupling configurations #1957

Subcomponent Pull Requests

None

Blocking Dependencies

None

Subcomponents involved:

Anticipated Changes

Input data

No changes are expected to input data.
Changes are expected to input data:
- New input data.
- Updated input data.

The file on hera /scratch1/NCEPDEV/global/glopara/fix/mom6/20220805/500/oceanda_zgrid_25L.nc needs to be added to @[INPUTDATA_ROOT]/MOM6_FIX/500/

Regression Tests:

No changes are expected to any regression test.
Changes are expected to the following tests:
Due to the change in output and the change of EPBL_LANGMUIR_SCHEME in the 5deg MOM input, the cpld_control_c48_intel will change answers.

Libraries

Not Needed
Needed
- Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
- Add issue link from JCSDA/spack-stack following this item

Code Managers Log

This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.
- N/A

Testing Log:

RDHPCS
- Hera
- Orion
- Hercules
- Jet
- Gaea
- Cheyenne
WCOSS2
- Dogwood/Cactus
- Acorn
CI
- Completed
opnReqTest
- N/A
- Log attached to comment

and update 5 deg template to be in line with g-w and OM4 settings

tests/fv3_conf/cpld_control_run.IN

JessicaMeixner-NOAA · 2023-11-28T16:33:29Z

Here's the new log file from hera, comparing against a newly created baseline:
RegressionTests_hera.log

jkbk2004 · 2023-12-04T18:40:06Z

@JessicaMeixner-NOAA Can we delay this pr one or two days more? RRFS operation application requires to merge #2021 ASAP. This week's schedule is not that busy. So, I think we can commit this pr tomorrow or 12/06 right after Dusan's PR.

JessicaMeixner-NOAA · 2023-12-04T18:49:59Z

Please prioritize the work needed for RRFS and thank you for communicating the reasons for the delay.

jkbk2004 · 2023-12-06T19:09:58Z

@JessicaMeixner-NOAA can you sync up branch? we can working on this pr.

JessicaMeixner-NOAA · 2023-12-06T21:21:33Z

@JessicaMeixner-NOAA can you sync up branch? we can working on this pr.

@jkbk2004 this branch has been sync'd.

epic-cicd-jenkins · 2023-12-07T04:11:14Z

Jenkins-ci ORTs passed

zach1221 · 2023-12-07T18:57:25Z

Testing is complete, let's move to review.

zach1221 · 2023-12-07T19:15:17Z

@JessicaMeixner-NOAA looks like there is one outdated conversation. Can you resolve it for us, please?

DeniseWorthen · 2023-12-07T19:15:59Z

I don't understand the hercules log. Why is control_wrtGauss_netcdf_parallel_intel showing up as test 235? It should be around test #30. It also shows up as a different rt_ directory. Is this a appended log? If so, the original log showing test failures should not be removed. If a second log is appended, it should clearly indicate that it is from a re-run.

Test 235 control_wrtGauss_netcdf_parallel_intel PASS

The test seems to have failed at the initial run, but does not show as a "failure"

ufs-weather-model/tests/logs/RegressionTests_hercules.log

Lines 1976 to 1983 in aa7dea8

    
           baseline dir = /work/noaa/epic/hercules/UFS-WM_RT/NEMSfv3gfs/develop-20231206/control_wrtGauss_netcdf_parallel_intel 
        
           working dir  = /work2/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_1844383/control_wrtGauss_netcdf_parallel_intel 
        
           Checking test 034 control_wrtGauss_netcdf_parallel_intel results .... 
        
            Comparing sfcf000.nc .........OK 
        
            Comparing sfcf024.nc .........OK 
        
            Comparing atmf000.nc .........OK 
        
            Comparing atmf024.nc ............ALT CHECK......ERROR

The RT seems to simply skip the reporting line? There should be a line like "Test 034 control_wrtGauss_netcdf_parallel_intel FAIL"

JessicaMeixner-NOAA · 2023-12-07T19:18:23Z

@JessicaMeixner-NOAA looks like there is one outdated conversation. Can you resolve it for us, please?

done

zach1221 · 2023-12-07T19:24:57Z

I don't understand the hercules log. Why is control_wrtGauss_netcdf_parallel_intel showing up as test 235? It should be around test #30. It also shows up as a different rt_ directory. Is this a appended log? If so, the original log showing test failures should not be removed. If a second log is appended, it should clearly indicate that it is from a re-run.

Test 235 control_wrtGauss_netcdf_parallel_intel PASS

The test seems to have failed at the initial run, but does not show as a "failure"

ufs-weather-model/tests/logs/RegressionTests_hercules.log

Lines 1976 to 1983 in aa7dea8

baseline dir = /work/noaa/epic/hercules/UFS-WM_RT/NEMSfv3gfs/develop-20231206/control_wrtGauss_netcdf_parallel_intel

working dir = /work2/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_1844383/control_wrtGauss_netcdf_parallel_intel

Checking test 034 control_wrtGauss_netcdf_parallel_intel results ....

Comparing sfcf000.nc .........OK

Comparing sfcf024.nc .........OK

Comparing atmf000.nc .........OK

Comparing atmf024.nc ............ALT CHECK......ERROR

The RT seems to simply skip the reporting line? There should be a line like "Test 034 control_wrtGauss_netcdf_parallel_intel FAIL"

Yes, it's the nccmp failure on Hercules. I have to run that case a second time to get it to pass. I did clean up the combined logs afterwards, so going forward I won't.

DeniseWorthen · 2023-12-07T19:59:03Z

@zach1221 OK, thanks. I'm also concerned though that there is an actual problem w/ the RT failure detection. I assume you didn't remove the line that reported the original test 034 failed "Test 034 control_wrtGauss_netcdf_parallel_intel FAIL"?

zach1221 · 2023-12-07T20:11:10Z

@zach1221 OK, thanks. I'm also concerned though that there is an actual problem w/ the RT failure detection. I assume you didn't remove the line that reported the original test 034 failed "Test 034 control_wrtGauss_netcdf_parallel_intel FAIL"?

No I didn't remove the actual failed 034 control_wrtGauss_netcdf_parallel_intel results piece, that should still in there.

DeniseWorthen · 2023-12-07T20:14:39Z

@junwang-noaa It seems we do have an issue w/ failure detection. It seems that when we get the "ALT CHECK ERROR" failure, the job is not always correctly reported.

zach1221 · 2023-12-07T20:37:42Z

Ok re-pushed the original logs without any edits.

DeniseWorthen · 2023-12-07T20:45:33Z

@zach1221 Thanks for posting the original logs. I see that at the end, the test 034 is noted as failed, but at the point where the files are being compared, it doesn't catch the fact that the comparison gives an error. It seems like after this line

Comparing atmf024.nc ............ALT CHECK......ERROR

We should get a report, something like

Test 034 control_wrtGauss_netcdf_parallel_intel FAIL

jkbk2004

I approve as new Hercules log has enough information for the failed case.

zach1221 · 2023-12-07T20:52:15Z

@zach1221 Thanks for posting the original logs. I see that at the end, the test 034 is noted as failed, but at the point where the files are being compared, it doesn't catch the fact that the comparison gives an error. It seems like after this line

Comparing atmf024.nc ............ALT CHECK......ERROR

We should get a report, something like

Test 034 control_wrtGauss_netcdf_parallel_intel FAIL

Ok, yah that's not happening it looks like. I can create a issue for it.

DeniseWorthen · 2023-12-07T21:11:42Z

tests/tests/cpld_control_c48

@@ -106,6 +106,8 @@ export CHLCLIM=''
 export FRUNOFF=''
 export MOM6_RIVER_RUNOFF=False
 export MOM6_RESTART_SETTING=r
+export MOM6_DIAG_COORD_DEF_Z_FILE=oceanda_zgrid_25L.nc


@JessicaMeixner-NOAA Does this MOM6_DIAG_COORD_DEF_Z_FILE need to be added to the cpld_warmstart_c48 and cpld_restart_c48? Those tests are currently commented out (I need to recreate the warmstart files that they use). Will we need this variable set for those two tests?

@DeniseWorthen good catch, yes you will want to set that for the other C48 tests.

JessicaMeixner-NOAA added 3 commits November 2, 2023 19:00

Update MOM templates to be in sync with recent additions to g-w

da68dd7

and update 5 deg template to be in line with g-w and OM4 settings

Merge remote-tracking branch 'UFS/develop' into feature/mom6sync

900b716

update datm MOM6 vars

8075263

junwang-noaa reviewed Nov 6, 2023

View reviewed changes

tests/fv3_conf/cpld_control_run.IN Outdated Show resolved Hide resolved

point to g-w fix instead of my linked copy

8bebbf6

BrianCurtis-NOAA added Baseline Updates Current baselines will be updated. Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. New Input Data Req'd This PR requires new data to be sync across platforms labels Nov 9, 2023

JessicaMeixner-NOAA added 6 commits November 9, 2023 13:07

Merge branch 'ufs-community:develop' into feature/mom6sync

00ff646

Merge branch 'ufs-community:develop' into feature/mom6sync

7bff903

Merge branch 'ufs-community:develop' into feature/mom6sync

c671115

Merge branch 'ufs-community:develop' into feature/mom6sync

d1e0d52

Merge branch 'ufs-community:develop' into feature/mom6sync

547d092

Merge branch 'ufs-community:develop' into feature/mom6sync

e6bdd5b

JessicaMeixner-NOAA commented Nov 27, 2023

View reviewed changes

tests/fv3_conf/cpld_control_run.IN Outdated Show resolved Hide resolved

Revert manual copying of file now that new file is in input directory

80f8126

JessicaMeixner-NOAA added 3 commits November 30, 2023 08:39

Merge branch 'ufs-community:develop' into feature/mom6sync

8aa8848

Merge branch 'ufs-community:develop' into feature/mom6sync

4dde4c8

Merge branch 'ufs-community:develop' into feature/mom6sync

028065e

Merge branch 'ufs-community:develop' into feature/mom6sync

6c7e269

Add new BL_DATE

368d9cf

zach1221 added the jenkins-ci Jenkins CI: ORT build/test on docker container label Dec 7, 2023

zach1221 added 2 commits December 7, 2023 08:20

add hercules RT logs: passed

aa7dea8

add orion RT logs: passed

9ffcf1a

FernandoAndrade-NOAA and others added 3 commits December 7, 2023 16:19

add jet RT passed

3576ecf

WCOSS2 RT Log

7639b8f

Acorn RT Log

544c7ef

zach1221 requested review from jkbk2004, DeniseWorthen and BrianCurtis-NOAA December 7, 2023 18:57

jkbk2004 previously approved these changes Dec 7, 2023

View reviewed changes

BrianCurtis-NOAA previously approved these changes Dec 7, 2023

View reviewed changes

add hercules RT logs: passed

61404c0

zach1221 dismissed stale reviews from BrianCurtis-NOAA and jkbk2004 via 61404c0 December 7, 2023 20:37

zach1221 requested review from jkbk2004 and BrianCurtis-NOAA December 7, 2023 20:37

BrianCurtis-NOAA approved these changes Dec 7, 2023

View reviewed changes

jkbk2004 approved these changes Dec 7, 2023

View reviewed changes

zach1221 merged commit 5be4f33 into ufs-community:develop Dec 7, 2023

DeniseWorthen reviewed Dec 7, 2023

View reviewed changes

This was referenced Mar 11, 2024

UFS-dev PR#139 NCAR/ufs-weather-model#123

Merged

UFS-dev PR#153 NCAR/ufs-weather-model#125

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update MOM6 templates for coordination with global-workflow updates #1979

Update MOM6 templates for coordination with global-workflow updates #1979

JessicaMeixner-NOAA commented Nov 6, 2023 •

edited by FernandoAndrade-NOAA

Loading

JessicaMeixner-NOAA commented Nov 28, 2023

jkbk2004 commented Dec 4, 2023

JessicaMeixner-NOAA commented Dec 4, 2023

jkbk2004 commented Dec 6, 2023

JessicaMeixner-NOAA commented Dec 6, 2023

epic-cicd-jenkins commented Dec 7, 2023

zach1221 commented Dec 7, 2023

zach1221 commented Dec 7, 2023

DeniseWorthen commented Dec 7, 2023 •

edited

Loading

JessicaMeixner-NOAA commented Dec 7, 2023

zach1221 commented Dec 7, 2023

DeniseWorthen commented Dec 7, 2023

zach1221 commented Dec 7, 2023

DeniseWorthen commented Dec 7, 2023

zach1221 commented Dec 7, 2023

DeniseWorthen commented Dec 7, 2023

jkbk2004 left a comment

zach1221 commented Dec 7, 2023

DeniseWorthen Dec 7, 2023

JessicaMeixner-NOAA Dec 7, 2023

Update MOM6 templates for coordination with global-workflow updates #1979

Update MOM6 templates for coordination with global-workflow updates #1979

Conversation

JessicaMeixner-NOAA commented Nov 6, 2023 • edited by FernandoAndrade-NOAA Loading

PR Author Checklist:

Description

Linked Issues and Pull Requests

Associated UFSWM Issue to close

Subcomponent Pull Requests

Blocking Dependencies

Subcomponents involved:

Anticipated Changes

Input data

Regression Tests:

Libraries

Testing Log:

JessicaMeixner-NOAA commented Nov 28, 2023

jkbk2004 commented Dec 4, 2023

JessicaMeixner-NOAA commented Dec 4, 2023

jkbk2004 commented Dec 6, 2023

JessicaMeixner-NOAA commented Dec 6, 2023

epic-cicd-jenkins commented Dec 7, 2023

zach1221 commented Dec 7, 2023

zach1221 commented Dec 7, 2023

DeniseWorthen commented Dec 7, 2023 • edited Loading

JessicaMeixner-NOAA commented Dec 7, 2023

zach1221 commented Dec 7, 2023

DeniseWorthen commented Dec 7, 2023

zach1221 commented Dec 7, 2023

DeniseWorthen commented Dec 7, 2023

zach1221 commented Dec 7, 2023

DeniseWorthen commented Dec 7, 2023

jkbk2004 left a comment

Choose a reason for hiding this comment

zach1221 commented Dec 7, 2023

DeniseWorthen Dec 7, 2023

Choose a reason for hiding this comment

JessicaMeixner-NOAA Dec 7, 2023

Choose a reason for hiding this comment

JessicaMeixner-NOAA commented Nov 6, 2023 •

edited by FernandoAndrade-NOAA

Loading

DeniseWorthen commented Dec 7, 2023 •

edited

Loading