Wrapper PR for "Remove optional arguments from CCPP metadata, remove effective radii computation from Thompson MP" and "Fix/improve logic for convective transportable tracers, add GFS_checktracer debugging routine, correct spelling "janic" --> "janjic", fix inconsistent types related with noahmp and log functions", "Modify opnReqTest related scripts for fv3_gsd test" #892

climbfuji · 2021-11-02T15:47:38Z

PR Checklist

Ths PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.
This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR
An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
are specified below.
If new or updated input data is required by this PR, it is clearly stated in the text of the PR.

Description

This PR combines #875, #887, #890:

Update submodule pointers for fv3atm and ccpp-physics for the changes described in the associated PRs below: fix/improve logic for convective transportable tracers in CCPP, add GFS_checktracer debugging routine, correct spelling "janic" --> "janjic" in several CCPP standard names.

#887:

Update the submodule pointers for fv3atm, ccpp-framework and ccpp-physics to:

remove support for the optional attribute in the CCPP framework and metadata
remove invalid optional and intent attributes from host model metadata (variable or type definitions)
remove unused code for calculating cloud effective radii from Thompson MP wrapper (this is done in the radiation interstitials)

#890:

Modify operation requirement test related scripts for the fv3_gsd test. All code changes from @MinsukJi-NOAA (thanks for doing this!)

No changes to the regression test results or the input data.

Issue(s) addressed

Fixes NCAR/ccpp-framework#407
Fixes NCAR/ccpp-physics#751

Testing

Regression tests were run repeatedly for the two sets of PRs combined in this PR. For the combined PRs, regression tests were run on gaea against the existing baseline on 2021/11/02, all tests passed.

Full regression tests will be run on all tier-1 platforms when it is time to commit.

Dependencies

Not a dependency, but an associated PR for the CCPP standard names dictionary:

ESCOMP/CCPPStandardNames#23

…nd testing

…r-model into remove_optional_arguments_from_ccpp

…r-model into ccpp_convective_transport_tracers_cleanup

…r-model into remove_optional_arguments_from_ccpp

…r-model into ccpp_convective_transport_tracers_cleanup

…r-model into remove_optional_arguments_from_ccpp

…ithub.com/climbfuji/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

…com/climbfuji/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

…noaa/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

junwang-noaa · 2021-11-03T02:15:17Z

Code changes look good to me, will approve when all the RT tests are done.

on-behalf-of @ufs-community

…github.com/climbfuji/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

BrianCurtis-NOAA · 2021-11-03T17:53:19Z

Automated RT Failure Notification
Machine: gaea
Compiler: intel
Job: RT
Repo location: /lustre/f2/pdata/ncep/emc.nemspara/autort/pr/771598162/20211103154509/ufs-weather-model
Please manually delete: /lustre/f2/scratch/emc.nemspara/FV3_RT/rt_40458
Test cpld_control_c192_p7 006 failed failed
Test cpld_control_c192_p7 006 failed in run_test failed
Test cpld_control_c384_p7 008 failed failed
Test cpld_control_c384_p7 008 failed in run_test failed
Test control_c192 020 failed failed
Test control_c192 020 failed in run_test failed
Test control_c384 021 failed failed
Test control_c384 021 failed in run_test failed
Test control_c384gdas 022 failed failed
Test control_c384gdas 022 failed in run_test failed
Test regional_control 030 failed failed
Test regional_control 030 failed in run_test failed
Test regional_quilt 032 failed failed
Test regional_quilt 032 failed in run_test failed
Test regional_quilt_hafs 034 failed failed
Test regional_quilt_hafs 034 failed in run_test failed
Test regional_quilt_netcdf_parallel 035 failed failed
Test regional_quilt_netcdf_parallel 035 failed in run_test failed
Please make changes and add the following label back:
gaea-intel-RT

climbfuji · 2021-11-03T18:00:04Z

Automated RT Failure Notification
Machine: gaea
Compiler: intel
Job: RT
Repo location: /lustre/f2/pdata/ncep/emc.nemspara/autort/pr/771598162/20211103154509/ufs-weather-model
Please manually delete: /lustre/f2/scratch/emc.nemspara/FV3_RT/rt_40458
Test cpld_control_c192_p7 006 failed failed
Test cpld_control_c192_p7 006 failed in run_test failed
Test cpld_control_c384_p7 008 failed failed
Test cpld_control_c384_p7 008 failed in run_test failed
Test control_c192 020 failed failed
Test control_c192 020 failed in run_test failed
Test control_c384 021 failed failed
Test control_c384 021 failed in run_test failed
Test control_c384gdas 022 failed failed
Test control_c384gdas 022 failed in run_test failed
Test regional_control 030 failed failed
Test regional_control 030 failed in run_test failed
Test regional_quilt 032 failed failed
Test regional_quilt 032 failed in run_test failed
Test regional_quilt_hafs 034 failed failed
Test regional_quilt_hafs 034 failed in run_test failed
Test regional_quilt_netcdf_parallel 035 failed failed
Test regional_quilt_netcdf_parallel 035 failed in run_test failed
Please make changes and add the following label back:
gaea-intel-RT

All these tests timed out. I will add back a 1hr wclock time for gaea and rerun the failed tests ...

on-behalf-of @ufs-community

climbfuji · 2021-11-03T19:06:43Z

@MinsukJi-NOAA Almost all the tests finished, just waiting for snail gaea. Please let me know if you posted those changes somewhere and I didn't see them. Thanks!

MinsukJi-NOAA · 2021-11-03T19:48:07Z

@climbfuji Apologize for the delay. Thanks in advance for making these changes!

diff --git a/tests/opnReqTests/dcp.sh b/tests/opnReqTests/dcp.sh
index b076e8c3..4dae196e 100644
--- a/tests/opnReqTests/dcp.sh
+++ b/tests/opnReqTests/dcp.sh
@@ -2,11 +2,14 @@ set -eu
 source $PATHRT/opnReqTests/std.sh
 
 if [[ $application == 'global' ]]; then
-  #temp=$INPES
-  #INPES=$JNPES
-  #JNPES=$temp
-  INPES=6
-  JNPES=4
+  if [[ $CI_TEST == 'true' ]]; then
+    temp=$INPES
+    INPES=$JNPES
+    JNPES=$temp
+  else
+    INPES=6
+    JNPES=4
+  fi
 elif [[ $application == 'regional' ]]; then
   if [[ $CI_TEST == 'true' ]]; then
     INPES=10

diff --git a/tests/opnReqTests/rst.sh b/tests/opnReqTests/rst.sh
index cdd6ab6f..1ee9d7d4 100644
--- a/tests/opnReqTests/rst.sh
+++ b/tests/opnReqTests/rst.sh
@@ -11,6 +11,14 @@ if [[ $application == 'global' ]]; then
   else
     RESTART_FILE_PREFIX="${SYEAR}${SMONTH}$(printf "%02d" $((SDAY+1))).$(printf "%02d" $(( SHOUR + FHROT - 24 )))0000"
   fi
+
+  if [[ ! -z $NSTF_NAME ]]; then
+    second_value=$(echo $NSTF_NAME | cut -d ',' -f 2)
+    if [[ $second_value -eq 1 ]]; then
+      NSTF_NAME=$(echo $NSTF_NAME | awk 'BEGIN{FS=OFS=","} { $2-=1; print}')
+    fi
+  fi
+
 elif [[ $application == 'regional' ]]; then
   echo "Regional application not yet implemented for restart"
   exit 1

…github.com/climbfuji/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

climbfuji · 2021-11-03T20:09:59Z

@climbfuji Apologize for the delay. Thanks in advance for making these changes!

diff --git a/tests/opnReqTests/dcp.sh b/tests/opnReqTests/dcp.sh
index b076e8c3..4dae196e 100644
--- a/tests/opnReqTests/dcp.sh
+++ b/tests/opnReqTests/dcp.sh
@@ -2,11 +2,14 @@ set -eu
 source $PATHRT/opnReqTests/std.sh
 
 if [[ $application == 'global' ]]; then
-  #temp=$INPES
-  #INPES=$JNPES
-  #JNPES=$temp
-  INPES=6
-  JNPES=4
+  if [[ $CI_TEST == 'true' ]]; then
+    temp=$INPES
+    INPES=$JNPES
+    JNPES=$temp
+  else
+    INPES=6
+    JNPES=4
+  fi
 elif [[ $application == 'regional' ]]; then
   if [[ $CI_TEST == 'true' ]]; then
     INPES=10

diff --git a/tests/opnReqTests/rst.sh b/tests/opnReqTests/rst.sh
index cdd6ab6f..1ee9d7d4 100644
--- a/tests/opnReqTests/rst.sh
+++ b/tests/opnReqTests/rst.sh
@@ -11,6 +11,14 @@ if [[ $application == 'global' ]]; then
  else
    RESTART_FILE_PREFIX="${SYEAR}${SMONTH}$(printf "%02d" $((SDAY+1))).$(printf "%02d" $(( SHOUR + FHROT - 24 )))0000"
  fi
+
+  if [[ ! -z $NSTF_NAME ]]; then
+    second_value=$(echo $NSTF_NAME | cut -d ',' -f 2)
+    if [[ $second_value -eq 1 ]]; then
+      NSTF_NAME=$(echo $NSTF_NAME | awk 'BEGIN{FS=OFS=","} { $2-=1; print}')
+    fi
+  fi
+
elif [[ $application == 'regional' ]]; then
  echo "Regional application not yet implemented for restart"
  exit 1

Thanks, Minsuk. Please check the latest commit.

MinsukJi-NOAA · 2021-11-03T20:27:23Z

@climbfuji NSTF_NAME is correct now, but I still see RESTART failing

baseline dir = /scratch1/NCEPDEV/stmp4/Minsuk.Ji/FV3_OPNREQ_TEST/OPNREQ_TEST/fv3_gsd_std_base
working dir  = /scratch1/NCEPDEV/stmp2/Minsuk.Ji/FV3_OPNREQ_TEST/opnReqTest_39402/fv3_gsd_rst
Checking test rst fv3_gsd results ....
 Comparing atmf012.tile1.nc .........OK
 Comparing atmf012.tile2.nc .........OK
 Comparing atmf012.tile3.nc .........OK
 Comparing atmf012.tile4.nc .........OK
 Comparing atmf012.tile5.nc .........OK
 Comparing atmf012.tile6.nc .........OK
 Comparing RESTART/coupler.res .........OK
 Comparing RESTART/fv_core.res.nc .........OK
 Comparing RESTART/fv_core.res.tile1.nc .........OK
 Comparing RESTART/fv_core.res.tile2.nc .........OK
 Comparing RESTART/fv_core.res.tile3.nc .........OK
 Comparing RESTART/fv_core.res.tile4.nc .........OK
 Comparing RESTART/fv_core.res.tile5.nc .........OK
 Comparing RESTART/fv_core.res.tile6.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile1.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile2.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile3.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile4.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile5.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile6.nc .........OK
 Comparing RESTART/fv_tracer.res.tile1.nc .........OK
 Comparing RESTART/fv_tracer.res.tile2.nc .........OK
 Comparing RESTART/fv_tracer.res.tile3.nc .........OK
 Comparing RESTART/fv_tracer.res.tile4.nc .........OK
 Comparing RESTART/fv_tracer.res.tile5.nc .........OK
 Comparing RESTART/fv_tracer.res.tile6.nc .........OK
 Comparing RESTART/phy_data.tile1.nc .........OK
 Comparing RESTART/phy_data.tile2.nc .........OK
 Comparing RESTART/phy_data.tile3.nc .........OK
 Comparing RESTART/phy_data.tile4.nc .........OK
 Comparing RESTART/phy_data.tile5.nc .........OK
 Comparing RESTART/phy_data.tile6.nc .........OK
 Comparing RESTART/sfc_data.tile1.nc ............ALT CHECK......NOT OK
 Comparing RESTART/sfc_data.tile2.nc ............ALT CHECK......NOT OK
 Comparing RESTART/sfc_data.tile3.nc ............ALT CHECK......NOT OK
 Comparing RESTART/sfc_data.tile4.nc ............ALT CHECK......NOT OK
 Comparing RESTART/sfc_data.tile5.nc ............ALT CHECK......NOT OK
 Comparing RESTART/sfc_data.tile6.nc ............ALT CHECK......NOT OK
 Comparing sfcf012.tile1.nc .........OK
 Comparing sfcf012.tile2.nc .........OK
 Comparing sfcf012.tile3.nc .........OK
 Comparing sfcf012.tile4.nc .........OK
 Comparing sfcf012.tile5.nc .........OK
 Comparing sfcf012.tile6.nc .........OK

  0: The total amount of wall time                        = 32.924707

Test rst fv3_gsd FAIL

climbfuji · 2021-11-03T20:31:47Z

@climbfuji NSTF_NAME is correct now, but I still see RESTART failing

baseline dir = /scratch1/NCEPDEV/stmp4/Minsuk.Ji/FV3_OPNREQ_TEST/OPNREQ_TEST/fv3_gsd_std_base
working dir  = /scratch1/NCEPDEV/stmp2/Minsuk.Ji/FV3_OPNREQ_TEST/opnReqTest_39402/fv3_gsd_rst
Checking test rst fv3_gsd results ....
 Comparing atmf012.tile1.nc .........OK
 Comparing atmf012.tile2.nc .........OK
 Comparing atmf012.tile3.nc .........OK
 Comparing atmf012.tile4.nc .........OK
 Comparing atmf012.tile5.nc .........OK
 Comparing atmf012.tile6.nc .........OK
 Comparing RESTART/coupler.res .........OK
 Comparing RESTART/fv_core.res.nc .........OK
 Comparing RESTART/fv_core.res.tile1.nc .........OK
 Comparing RESTART/fv_core.res.tile2.nc .........OK
 Comparing RESTART/fv_core.res.tile3.nc .........OK
 Comparing RESTART/fv_core.res.tile4.nc .........OK
 Comparing RESTART/fv_core.res.tile5.nc .........OK
 Comparing RESTART/fv_core.res.tile6.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile1.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile2.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile3.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile4.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile5.nc .........OK
 Comparing RESTART/fv_srf_wnd.res.tile6.nc .........OK
 Comparing RESTART/fv_tracer.res.tile1.nc .........OK
 Comparing RESTART/fv_tracer.res.tile2.nc .........OK
 Comparing RESTART/fv_tracer.res.tile3.nc .........OK
 Comparing RESTART/fv_tracer.res.tile4.nc .........OK
 Comparing RESTART/fv_tracer.res.tile5.nc .........OK
 Comparing RESTART/fv_tracer.res.tile6.nc .........OK
 Comparing RESTART/phy_data.tile1.nc .........OK
 Comparing RESTART/phy_data.tile2.nc .........OK
 Comparing RESTART/phy_data.tile3.nc .........OK
 Comparing RESTART/phy_data.tile4.nc .........OK
 Comparing RESTART/phy_data.tile5.nc .........OK
 Comparing RESTART/phy_data.tile6.nc .........OK
 Comparing RESTART/sfc_data.tile1.nc ............ALT CHECK......NOT OK
 Comparing RESTART/sfc_data.tile2.nc ............ALT CHECK......NOT OK
 Comparing RESTART/sfc_data.tile3.nc ............ALT CHECK......NOT OK
 Comparing RESTART/sfc_data.tile4.nc ............ALT CHECK......NOT OK
 Comparing RESTART/sfc_data.tile5.nc ............ALT CHECK......NOT OK
 Comparing RESTART/sfc_data.tile6.nc ............ALT CHECK......NOT OK
 Comparing sfcf012.tile1.nc .........OK
 Comparing sfcf012.tile2.nc .........OK
 Comparing sfcf012.tile3.nc .........OK
 Comparing sfcf012.tile4.nc .........OK
 Comparing sfcf012.tile5.nc .........OK
 Comparing sfcf012.tile6.nc .........OK

  0: The total amount of wall time                        = 32.924707

Test rst fv3_gsd FAIL

That is ok. Development in the authoritative repositories broke the GSL restart tests a while ago, we fixed it in the NOAA-GSL fork, gsl/develop branch, but these changes have not been merged back yet.

junwang-noaa

After the fv3 points back to the official repo, the PR can be committed.

…github.com/climbfuji/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

climbfuji · 2021-11-04T02:39:10Z

Submodule pointer for fv3atm looks ok to me, please check and merge if you agree.

climbfuji and others added 15 commits October 18, 2021 09:57

Update .gitmodules and submodule pointer for fv3atm for code review a…

11a8389

…nd testing

Update submodule pointer for fv3atm

b65ac41

Update submodule pointer for fv3atm

6659a01

Update .gitmodules and submodule pointer for fv3atm for code review a…

c1dd0ff

…nd testing

Merge branch 'develop' of https://github.com/ufs-community/ufs-weathe…

353915a

…r-model into remove_optional_arguments_from_ccpp

Update submodule pointer for fv3atm

03a194e

Merge branch 'develop' of https://github.com/ufs-community/ufs-weathe…

c92a961

…r-model into ccpp_convective_transport_tracers_cleanup

Modify opnReqTest related scripts for fv3_gsd test

ab67468

Merge branch 'develop' of https://github.com/ufs-community/ufs-weathe…

0c61195

…r-model into ccpp_convective_transport_tracers_cleanup

Merge branch 'develop' of https://github.com/ufs-community/ufs-weathe…

13f25b1

…r-model into remove_optional_arguments_from_ccpp

Merge branch 'develop' of https://github.com/ufs-community/ufs-weathe…

4db7f07

…r-model into ccpp_convective_transport_tracers_cleanup

Merge branch 'develop' of https://github.com/ufs-community/ufs-weathe…

1b68b21

…r-model into remove_optional_arguments_from_ccpp

Update .gitmodules and submodule pointer for fv3atm

f175795

Merge branch 'ccpp_convective_transport_tracers_cleanup' of https://g…

c9c3d24

…ithub.com/climbfuji/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

Merge branch 'remove_optional_arguments_from_ccpp' of https://github.…

46c4bef

…com/climbfuji/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

climbfuji marked this pull request as ready for review November 2, 2021 17:56

climbfuji requested a review from junwang-noaa November 2, 2021 17:56

Merge branch 'feature/gsd_opnreqtest' of https://github.com/minsukji-…

2dfe071

…noaa/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

climbfuji requested review from MinsukJi-NOAA, DeniseWorthen and DusanJovic-NOAA November 2, 2021 18:54

junwang-noaa added the No Baseline Change No Baseline Change label Nov 3, 2021

BrianCurtis-NOAA and others added 5 commits November 3, 2021 11:42

RT JOBS PASSED: orion.intel. Log file uploaded.

710cb37

on-behalf-of @ufs-community

RT JOBS PASSED: cheyenne.intel. Log file uploaded.

7a65b95

on-behalf-of @ufs-community

RT JOBS PASSED: hera.intel. Log file uploaded.

a9c3ffd

on-behalf-of @ufs-community

Regression test log for wcoss_cray

2b47d33

Merge branch 'cleanup-ccpp-and-remove-optional-arguments' of https://…

3ce4302

…github.com/climbfuji/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

BrianCurtis-NOAA and others added 2 commits November 3, 2021 18:13

RT JOBS PASSED: jet.intel. Log file uploaded.

64e421a

on-behalf-of @ufs-community

Regresssion test log for wcoss_dell_p3

991ecc0

climbfuji added 3 commits November 3, 2021 14:08

Set wallclock time on gaea back to 3h

f666c8e

Update to opnReqTest scrips from @MinsukJi-NOAA

0052eea

Merge branch 'cleanup-ccpp-and-remove-optional-arguments' of https://…

8108d02

…github.com/climbfuji/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

climbfuji added the run-ci label Nov 3, 2021

github-actions bot removed the run-ci label Nov 3, 2021

Regression test log for gaea.intel

85a26af

climbfuji added the Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. label Nov 3, 2021

DeniseWorthen approved these changes Nov 3, 2021

View reviewed changes

MinsukJi-NOAA approved these changes Nov 3, 2021

View reviewed changes

DusanJovic-NOAA approved these changes Nov 4, 2021

View reviewed changes

junwang-noaa approved these changes Nov 4, 2021

View reviewed changes

climbfuji added 2 commits November 3, 2021 20:37

Revert change to .gitmodules and update submodule pointer for fv3atm

24d0a08

Merge branch 'cleanup-ccpp-and-remove-optional-arguments' of https://…

24cbe65

…github.com/climbfuji/ufs-weather-model into cleanup-ccpp-and-remove-optional-arguments

junwang-noaa merged commit f76123c into ufs-community:develop Nov 4, 2021

MinsukJi-NOAA mentioned this pull request Nov 4, 2021

Modify opnReqTest related scripts for fv3_gsd test #890

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

climbfuji commented Nov 2, 2021 •

edited

Loading

junwang-noaa commented Nov 3, 2021

BrianCurtis-NOAA commented Nov 3, 2021

climbfuji commented Nov 3, 2021

climbfuji commented Nov 3, 2021

MinsukJi-NOAA commented Nov 3, 2021 •

edited

Loading

climbfuji commented Nov 3, 2021

MinsukJi-NOAA commented Nov 3, 2021 •

edited

Loading

climbfuji commented Nov 3, 2021

junwang-noaa left a comment

climbfuji commented Nov 4, 2021

Conversation

climbfuji commented Nov 2, 2021 • edited Loading

PR Checklist

Description

No changes to the regression test results or the input data.

Issue(s) addressed

Testing

Dependencies

junwang-noaa commented Nov 3, 2021

BrianCurtis-NOAA commented Nov 3, 2021

climbfuji commented Nov 3, 2021

climbfuji commented Nov 3, 2021

MinsukJi-NOAA commented Nov 3, 2021 • edited Loading

climbfuji commented Nov 3, 2021

MinsukJi-NOAA commented Nov 3, 2021 • edited Loading

climbfuji commented Nov 3, 2021

junwang-noaa left a comment

Choose a reason for hiding this comment

climbfuji commented Nov 4, 2021

climbfuji commented Nov 2, 2021 •

edited

Loading

MinsukJi-NOAA commented Nov 3, 2021 •

edited

Loading

MinsukJi-NOAA commented Nov 3, 2021 •

edited

Loading