Supplementary to PR # 1766: Updates for the Community Convective Cloud (C3) scheme

[haiqinli]

Description

Name the new unified convective parameterization as "Community Convective Cloud (c3) parameterization";
Update the aerosol-aware for GF;
Merge the aerosol-aware and RRFS configuration from GF into C3;
Make the closures as input namelist options for C3 and GF.

Input data additions/changes

• No changes are expected to input data.
• Changes are expected to input data:
  • New input data.
  • Updated input data.

Anticipated changes to regression tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

The following RT's are failed as expected because of the GF updates.

• rap_2threads_dyn32_phy32
• rap_control_dyn32_phy32.err
• rap_flake_debug
• rap_sfcdiff_decomp
• rap_2threads
• rap_control_dyn64_phy32_debug
• rap_lndp_debug
• rap_sfcdiff
• rap_cires_ugwp_debug
• rap_control_dyn64_phy32
• rap_noah_debug
• rap_unified_drag_suite_debug
• rap_clm_lake_debug
• rap_control
• rap_noah_sfcdiff_cires_ugwp_debug
• rap_unified_ugwp_debug
• rap_control_debug_dyn32_phy32
• rap_decomp
• rap_progcld_thompson_debug
• rap_control_debug
• rap_diag_debug
• rap_sfcdiff_debug

Subcomponents involved:

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Library Updates/Changes

• Not Needed
• Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
• Add issue link from JCSDA/spack-stack following this item

Combined with PR's (If Applicable):

Commit Queue Checklist:

• Link PR's from all sub-components involved in section below
• Confirm reviews completed in ALL sub-component PR's
• Add all appropriate labels to this PR.
• Run full RT suite on either Hera/Cheyenne AND attach log to a PR comment.
• Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

• depends on Community Convective Cloud (C3) scheme NOAA-EMC/fv3atm#657

Testing Day Checklist:

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
• Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

• RDHPCS
  • Hera
  • Orion
  • Jet
  • Gaea
  • Cheyenne
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[BrianCurtis-NOAA]

@haiqinli we need you to link the PR from FV3 - depends on NOAA-EMC/fv3atm/pull/<number> will work.
Also we need you to run the full RT suite on Hera. This is the evidence to show us which tests fail, and confirmation of what is expected to fail. I don't see either of these in the previous PR.

[haiqinli]

@BrianCurtis-NOAA
Sure,

• depends on Community Convective Cloud (C3) scheme NOAA-EMC/fv3atm#657
  The regression test on Hera at /scratch1/BMC/gsd-fv3-dev/NCEPDEV/stmp4/Haiqin.Li/Haiqin.Li/FV3_RT/rt_286522. The following RT's are failed as expected because of the GF updates.
  rap_2threads_dyn32_phy32 rap_control_dyn32_phy32.err rap_flake_debug rap_sfcdiff_decomp
  rap_2threads rap_control_dyn64_phy32_debug rap_lndp_debug rap_sfcdiff
  rap_cires_ugwp_debug rap_control_dyn64_phy32 rap_noah_debug rap_unified_drag_suite_debug
  rap_clm_lake_debug rap_control rap_noah_sfcdiff_cires_ugwp_debug rap_unified_ugwp_debug
  rap_control_debug_dyn32_phy32 rap_decomp rap_progcld_thompson_debug
  rap_control_debug rap_diag_debug rap_sfcdiff_debug

[BrianCurtis-NOAA]

@BrianCurtis-NOAA Sure,

• depends on Community Convective Cloud (C3) scheme NOAA-EMC/fv3atm#657
  The regression test on Hera at /scratch1/BMC/gsd-fv3-dev/NCEPDEV/stmp4/Haiqin.Li/Haiqin.Li/FV3_RT/rt_286522. The following RT's are failed as expected because of the GF updates.
  rap_2threads_dyn32_phy32 rap_control_dyn32_phy32.err rap_flake_debug rap_sfcdiff_decomp
  rap_2threads rap_control_dyn64_phy32_debug rap_lndp_debug rap_sfcdiff
  rap_cires_ugwp_debug rap_control_dyn64_phy32 rap_noah_debug rap_unified_drag_suite_debug
  rap_clm_lake_debug rap_control rap_noah_sfcdiff_cires_ugwp_debug rap_unified_ugwp_debug
  rap_control_debug_dyn32_phy32 rap_decomp rap_progcld_thompson_debug
  rap_control_debug rap_diag_debug rap_sfcdiff_debug

Can you take the RegressionTests_hera.log and attach it to a comment here? If not I'd be happy to go grab it.

[haiqinli]

@BrianCurtis-NOAA Sure, yesterday I did the regression test for Hera/Intel and Hera/GNU in new baseline mode, and the log file is attached. Thanks.
RegressionTests_hera.log

[BrianCurtis-NOAA]

@BrianCurtis-NOAA Sure, yesterday I did the regression test for Hera/Intel and Hera/GNU in new baseline mode, and the log file is attached. Thanks. RegressionTests_hera.log

Ah, I see. We will need you to run it just with -e so we can see what fails. ./rt.sh -e -a <account> > rt.out 2>&1 & then attach that here.

[haiqinli]

@BrianCurtis-NOAA Sure, I will start to run regression test with -e, and attached the log file when it is done.

[haiqinli]

@BrianCurtis-NOAA Finally the regression test log file with "-e" was generated, and it is attached as below. Thanks.
RegressionTests_hera.log

[BrianCurtis-NOAA]

@BrianCurtis-NOAA Finally the regression test log file with "-e" was generated, and it is attached as below. Thanks. RegressionTests_hera.log

Thanks for your work towards this. It looks like the tests that failed are more than what you have above:

regional_control_intel 
regional_control_qr_intel 
regional_decomp_intel 
regional_2threads_intel 
regional_noquilt_intel 
regional_netcdf_parallel_intel 
regional_2dwrtdecomp_intel 
regional_wofs_intel 
rap_control_intel 
regional_spp_sppt_shum_skeb_intel 
rap_decomp_intel 
rap_2threads_intel 
rap_sfcdiff_intel 
rap_sfcdiff_decomp_intel 
rrfs_smoke_conus13km_hrrr_warm_intel 
rrfs_smoke_conus13km_hrrr_warm_2threads_intel 
rrfs_conus13km_hrrr_warm_intel 
rrfs_smoke_conus13km_radar_tten_warm_intel 
regional_control_faster_intel 
rap_control_debug_intel 
rap_unified_drag_suite_debug_intel 
rap_diag_debug_intel 
rap_cires_ugwp_debug_intel 
rap_unified_ugwp_debug_intel 
rap_lndp_debug_intel 
rap_progcld_thompson_debug_intel 
rap_noah_debug_intel 
rap_sfcdiff_debug_intel 
rap_noah_sfcdiff_cires_ugwp_debug_intel 
rap_clm_lake_debug_intel 
rap_flake_debug_intel 
regional_spp_sppt_shum_skeb_dyn32_phy32_intel 
rap_control_dyn32_phy32_intel
rap_2threads_dyn32_phy32_intel 
rap_control_dyn64_phy32_intel 
rap_control_debug_dyn32_phy32_intel 
rap_control_dyn64_phy32_debug_intel 
hafs_regional_atm_intel 
hafs_regional_atm_thompson_gfdlsf_intel 
hafs_regional_atm_ocn_intel 
hafs_regional_atm_wav_intel 
hafs_regional_atm_ocn_wav_intel 
hafs_regional_1nest_atm_intel 
hafs_regional_1nest_atm_qr_intel 
hafs_regional_telescopic_2nests_atm_intel 
hafs_global_1nest_atm_intel 
hafs_global_1nest_atm_qr_intel 
hafs_global_multiple_4nests_atm_intel 
hafs_global_multiple_4nests_atm_qr_intel 
hafs_regional_specified_moving_1nest_atm_intel 
hafs_regional_storm_following_1nest_atm_intel 
hafs_regional_storm_following_1nest_atm_qr_intel 
hafs_regional_storm_following_1nest_atm_ocn_intel 
hafs_global_storm_following_1nest_atm_intel 
hafs_regional_storm_following_1nest_atm_ocn_wav_intel 
hafs_regional_docn_intel
hafs_regional_docn_oisst_intel 
rap_control_gnu 
rap_decomp_gnu 
rap_2threads_gnu 
rap_sfcdiff_gnu 
rap_sfcdiff_decomp_gnu 
rap_control_debug_gnu 
rap_diag_debug_gnu 
rap_noah_sfcdiff_cires_ugwp_debug_gnu 
rap_progcld_thompson_debug_gnu 
rap_flake_debug_gnu 
rap_clm_lake_debug_gnu 
rap_control_dyn32_phy32_gnu 
rap_2threads_dyn32_phy32_gnu 
rap_control_dyn64_phy32_gnu 
rap_control_debug_dyn32_phy32_gnu 
rap_control_dyn64_phy32_debug_gnu 

Does it make sense as to what failed vs the changes made in this PR? If so, can you explain what?

[haiqinli]

@BrianCurtis-NOAA Thanks for checking it. The 1st set of cases were from the regression test about 3 weeks ago. The 2nd list of cases are from the merging with latest authoritative repository. Not sure why the merging introduces more cases.

[jkbk2004]

@haiqinli Do the code changes at GF routine affect across other regional cases? @Qingfu-Liu @SamuelTrahanNOAA @BinLiu-NOAA can you make some comments?

[jkbk2004]

@grantfirl any comment about the impact on other regional cases?

[SamuelTrahanNOAA]

The fv3atm pull request changes the atmos_cubed_sphere hash. Maybe that's where all the unexpected changes come from?

That may have been an accident while merging fv3atm develop.

[jkbk2004]

@SamuelTrahanNOAA great catch! @haiqinli can you update the atmos_cubed_sphere hash and redo the hera test? correct hash is https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/tree/49f15ecbbc16405025fae8d672dced19c2073d9e

[haiqinli]

@haiqinli Do the code changes at GF routine affect across other regional cases? @Qingfu-Liu @SamuelTrahanNOAA @BinLiu-NOAA can you make some comments?

@jkbk2004 No, the GF should not affect other regional cases if the GF convection was not called. The regression test I did about 3 weeks ago, only the cases with GF were affected. Thanks.

[haiqinli]

@SamuelTrahanNOAA great catch! @haiqinli can you update the atmos_cubed_sphere hash and redo the hera test? correct hash is https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/tree/49f15ecbbc16405025fae8d672dced19c2073d9e

@jkbk2004 Sure, I will update the atmos_cubed_sphere hash and redo the hera test.

[haiqinli]

@jkbk2004 The regression test is done, and log file is attached.
RegressionTests_hera.log

As expected, only the rap cases (including GF convection) failed.
FAILED TESTS:

rap_control_intel
rap_decomp_intel
rap_2threads_intel
rap_sfcdiff_intel
rap_sfcdiff_decomp_intel
rap_control_debug_intel
rap_unified_drag_suite_debug_intel
rap_diag_debug_intel
rap_cires_ugwp_debug_intel
rap_unified_ugwp_debug_intel
rap_lndp_debug_intel
rap_progcld_thompson_debug_intel
rap_noah_debug_intel
rap_sfcdiff_debug_intel
rap_noah_sfcdiff_cires_ugwp_debug_intel
rap_clm_lake_debug_intel
rap_flake_debug_intel 114
rap_control_dyn32_phy32_intel
rap_2threads_dyn32_phy32_intel
rap_control_dyn64_phy32_intel
rap_control_debug_dyn32_phy32_intel
rap_control_dyn64_phy32_debug_intel
rap_control_gnu
rap_decomp_gnu
rap_2threads_gnu
rap_sfcdiff_gnu
rap_sfcdiff_decomp_gnu
rap_control_debug_gnu
rap_diag_debug_gnu
rap_noah_sfcdiff_cires_ugwp_debug_gnu
rap_progcld_thompson_debug_gnu
rap_flake_debug_gnu
rap_clm_lake_debug_gnu
rap_control_dyn32_phy32_gnu
rap_2threads_dyn32_phy32_gnu
rap_control_dyn64_phy32_gnu
rap_control_debug_dyn32_phy32_
rap_control_dyn64_phy32_debug_gnu

[jkbk2004]

@haiqinli what conflicts on FV3?

[haiqinli]

@jkbk2004 I don't think there are conflicts with FV3. Thanks.

[jkbk2004]

@haiqinli #1769 was merged yesterday. Please, continue to sync up.

[haiqinli]

@jkbk2004 Sure, I will sync both fv3atm and ufs-weather-model, and update when it is done. Thanks.

[haiqinli]

@jkbk2004 The syncing of fv3atm and ufs-weather-model are done. Thanks.

[haiqinli]

@jkbk2004 What's the status of it? We are waiting it for the comping HFIP runs. Thanks.

[jkbk2004]

We are working on #1776? What is timeline for this pr to be merged?

[haiqinli]

@jkbk2004 We will start the HFIP retro test runs next week, and the formal HFIP realtime run will start on early July. My PR has been synced two times, and I also did the regression tests twice. I received email about the Commit Queue this Monday, and it indicated be scheduled on this Tuesday. Thus I promised to my division that my PR should be merged this week. I have not heard any news after my 2nd sync up. So I asked the status. Thanks.

[jkbk2004]

We just merged #1776. Can you sync up at ufs-weather model level again? Then we will give priority to let the tests run over weekend. If all goes smoothly, then we will able to merge Monday or first thing Tuesday.

[haiqinli]

@jkbk2004 This PR has been synced up at ufs-weather-model level. Thanks.

[jkbk2004]

@BrianCurtis-NOAA This PR is priority for the HFIP retro test scheduled next week.

[zach1221]

Please see jenkins-ci logs attached. ORTs all passed.
ufs-weather-model » ort-docker-pipeline » PR-1795 #1 Console [Jenkins].pdf

[jkbk2004]

Jet regression was running ok. But due to lsf4 file system issue, not able to push jet log. We can start merging process. @haiqinli I am moving to ccpp dependency to follow up there.

[haiqinli]

@jkbk2004 Thank you very much!

[jkbk2004]

@haiqinli fv3 pr was merged. Can you update fv3 submodule pointer and revert change in gitmodules? correct hash is NOAA-EMC/fv3atm@2363f5b

[SadeghTabas-NOAA]

Please update .gitmodules and pointer.

[haiqinli]

@jkbk2004 Sure, I will do it.

[haiqinli]

@jkbk2004 It is done. Thanks.