"to include GF updates in GSDv0beta4"

[haiqinli]

To include Grell-Freitas convection update in regression test of GSDv0beta4/GSDv0beta4mRUC, which has improved forecast skill.
1). Use FCT to represent cloud subsidence effect of cloud water/ice;
2). modified convective cloud detrainment;
3). to include number concentrations;
2). to include rain evaporation below cloud base for deep and congestus convection.

[climbfuji]

I cannot say much about the physics changes in the GF scheme itself, but regarding GFS_suite_interstitial_4: please do not pass in the Model/GFS_control DDT. You only need Model%imfdeepcnv, and you can pass this in instead using the following metadata:

[imfdeepcnv]
  standard_name = flag_for_mass_flux_deep_convection_scheme
  long_name = flag for mass-flux deep convection scheme
  units = flag
  dimensions = ()
  type = integer
  intent = in
  optional = F

[haiqinli]

Dom, thank you very much for your suggestions. I have updated the input variable of imfdeepcnv for GFS_suite_interstitial.F90 (meta) accordingly.

[climbfuji]

I think I understand by now what the code changes are about, running a few tests on hera, cheyenne and my macbook before merging. Thanks for making the changes that I requested.

[climbfuji]

Reverting my previous approval. After adding the changes, all regression tests in rt_ccpp_gsd.conf that include GF segfault (all tests w/o GF still run fine).

[haiqinli]

1). correct the number of tracers (ntrac) for gq0
2). initialize dtime_max for FCT
2). stricter condition of rain_evap_blew_cloudbase

Passed the regression test of rt_ccpp_gsd.conf with the above updates.

[climbfuji]

Associated PR: https://github.com/NCAR/NEMSfv3gfs/pull/272, see this PR for testing information.

[haiqinli]

Update cu_gf_driver.F90 to pass the regression test case of ccpp_gsd_noah_repro. Now all the cases in rt_ccpp_gsd.conf passed the regression test of static build and dynamic build.

[climbfuji]

Thanks, Haiqin! I will give this a try as well. I do not understand how the change in your recent commit can cause such an error, but let's see how the testing goes!

[climbfuji]

I pulled this PR into the latest version of gsd/develop and ran the rt_ccpp_gsd.conf tests in create and verify mode. All tests now pass. The change made in 3a28055 doesn't give a clear explanation as to why the fv3_ccpp_gsd_noah tests failed before making this change, however this is beyond the scope of this PR. I am going to merge this PR and update the submodule pointer in NEMSfv3gfs/gsd/develop afterwards.

However, I would like to note that during my investigation of the above issues, I ran all tests in rt_ccpp_gsd.conf in DEBUG (instead of REPRO) mode, and the following three tests crash:

fv3_ccpp_gf_thompson
fv3_ccpp_gsd
fv3_ccpp_gsd_noah

The crash occurs for the same reason in all three tests, an out of bounds read of a lookup table in Thompson MP:

  3: forrtl: severe (408): fort: (2): Subscript #2 of the array T_EFRW has value 101 which is greater than the upper bound of 100
  3:
  3: Image              PC                Routine            Line        Source
  3: fv3.exe            000000000647D3C0  Unknown               Unknown  Unknown
  3: fv3.exe            00000000041D7ECE  module_mp_thompso        2060  module_mp_thompson.F90
  3: fv3.exe            00000000041847B6  module_mp_thompso        1229  module_mp_thompson.F90
  3: fv3.exe            0000000003F15F04  mp_thompson_mp_mp         347  mp_thompson.F90
  3: fv3.exe            000000000356512F  ccpp_fv3_gsd_noah        1491  ccpp_FV3_GSD_noah_physics_cap.F90
  3: fv3.exe            0000000002A104B2  ccpp_static_api_m         349  ccpp_static_api.F90
  3: fv3.exe            00000000005E4DEA  ccpp_driver_mp_cc         234  CCPP_driver.F90
  3: libiomp5.so        00002AF90E592A43  __kmp_invoke_micr     Unknown  Unknown
  3: libiomp5.so        00002AF90E5562C6  __kmp_fork_call       Unknown  Unknown
  3: libiomp5.so        00002AF90E515BB0  __kmpc_fork_call      Unknown  Unknown
  3: fv3.exe            00000000005E25FE  ccpp_driver_mp_cc         214  CCPP_driver.F90
  3: fv3.exe            00000000004FC207  atmos_model_mod_m         366  atmos_model.F90
  3: fv3.exe            00000000004EF0FA  module_fcst_grid_         705  module_fcst_grid_comp.F90
  3: libesmf.so         00002AF9098F5E41  _ZN5ESMCI6FTable1        2010  ESMCI_FTable.C
  3: libesmf.so         00002AF9098F9A26  ESMCI_FTableCallE         746  ESMCI_FTable.C
  3: libesmf.so         00002AF909DE579A  _ZN5ESMCI2VM5ente        1178  ESMCI_VM.C
  3: libesmf.so         00002AF9098F7477  c_esmc_ftablecall         898  ESMCI_FTable.C
  3: libesmf.so         00002AF90A028521  esmf_compmod_mp_e        1209  ESMF_Comp.F90
  3: libesmf.so         00002AF90A2E57C4  esmf_gridcompmod_        1889  ESMF_GridComp.F90
  3: fv3.exe            00000000004CC4BF  fv3gfs_cap_mod_mp         988  fv3_cap.F90
  3: libesmf.so         00002AF909CC3080  _ZN5ESMCI13Method         287  ESMCI_MethodTable.C
...

The line in question is:

          Ef_rw = t_Efrw(idx, INT(mvd_c(k)*1.E6))

The variable mvd_c represents the diameter of cloud particles. I do not have sufficient understanding of whether a value of 101 (or 104 as in one of the other crashes) is completely unreasonable (given that the table goes to index 100), but in any case there should be a guard in the code that prevents out-of-bounds reads. I will continue to investigate this issue after the merge of the current PR.

@haiqinli @joeolson42 @tanyasmirnova @hannahcbarnes

[climbfuji]

Regression test logs on hera/intel attached.

rt_ccpp_gsd_verify.log
rt_ccpp_gsd_create.log