update and maintain coupled regression tests for prototype configurations

[DeniseWorthen]

The purpose of this Discussion is to centralize a forum for namelist settings and ICs in order to maintain and update the coupled regression tests.

The current low resolution coupled RTs are using GFS 2017 physics (v14). These tests need to be updated to match the configurations being tested in the coupled Prototypes.

As new features are added on the physics side and planned for inclusion in upcoming prototypes, new regression tests for those features in the low resolution coupled RT tests are required. These tests need to be added at the time these features are committed to their various component repositories.

The benchmark RT (S2SW) will be maintained for any current coupled Prototype. As new Prototype configurations are finalized, the current benchmark RT will be deprecated and replaced with the new Prototype configuration.

[DeniseWorthen]

@jiandewang @JessicaMeixner-NOAA For the non-benchmark tests, the MOM6 ICs are copied in fv3_conf/cpld_control_run.IN. The ICs are non-resolution dependent; the file it is copying in is MOM6_IC_TS.nc.

  cp  @[INPUTDATA_ROOT]/MOM6_IC/MOM*.nc ./INPUT

[JessicaMeixner-NOAA]

I remember the MOM6_IC_TS.nc as being generated from CFS/CFSR for the specific date.

[jiandewang]

@JessicaMeixner-NOAA correct. I don't think any climatological T/S files were being generated from EMC to be used in MOM6.

[DeniseWorthen]

I do not believe the file is created from any CFS/CFSR restart for any specific date. It is a 1/2x1/2 regular lat/lon file with 40 vertical levels defined on z-levels.

[jiandewang]

@DeniseWorthen 40L 0.5x0.5 is exactly the format that CFSR data being processed to feed for MOM6.

[DeniseWorthen]

OK, I stand corrected. But if it is for a specific date, it should have a date in the name.

Can that file be created for this setting:

export SYEAR=2021
export SMONTH=03
export SDAY=22
export SHOUR=06

[jiandewang]

OK, I stand corrected. But if it is for a specific date, it should have a date in the name.

Can that file be created for this setting:

export SYEAR=2021
export SMONTH=03
export SDAY=22
export SHOUR=06

OK will do that.

[DeniseWorthen]

1. For the coupled model, the current c384 Prototype 7b configuration uses the settings below. What settings should be used for a P7 configuration for C96?

  d2_bg_k1 = 0.20
  d2_bg_k2 = 0.04
  dz_min = 2
  psm_bc = 1
  dddmp = 0.2

2. The current control_ugwpv1 RT includes a setting for knob_ugwp_nslope = 0. The current C384 P7 uses knob_ugwp_nslope = 1. Which setting should the coupled C96 P7 configuration use?

[yangfanglin]

Denise, you can use the same settings for C96. Please set knob_ugwp_nslope=1

[DeniseWorthen]

Thanks very much.

[SMoorthi-emc]

That "cap" is auto-generated code.

…

On Fri, Aug 6, 2021 at 12:44 PM Denise Worthen ***@***.***> wrote: Thanks very much. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#737 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYTVJIE7WOXIXLYCFSDT3QGNLANCNFSM5BV5QOPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[SMoorthi-emc]

Unfortunately, my test on dell failed with "cat: write error: Disk quota exceeded"

…

On Fri, Aug 6, 2021 at 12:44 PM Denise Worthen ***@***.***> wrote: Thanks very much. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#737 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYTVJIE7WOXIXLYCFSDT3QGNLANCNFSM5BV5QOPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718

[DeniseWorthen]

The MOM6 setting MOM6_REPRO_LA is confusingly named. The original name was chosen because the it controlled a parameter in MOM6 which needed to be false in order to obtain restart reproducibility.

The fix for that bug was made in MOM6 PR #49. The actual MOM6 setting USE_LA_LI2016 controlled by this parameter has been set true for the benchmark RTs but retained as false for non-benchmark RTs.

I will be renaming this setting to something more logical and changing the default value to true for all updated v16 RTs.

[DeniseWorthen]

@jiandewang @binli2337 I have a vague memory that for the 1deg MOM6 for the NG-GODAS application, someone was able to obtain a 1deg Runoff file for MOM6. Do either of you know whether such a file exists for use?

[jiandewang]

@DeniseWorthen I had some email exchange with Kim long time ago on this, my opinion is that 1x1 is too coarse to include runoff. There is no this fixed file at this moment.

[binli2337]

@DeniseWorthen @jiandewang This file is not available.

[DeniseWorthen]

Are there any restart-related changes I need to make in input.nml when do_ca is set true?

I have the YYYYMMDD.HH0000.ca_data.tileX.nc copied to the right name in the INPUT directory. But I cannot obtain restart reproducibility for the low-res P7 tests with do_ca=true; multiple fields exported by the ATM on the first timestep after restart do not reproduce:

56: RMS atmImp_Faxa_lwnet                2.4965E-04            NORMALIZED  3.9303E-06
65: RMS atmImp_Faxa_rain                 5.2501E-06            NORMALIZED  1.7072E-01
137: RMS atmImp_Sa_pbot                   3.7080E-02            NORMALIZED  3.7649E-07
153: RMS atmImp_Sa_shum                   9.6443E-08            NORMALIZED  1.0789E-05
162: RMS atmImp_Sa_tbot                   1.1705E-03            NORMALIZED  4.0705E-06
185: RMS atmImp_Sa_z                      3.7501E-05            NORMALIZED  3.5789E-06

I can almost obtain restart reproducibility (there are still 6 points on tile 3 for lwnet exported by the ATM which do not reproduce) if I set do_ca=false, do_sgs=false. These points are all on land.

The settings I am using are

  do_ca        = .true.
  ca_sgs       = .true.
  nca          = 1
  scells       = 2600
  tlives       = 1800
  nseed        = 1
  nfracseed    = 0.5
  rcell        = 0.72
  ca_trigger   = .true.
  nspinup      = 1
  iseed_ca     = 12345

[DeniseWorthen]

After help from @SMoorthi-emc I now understand that with do_ca set false, the model does reproduce if the restart files are checked. The field lwnet is exported to ocean, but since the only differences at restart are on land, the model reproduces (this field must be diagnostic only for FV3). This means that using do_ca=true does cause the failure in restart reproducibility.

@lisa-bengtsson Can you tell me if any settings need to be changed in input.nml for restarting when do_ca = true?

[DeniseWorthen]

OK, then there is no issue w/ the current fv3_conf/cpld_control.run.IN. It has a step where it removes from the INPUT any IC that was already present. If no CA has no ICs, then there is nothing to remove.

[lisa-bengtsson]

Ok, thanks, I'll keep looking

[lisa-bengtsson]

@pjpegion I see that there is an update in the stochastic_physics_wrapper.F90 in where there is a check "is_initialized" is added, in this case is the CA called (evolved) at kdt=0? In the version I submitted I made sure not to call the CA if kdt == 0, I just want to confirm with you that this is still the case? Otherwise there could be problems when restarting the CA. Thanks.

[pjpegion]

@lisa-bengtsson It looks like the "is_initialized" came in with Moorthi's PR
NOAA-EMC/fv3atm#303
Does anyone know what this was put there?

[lisa-bengtsson]

@SMoorthi-emc did you make the changes in stochastic_physics_wrapper.F90 in your July 13th PR?

[SMoorthi-emc]

Yes

…

Sent from my iPhone

On Aug 16, 2021, at 4:13 PM, lisa-bengtsson ***@***.***> wrote: @SMoorthi-emc did you make the changes in stochastic_physics_wrapper.F90 in your July 13th PR? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[pjpegion]

Can you tell us why? We need to make changes to the wrapper to allow for CA restart reproducibility, and would like to understand the need for adding this logic?

[SMoorthi-emc]

Because kdt is not zero for restart runs.

…

Sent from my iPhone

On Aug 16, 2021, at 4:23 PM, Phil Pegion ***@***.***> wrote: Can you tell us why? We need to make changes to the wrapper to allow for CA restart reproducibility, and would like to understand the need for adding this logic? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[SMoorthi-emc]

Kdt=fhour/dt + 1

…

Sent from my iPhone

On Aug 16, 2021, at 4:25 PM, Shrinivas Moorthi - NOAA Federal ***@***.***> wrote: Because kdt is not zero for restart runs. Sent from my iPhone > On Aug 16, 2021, at 4:23 PM, Phil Pegion ***@***.***> wrote: > > Can you tell us why? We need to make changes to the wrapper to allow for CA restart reproducibility, and would like to understand the need for adding this logic? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or unsubscribe. > Triage notifications on the go with GitHub Mobile for iOS or Android.

[lisa-bengtsson]

Hi Moorthi, in atmos_model.F90 there is an initialization step, here kdt = 0, regardless of restart or not. "stochastic_physics_wrapper.F90" and other routines are called here purely as initialization. I want to start to evolve the CA from the time_vary step, which is the second call to stochastic_physics_wrapper.F90 in atmos_model.F90. Thus I forced the CA to be 0 at kdt = 0.

[SMoorthi-emc]

I have to look at the code later. I don’t think kdt is zero; if it is it should not be. Moorthi

…

Sent from my iPhone

On Aug 16, 2021, at 4:46 PM, lisa-bengtsson ***@***.***> wrote: Hi Moorthi, in atmos_model.F90 there is an initialization step, here kdt = 0, regardless of restart or not. "stochastic_physics_wrapper.F90" and other routines are called here purely as initialization. I want to start to evolve the CA from the time_vary step, which is the second call to stochastic_physics_wrapper.F90 in atmos_model.F90. Thus I forced the CA to be 0 at kdt = 0. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[climbfuji]

I don't think that kdt is zero for restarts/warmstarts. But note that there are several flags controlling whether it is a warmstart or coldstart, whether it is the init step, i.e. the first call to the time integration after a coldstart or restart, ... the combination of these flags should allow you to do what you need to.

…

On Aug 16, 2021, at 2:50 PM, SMoorthi-emc ***@***.***> wrote: I have to look at the code later. I don’t think kdt is zero; if it is it should not be. Moorthi Sent from my iPhone > On Aug 16, 2021, at 4:46 PM, lisa-bengtsson ***@***.***> wrote: > > Hi Moorthi, in atmos_model.F90 there is an initialization step, here kdt = 0, regardless of restart or not. "stochastic_physics_wrapper.F90" and other routines are called here purely as initialization. I want to start to evolve the CA from the time_vary step, which is the second call to stochastic_physics_wrapper.F90 in atmos_model.F90. Thus I forced the CA to be 0 at kdt = 0. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or unsubscribe. > Triage notifications on the go with GitHub Mobile for iOS or Android. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#737 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RIJGCPN7K6SH2CKJVLT5F2ZDANCNFSM5BV5QOPA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>.

[pjpegion]

it used to be zero, but no longer the case. I think Lisa and I have figured out what the new logic needs to be. She is going to do some tests, and add in a CA restart test.

[lisa-bengtsson]

Ok, we think we know what is going on. The kdt used to be 0 at the level of the call to stochastic_physics_wrapper, but in Moorthi's PR there is this change in GFS_typedfs.F90:

< Model%kdt = 0

Model%kdt = nint(Model%fhour*con_hr/Model%dtp)

This would have caused the stochy_restart test to fail, which probably lead to the new is_initialize logic to enter. The CA takes input field from the physics, so in case of restarts I don't want to call it during when "is_initialize = True". I will do a test in where I put the call to CA under the "else" part of the "is_initialize' logic. I will let you know how it goes. Thanks.

[lisa-bengtsson]

Ok, the updated code works! @DeniseWorthen can I put this change into someone else's PR as an emergency bugfix. It is only 2 lines of code that needs to be changed.

[DeniseWorthen]

and thanks for helping to get this resolved.

[lisa-bengtsson]

Ok, yes, it does not change any baselines.

[lisa-bengtsson]

Thank you Denise, good team work. I will make sure to generate a ca_restart test so it doesn't break again.

[DeniseWorthen]

The c384 1/4deg test did not pass restart reproducibility. I will need to set up on Gaea in order to debug this. I have not been testing the c384 previously so I could still have some scripting errors in the p7 test for this configuration.

[DeniseWorthen]

The c384 1/4 deg restart test fails regardless of whether do_ca is true of false. As seen w/ the C96 case, the C384 case has lwnet exported from the ATM which does not reproduce on restart. There are 50 pts in the C384 case, all on tile3. In the C384 case, 5 (or 7 depending on CA) are exported to the ocean. In the C96 case, all these non-repro values were on land so had no impact on the export to the ocean.

I believe the fix for CA is correct. But in the case of C384 the non-repro lwnet field now breaks the restart.

[junwang-noaa]

@lisa-bengtsson can you send us the two line fix for the coupled restart test?

[lisa-bengtsson]

This is the updated file on Hera:
/scratch2/BMC/rem/Lisa.Bengtsson/Denise_sandbox/FV3/stochastic_physics/stochastic_physics_wrapper.F90

Is that enough, or do you want the actual code updates? You can do a diff to the one at the top of develop.

[climbfuji]

Unfortunately the regression tests had started before you posted this comment, so it won't go in with my current PRs. But we have some time left this week to work on another PR or two, and we can include it there. And yes, getting the updates in form of a PR would be best.

[lisa-bengtsson]

Ok, too bad because it was the same routine as I posted this morning. I will see if I can find some time tonight to work on a new PR.

[climbfuji]

Or just a branch/single commit that we can pull into one of our next PRs. No need to create the whole hierarchy of pull requests.

…

On Aug 17, 2021, at 3:17 PM, lisa-bengtsson ***@***.***> wrote: Ok, too bad because it was the same routine as I posted this morning. I will see if I can find some time tonight to work on a new PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#737 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RLKLUBI252A3DBF3VDT5LGVVANCNFSM5BV5QOPA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>.

[lisa-bengtsson]

@climbfuji @junwang-noaa I made a branch with the single change here:

https://github.com/lisa-bengtsson/fv3atm/tree/stochy_wrapper

Let me know if this is OK.

Lisa

[climbfuji]

Hey Lisa, great, thanks! I will pull this into an fv3atm PR after the current PRs are merged.

[lisa-bengtsson]

Great, thank you!

[climbfuji]

Draft PRs are here: NOAA-EMC/fv3atm#371 and #751 - is there an issue that your changes fix that should be added to the PR descriptions?

[lisa-bengtsson]

I did not create and issue, there is only this discussion thread. Should I create one?

[climbfuji]

It's ok from my side without an issue, I will just refer to the discussion thread for background information. Thanks!

…

On Aug 18, 2021, at 3:12 PM, lisa-bengtsson ***@***.***> wrote: I did not create and issue, there is only this discussion thread. Should I create one? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#737 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RK3FOMTJ3NHDWHMHVDT5QO5PANCNFSM5BV5QOPA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>.

[lisa-bengtsson]

Ok, thanks!