Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update to use p8b settings in regression tests #1042

Closed
JessicaMeixner-NOAA opened this issue Feb 9, 2022 · 18 comments · Fixed by #1071
Closed

Update to use p8b settings in regression tests #1042

JessicaMeixner-NOAA opened this issue Feb 9, 2022 · 18 comments · Fixed by #1071
Assignees
@JessicaMeixner-NOAA
Labels
enhancement New feature or request

Comments

@JessicaMeixner-NOAA
Copy link
Collaborator

JessicaMeixner-NOAA commented Feb 9, 2022 •
edited
Loading

Description

Update p8 (coupled and standalone atm) regression tests to use p8b settings. Including a test for aerosols. The benchmark tests will point to the p8b ICs (which are the same as p8a/p7.2).

Solution

Update the regtests

Alternatives

None

Related to

NOAA-EMC/fv3atm#478
@JessicaMeixner-NOAA JessicaMeixner-NOAA added the enhancement New feature or request label Feb 9, 2022
@JessicaMeixner-NOAA JessicaMeixner-NOAA self-assigned this Feb 9, 2022
@JessicaMeixner-NOAA JessicaMeixner-NOAA mentioned this issue Feb 9, 2022
Closed
@junwang-noaa
Copy link
Collaborator

@JessicaMeixner-NOAA would you please list your ufs-weather-model branch? Also my understanding is the the current benchmark test is running with P8a configuration and benchmark ICs. If there are any input file updates for this p8b RT update, would you please list the file location and we can add the data to RT location. Thanks

@JessicaMeixner-NOAA
Copy link
Collaborator Author

The ufs-weahter-model branch can be found here: https://github.com/JessicaMeixner-NOAA/ufs-weather-model/tree/feature/p8b It was updated with the most recent ufs-weather-model branch as of yesterday early afternoon and tested with a created baseline. The following is the status as of this morning.

Note, the input is set-up for hera only right now (although if you commented out the line for the newest ICs, everything should work on any machine). The ICs are only on hera in my personal directory until things are more stable, they can be found here:
/scratch2/NCEPDEV/climate/Jessica.Meixner/p8b/BM_IC-20220207
I made sure that the p7 tests replicated their old baselines before pointing to the new ICs.

Current issues/to do:

  • All settings still need to be confirmed by the physics group (although @yangfanglin has looked at a lot of this)
  • Add aerosol now that atm-ocn-ice-wav coupled is running (FYI @rmontuoro)
  • The control_p8 (standalone atm, C96 with p8b settings) is failing with:
96: FATAL from PE    96: NaN in input field of mpp_reproducing_sum(_2d), this indicates numerical instability
 96:
128: Image              PC                Routine            Line        Source
128: fv3.exe            0000000003E701FA  Unknown               Unknown  Unknown
128: fv3.exe            00000000029BF3FC  mpp_mod_mp_mpp_er          71  mpp_util_mpi.inc
128: fv3.exe            0000000002C6173F  mpp_efp_mod_mp_mp         198  mpp_efp.F90
128: fv3.exe            0000000002BBF319  mpp_domains_mod_m         137  mpp_global_sum.h
128: fv3.exe            0000000001D88EE0  fv_grid_utils_mod        3064  fv_grid_utils.F90
128: fv3.exe            0000000001DC1B0B  fv_mapz_mod_mp_la         793  fv_mapz.F90
128: libiomp5.so        00002ABDCF102A43  __kmp_invoke_micr     Unknown  Unknown
128: libiomp5.so        00002ABDCF0C62C6  __kmp_fork_call       Unknown  Unknown
128: libiomp5.so        00002ABDCF085BB0  __kmpc_fork_call      Unknown  Unknown
128: fv3.exe            0000000001DBD1ED  fv_mapz_mod_mp_la         682  fv_mapz.F90
128: fv3.exe            0000000001D67564  fv_dynamics_mod_m         756  fv_dynamics.F90
128: fv3.exe            0000000001CE34AB  atmosphere_mod_mp         669  atmosphere.F90
128: fv3.exe            0000000001C1676E  atmos_model_mod_m         763  atmos_model.F90
128: fv3.exe            0000000001AE1F9A  module_fcst_grid_         799  module_fcst_grid_comp.F90

I've tried a few settings of dt_inner, but all still result in a numerical instability. The control_debug_p8 test does pass without issue, so I'm not sure if my next thought of choosing a smaller time step is the best next step. Any ideas @bingfu @yangfanglin @AnningCheng-NOAA @RuiyuSun others?

  • The cpld_debug_p8 is failing, but it was the first time this test was ran and it's a set-up issue I should be able to fix.

@AnningCheng-NOAA
Copy link
Contributor

AnningCheng-NOAA commented Feb 10, 2022 via email

@RuiyuSun
Copy link
Contributor

RuiyuSun commented Feb 10, 2022 via email

@JessicaMeixner-NOAA
Copy link
Collaborator Author

@AnningCheng-NOAA @RuiyuSun A run directory can be found here: /scratch1/NCEPDEV/stmp2/Jessica.Meixner/FV3_RT/rt_18422/control_p8 (note this might get deleted as I make space for new runs, but you can always run the branch to get a new run directory).

The field table for control_p8 (and the other coupled runs) is:
FIELD_TABLE=field_table_thompson_noaero_tke
Which corresponds to the file here.

@AnningCheng-NOAA I'm assuming by setting to zero that value in the field table, would be to change these lines to:

# prognostic ice water mixing ratio
 "TRACER", "atmos_mod", "ice_wat"
           "longname",     "cloud ice mixing ratio"
           "units",        "kg/kg"
       "profile_type", "fixed", "surface_value=0" /
# prognostic rain water mixing ratio
 "TRACER", "atmos_mod", "rainwat"
           "longname",     "rain water mixing ratio"
           "units",        "kg/kg"
       "profile_type", "fixed", "surface_value=0" /

or something else? Also I'm assuming this would require creating a different field table? Would we want to use the same field table in the coupled and uncoupled cases?

@RuiyuSun
Copy link
Contributor

@JessicaMeixner-NOAA What is the smaller dt_inner that you tried? Could you use dt_inner = 60s or 120s?

@AnningCheng-NOAA
Copy link
Contributor

AnningCheng-NOAA commented Feb 10, 2022 via email

@JessicaMeixner-NOAA
Copy link
Collaborator Author

@RuiyuSun I had only tried 120 before best I can remember. Maybe 75?
@AnningCheng-NOAA thanks for confirming the field table is not an issue.

Let me know if there's another test I can try from my side. It's odd because the same ICs and everything work when coupled and also works in debug for the standalone test.

The coupled debug test also now runs but required extending the wall clock time. I'm checking to see if it can reproduce it's own baseline now.

@yangfanglin
Copy link
Collaborator

@JessicaMeixner-NOAA
Please set cdmbgwd="0.14,1.8,1.0,1.0" for C96

@JessicaMeixner-NOAA
Copy link
Collaborator Author

The standalone runs are now running with the updated cdmbgwd for C96 setting!

Also, the cpld debug reproduces its own baseline.

@arunchawla-NOAA
Copy link

@yangfanglin is there some documentation on how these numbers are determined for cdmbgwd ?

@yangfanglin
Copy link
Collaborator

@arunchawla-NOAA They are defined in config.fv3 of the workflow.

@yangfanglin
Copy link
Collaborator

For p8b, please see the numbers for different resolution given in /gpfs/dell6/emc/modeling/noscrub/Fanglin.Yang/para_gfs/gfsv17p8b/config.fv3

@JessicaMeixner-NOAA
Copy link
Collaborator Author

The remaining issues have been resolved and a PR will be raised after moving the <cpl?_aero tests to the tests. In the meantime, the input updates that are required are on hera here:
/scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/BM_IC-20220207
/scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/input-data-20211210/GOCART/p8
it would be great to get help to transfer that data to the other platforms if possible.

@junwang-noaa
Copy link
Collaborator

@jkbk2004 Would you please sync the new data sets to Cheyenne? I am sync-ing them to other platforms. Thanks

@junwang-noaa
Copy link
Collaborator

@JessicaMeixner-NOAA The input data you listed are now available on Orion, jet, gaea and dell. It will be populated to surge when it's available.

@jkbk2004
Copy link
Collaborator

@junwang-noaa @JessicaMeixner-NOAA let me try to synch from her to cheyenne. Hopefully, this time it will go thru ok with epic account. I will keep you posted.

@JessicaMeixner-NOAA JessicaMeixner-NOAA mentioned this issue Feb 28, 2022
Merged
16 tasks
@jkbk2004
Copy link
Collaborator

jkbk2004 commented Mar 1, 2022

@junwang-noaa @JessicaMeixner-NOAA completed synch-ing to cheyenne.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JessicaMeixner-NOAA JessicaMeixner-NOAA

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

p8b (with aerosols) JessicaMeixner-NOAA/ufs-weather-model
7 participants
@JessicaMeixner-NOAA @junwang-noaa @jkbk2004 @RuiyuSun @yangfanglin @AnningCheng-NOAA @arunchawla-NOAA