Restart runs with stochastic physics due not reproduce #318

climbfuji · 2020-12-04T14:58:21Z

Description

Runs with stochastic physics turned on do not give b4b identical results in restart runs. This is because the initialization logic does not support restart runs, and the necessary fields aren't written to the restart fields (sppt/shum/skeb/... weights, ...)

To Reproduce:

Use GFs v16beta regression test run directories to set up a 0->48h run, compare to a 0->24h + 24h->48h run. This does not reproduce. Turn off stochastic physics in all three runs to get b4b identical results (in PROD mode).

pjpegion · 2020-12-04T15:10:17Z

Are you using the stochastic physics restart logic? In the original run, you need to set FHSTOCH=24 in this example. it will then dump out a restart for the stochastic physics called stoch_out.F<FHSTOCH>. And then for the restart run, you will rename stoch_out.F<FHSTOCH> to stoch_ini and set STOCHINI=.true. in the stochastic physics namelist block.

climbfuji · 2020-12-04T15:18:28Z

Are you using the stochastic physics restart logic? In the original run, you need to set FHSTOCH=24 in this example. it will then dump out a restart for the stochastic physics called stoch_out.F. And then for the restart run, you will rename stoch_out.F to stoch_ini and set STOCHINI=.true. in the stochastic physics namelist block.

No. This needs to be incorporated into the ufs restart capability, we unfortunately can't move files around and recompile for restart runs.

pjpegion · 2020-12-04T15:35:04Z

I'm confused, if you cannot move around files, how do the existing restart tests work? Maybe we should have a call.

climbfuji · 2020-12-04T15:42:24Z

I'm confused, if you cannot move around files, how do the existing restart tests work? Maybe we should have a call.

To conduct restart runs with the ufs-weather-model, one needs to update the forecast length in model_configure, flip 3-6 options in input.nml, and copy the NetCDF files written to RESTART/ to INPUT/. But no Fortran source code changes are required. Is stoch_out.F not a Fortran file?

pjpegion · 2020-12-04T15:46:57Z

no, it is a data file, and it would look like stoch_out.F000024 so it would require moving stoch_out.F000024 to stoch_ini and setting STOCHINI=.true. in input.nml So that it is clear to me. Do you want me to add stochastic physics to the current restart test, or do I need to create a new restart run with stochastic physics?

…

-Phil

On Fri, Dec 4, 2020 at 8:42 AM Dom Heinzeller ***@***.***> wrote: I'm confused, if you cannot move around files, how do the existing restart tests work? Maybe we should have a call. To conduct restart runs with the ufs-weather-model, one needs to update the forecast length in model_configure, flip 3-6 options in input.nml, and copy the NetCDF files written to RESTART/ to INPUT/. But no Fortran source code changes are required. Is stoch_out.F not a Fortran file? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#318 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJIRVJCEPSEMMXNAC6UQFM3STD7PFANCNFSM4UNRQBVQ> .

-- Philip Pegion (he/him/his) NOAA/Physical Sciences Laboratory-CIRES (303) 497-7897 philip.pegion@noaa.gov

pjpegion · 2020-12-04T15:49:38Z

Now I see the source of the confusion. I did not escape the < and > signs around <FHSTOCH>. So it looked like a fortran file.
I have since edited my comment above.

SMoorthi-emc · 2020-12-04T15:52:52Z

Why not change the restart file name to something like "stoch_restart$fhour"?

…

On Fri, Dec 4, 2020 at 10:50 AM Phil Pegion ***@***.***> wrote: Now I see the source of the confusion. I did not escape the < and > signs around <FHOUR>. So it looked like a fortran file. I have since edited my comment above. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#318 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYTEXXTUW5KZYK74KNDSTEAKFANCNFSM4UNRQBVQ> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

junwang-noaa · 2020-12-04T15:54:12Z

Phil, I think it is good to have a stochy_restart test.

…

On Fri, Dec 4, 2020 at 10:50 AM Phil Pegion ***@***.***> wrote: Now I see the source of the confusion. I did not escape the < and > signs around <FHOUR>. So it looked like a fortran file. I have since edited my comment above. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#318 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TO4AXCREOIPFWPTJ3LSTEAKFANCNFSM4UNRQBVQ> .

climbfuji · 2020-12-04T15:55:19Z

Ahhh ... got it. I like @SMoorthi-emc's suggestion for the name. As long as these are binary files, and we only need to rename those and update input.nml, that's fine.

This is definitely not included in any of the tests. I wonder if we could modify the ufs-weather-model logic (and possibly some code in stochastic_physics) that FHSTOCH will always be set to the the forecast length. Otherwise this will be totally confusing and lead to a lot of repeated runs. Restart files are written at the end of the model run by default, and the same should be true for stochastic physics in my opinion.

pjpegion · 2020-12-04T16:04:12Z

I like Moorthi's suggestion too,, I will just have to add logic in the initialization subroutine to make sure it has the forecast hour and can just read in the same named file. I will also have to coordinate with the DA workflow people since they are already handling the moving to stoch_out.F<FHSTOCH> to stoch_ini in the global_workflow.

Actually, I believe the logic that was added by Bing Fu does cause the restarts to be written out at the atmosphere's restart interval. But FHSTOCH needs to still be non zero in the namelist to trigger the restarting.

climbfuji · 2020-12-04T16:05:38Z

I like Moorthi's suggestion too,, I will just have to add logic in the initialization subroutine to make sure it has the forecast hour and can just read in the same named file. I will also have to coordinate with the DA workflow people since they are already handling the moving to stoch_out.F to stoch_ini in the global_workflow.

Actually, I believe the logic that was added by Bing Fu does cause the restarts to be written out at the atmosphere's restart interval. But FHSTOCH needs to still be non zero in the namelist to trigger the restarting.

We can update/overwrite FHSTOCH with the forecast length automatically. But, @junwang-noaa, isn't there a restart interval setting in model_configure that triggers the UFS to write out restart files at regular intervals? We need to support that if it exists.

SMoorthi-emc · 2020-12-05T01:54:21Z

Dom, we need to have the ability to write intermediate restarts. For example, in the coupled model I specify the interval at which to write restart.
Some times hourly, some times 3 hourly, some time 15 days etc. We need this flexibility. It should also write restart at FHMAX, which should be the default.

junwang-noaa · 2020-12-05T03:09:51Z

Moorthi, on model side, it is very flexible, you can actually write out restart files at several specific times and for checkpoint restart you don't need to write out the restart at FHMAX.

…

On Fri, Dec 4, 2020 at 8:54 PM SMoorthi-emc ***@***.***> wrote: Dom, we need to have the ability to write intermediate restarts. For example, in the coupled model I specify the interval at which to write restart. Some times hourly, some times 3 hourly, some time 15 days etc. We need this flexibility. It should also write restart at FHMAX, which should be the default. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#318 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TILLVLHPF5TBYR2PXTSTGHFTANCNFSM4UNRQBVQ> .

climbfuji · 2020-12-05T17:22:20Z

I agree with both of you - intermittent restarts and at the end of the run. @pjpegion please let me know if you need help implementing this.

SMoorthi-emc · 2020-12-05T17:34:48Z

I know that. I was asking about stochastic physics. Dom seems to have suggested writing only at fhmax. Moorthi

…

Sent from my iPhone

On Dec 4, 2020, at 10:10 PM, Jun Wang ***@***.***> wrote: Moorthi, on model side, it is very flexible, you can actually write out restart files at several specific times and for checkpoint restart you don't need to write out the restart at FHMAX. On Fri, Dec 4, 2020 at 8:54 PM SMoorthi-emc ***@***.***> wrote: > Dom, we need to have the ability to write intermediate restarts. For > example, in the coupled model I specify the interval at which to write > restart. > Some times hourly, some times 3 hourly, some time 15 days etc. We need > this flexibility. It should also write restart at FHMAX, which should be > the default. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#318 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AI7D6TILLVLHPF5TBYR2PXTSTGHFTANCNFSM4UNRQBVQ> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

climbfuji · 2020-12-05T17:49:17Z

I know that. I was asking about stochastic physics. Dom seems to have suggested writing only at fhmax. Moorthi
…

Initially I did, but then I revised it saying that intermittent restarts are possible and need to be supported - so all good.

pjpegion · 2020-12-08T15:25:09Z

Hi All, I will see how to get the restart_interval attribute into the stochastic physics so we don't need to have an additional namelist parameter to specify the restart frequency. The intention is to have restarts at the restart interval and one at the end of the run. Should I use the same naming convention at the atmosphere restarts, which is <YYYYMMDD>.<hhmmss>.<restart_file> for intermediate restarts and just <restart_file> for end of run restarts? Also, Should I go through the trouble of switching to netCDF? -Phil

…

On Sat, Dec 5, 2020 at 10:49 AM Dom Heinzeller ***@***.***> wrote: I know that. I was asking about stochastic physics. Dom seems to have suggested writing only at fhmax. Moorthi … <#m_-782539667609326166_> Initially I did, but then I revised it saying that intermittent restarts are possible and need to be supported - so all good. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#318 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJIRVJE33EBTYOC4LE6WFELSTJXCTANCNFSM4UNRQBVQ> .

-- Philip Pegion (he/him/his) NOAA/Physical Sciences Laboratory-CIRES (303) 497-7897 philip.pegion@noaa.gov

climbfuji · 2020-12-08T15:27:52Z

Consistency is always good, but in the end the decision on the file names and file format is yours. I'd probably use consistent filenames but keep the format as-is in a first step, and in a second step switch to NetCDF. But, up to you!

junwang-noaa · 2020-12-08T15:29:22Z

I think it is a good idea to use the same naming convention at the atmosphere restarts, (<YYYYMMDD>.<hhmmss>.<restart_file> for intermediate restarts and <restart_file> for end of run restarts. I'd think in the future it's good that everything is in netcdf. On Tue, Dec 8, 2020 at 10:25 AM Phil Pegion <notifications@github.com> wrote:

…

Hi All, I will see how to get the restart_interval attribute into the stochastic physics so we don't need to have an additional namelist parameter to specify the restart frequency. The intention is to have restarts at the restart interval and one at the end of the run. Should I use the same naming convention at the atmosphere restarts, which is <YYYYMMDD>.<hhmmss>.<restart_file> for intermediate restarts and just <restart_file> for end of run restarts? Also, Should I go through the trouble of switching to netCDF? -Phil On Sat, Dec 5, 2020 at 10:49 AM Dom Heinzeller ***@***.***> wrote: > I know that. I was asking about stochastic physics. Dom seems to have > suggested writing only at fhmax. Moorthi > … <#m_-782539667609326166_> > > Initially I did, but then I revised it saying that intermittent restarts > are possible and need to be supported - so all good. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #318 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AJIRVJE33EBTYOC4LE6WFELSTJXCTANCNFSM4UNRQBVQ > > . > -- Philip Pegion (he/him/his) NOAA/Physical Sciences Laboratory-CIRES (303) 497-7897 ***@***.*** — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#318 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TID2GPNDMJM3FZ4GEDSTZAOLANCNFSM4UNRQBVQ> .

pjpegion · 2020-12-09T20:51:15Z

@climbfuji I want to try to call the stochastic_physics restart writing from module_fcst_grid_comp.F90, but I am having issues compiling. My issue is that the build system doesn't know about including the stochastic_physics directory as part of the include path at this point. How did you do it for FV3/stochastic_physics/stochastic_physics_wrapper.F90?

climbfuji · 2020-12-09T20:57:19Z

I added

target_include_directories(stochastic_physics_wrapper PRIVATE ${CMAKE_BINARY_DIR}/stochastic_physics)
target_include_directories(stochastic_physics_wrapper PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/mod>)
target_link_libraries(stochastic_physics_wrapper PUBLIC fms
                                                        stochastic_physics
                                                        gfsphysics
                                                        fv3dycore)

to stochastic_physics_wrapper/CMakeLists.txt. Not sure if you need the second line, try without. Also try to just add the stochastic_physics library to the exiting target_link_libraries if it is not there already.

climbfuji · 2020-12-10T20:58:56Z

Update: @pjpegion and I were able to figure this out.

I added
target_include_directories(stochastic_physics_wrapper PRIVATE ${CMAKE_BINARY_DIR}/stochastic_physics)
target_include_directories(stochastic_physics_wrapper PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/mod>)
target_link_libraries(stochastic_physics_wrapper PUBLIC fms
                                                        stochastic_physics
                                                        gfsphysics
                                                        fv3dycore)
to stochastic_physics_wrapper/CMakeLists.txt. Not sure if you need the second line, try without. Also try to just add the stochastic_physics library to the exiting target_link_libraries if it is not there already.

Update: @pjpegion and I were able to figure this out.

junwang-noaa · 2021-06-21T03:11:15Z

@climbfuji @pjpegion May I ask if the issue is fixed in Lisa PR #589?

pjpegion · 2021-06-21T18:40:33Z

This should be been fixed with my PR from March: #372

pjpegion · 2021-06-21T18:41:07Z

This should be been fixed with my PR from March: #372

climbfuji · 2021-06-21T21:19:20Z

Do we have a test for it? Would be good.

DeniseWorthen · 2021-06-30T14:59:26Z

@pjpegion It looks like your PR #668 will add this restart test so I will link this issue.

…ates from release branch (#318) * updated docs * added git submodule * fix formatting * added new submodule commits * fixed ref links * finished Intro * finish Components & Intro edits * edited Rocoto workflow section of Quickstart * added minor hpc submodule commits * Updates to Rocoto Workflow in Quick Start * add to HPC-stack intro * submodule updates * added submodule docs edits * hpc-stack updates & formatting fixes * hpc-stack intro edits * bibtex attempted fix * add hpc-stack module edits * update sphinxcontrib version * add .readthedocs.yaml file * update .readthedocs.yaml file * update .readthedocs.yaml file * update conf.py * updates .readthedocs.yaml with submodules * updates .readthedocs.yaml with submodules * submodule updates * submodule updates * minor Intro edits * minor Intro edits * minor Intro edits * submodule updates * fixed typos in QS * QS updates * QS updates * QS updates * updates to InputOutput and QS * fix I/O doc typos * pull updates to hpc-stack docs * pull updates to hpc-stack docs * fix table wrapping * updates to QS for cloud * fix QS export statements * fix QS export statements * QS edits on bind, config * add bullet points to notes * running without rocoto * add HPC-Stack submodule w/docs * split QS into container/non-container approaches * added filepath changes for running in container on Orion, et al. * edits to overview and container QS * moved CodeReposAndDirs.rst info to the Introduction & deleted file * continued edits to SRWAppOverview * combine overview w/non-container docs * finish merging non-container guide & SRWOverview, rename/remove files, update FAQ * minor edits for Intro & QS * updates to BuildRun doc through 3.8.1 * edits to Build/Run and Components * remove .gitignore * fix Ch 3 title, 4 supported platform levels note * fix typos, add term links * other minor fixes/suggestions implemented * updated Intro based on feedback; changed SRW to SRW App throughout * update comment to Intro citation * add user-defined vertical levels to future work * Add instructions for srw_common module load * fix typo * update Intro & BuildRunSRW based on Mark's feedback * minor intro updates * 1st round of jwolff's edits * 2nd round of jwolff updates * update QS intro * fix minor physics details * update citation and physics suite name * add compute node allocation info to QS * add authoritative hpc-stack docs to Intro * create MacOS install/build instructions * update HPC-Stack submodule/docs * remove extra macinstall document * revert hpc-stack submodule update * update hpc-stack submod * update w/release PR#282 changes * update w/doc changes from PR#312 * clone develop * Build/Run updates * wcoss * B/R check complete * update UFS_UTILS links to latest * update WM links to latest * update ccpp/upp links to latest * finish components update * update config workflow chapter * contrib guide updates * update FAQ, fix formatting * update Glossary * update Graphics * I/O updates * intro updates * update LAM grid chapter * LAM grid update * NCQS updates * update WE2E * rename quickstart files * rename quickstart files * add/update mac/linux modulefiles Co-authored-by: gspetro <gillian.s.petro@gmail.com> Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com>

climbfuji added the bug Something isn't working label Dec 4, 2020

climbfuji assigned pjpegion Dec 4, 2020

DeniseWorthen linked a pull request Jun 30, 2021 that will close this issue

MOM6 IAU and atmos stochy restart test #668

Merged

14 tasks

DusanJovic-NOAA closed this as completed in #668 Jul 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restart runs with stochastic physics due not reproduce #318

Restart runs with stochastic physics due not reproduce #318

climbfuji commented Dec 4, 2020

pjpegion commented Dec 4, 2020 •

edited

Loading

climbfuji commented Dec 4, 2020

pjpegion commented Dec 4, 2020

climbfuji commented Dec 4, 2020

pjpegion commented Dec 4, 2020 via email

pjpegion commented Dec 4, 2020 •

edited

Loading

SMoorthi-emc commented Dec 4, 2020 via email

junwang-noaa commented Dec 4, 2020 via email

climbfuji commented Dec 4, 2020

pjpegion commented Dec 4, 2020

climbfuji commented Dec 4, 2020

SMoorthi-emc commented Dec 5, 2020

junwang-noaa commented Dec 5, 2020 via email

climbfuji commented Dec 5, 2020

SMoorthi-emc commented Dec 5, 2020 via email

climbfuji commented Dec 5, 2020

pjpegion commented Dec 8, 2020 via email

climbfuji commented Dec 8, 2020

junwang-noaa commented Dec 8, 2020 via email

pjpegion commented Dec 9, 2020

climbfuji commented Dec 9, 2020

climbfuji commented Dec 10, 2020

junwang-noaa commented Jun 21, 2021

pjpegion commented Jun 21, 2021

pjpegion commented Jun 21, 2021

climbfuji commented Jun 21, 2021

DeniseWorthen commented Jun 30, 2021

Restart runs with stochastic physics due not reproduce #318

Restart runs with stochastic physics due not reproduce #318

Comments

climbfuji commented Dec 4, 2020

Description

To Reproduce:

pjpegion commented Dec 4, 2020 • edited Loading

climbfuji commented Dec 4, 2020

pjpegion commented Dec 4, 2020

climbfuji commented Dec 4, 2020

pjpegion commented Dec 4, 2020 via email

pjpegion commented Dec 4, 2020 • edited Loading

SMoorthi-emc commented Dec 4, 2020 via email

junwang-noaa commented Dec 4, 2020 via email

climbfuji commented Dec 4, 2020

pjpegion commented Dec 4, 2020

climbfuji commented Dec 4, 2020

SMoorthi-emc commented Dec 5, 2020

junwang-noaa commented Dec 5, 2020 via email

climbfuji commented Dec 5, 2020

SMoorthi-emc commented Dec 5, 2020 via email

climbfuji commented Dec 5, 2020

pjpegion commented Dec 8, 2020 via email

climbfuji commented Dec 8, 2020

junwang-noaa commented Dec 8, 2020 via email

pjpegion commented Dec 9, 2020

climbfuji commented Dec 9, 2020

climbfuji commented Dec 10, 2020

junwang-noaa commented Jun 21, 2021

pjpegion commented Jun 21, 2021

pjpegion commented Jun 21, 2021

climbfuji commented Jun 21, 2021

DeniseWorthen commented Jun 30, 2021

pjpegion commented Dec 4, 2020 •

edited

Loading

pjpegion commented Dec 4, 2020 •

edited

Loading