Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use stochastic restart patterns on rerun #3077

Open
wants to merge 7 commits into
base: develop
Choose a base branch
from

Conversation

WalterKolczynski-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA commented Nov 7, 2024

Description

The stochastic pattern restart files were not being copied into the input directory when restarting the model after a segment/failure. These files are now copied in and the stochini flag set to .true. in the namelist on a rerun.

The files are NOT copied in for non-rerun warm starts.

Also removes the restriction that stochastic physics cannot be run on member 0, as this is desired down the line. Additional settings are added to the fcst and efcs configs to retain the current behavior.

A bug was also discovered and corrected during this work. The stage, forecast, and archive job all assumed that ca_data tile files are always present, but these are only created when cellular automata is on. Now ca_data files are handled if CA is on. To implement this change, the DO_CA setting had to be moved from the forecast configs to base so it is available to the stage_ic and archive jobs.

Resolves #2937

Type of change

  • Bug fix (fixes something broken)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

Multi-segment test on Hercules

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

aerorahul
aerorahul previously approved these changes Nov 12, 2024
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me. No comments.
Please invite review from @pjpegion @NeilBarton-NOAA

@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Nov 13, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Nov 13, 2024
@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96_atm3DVar FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96_atm3DVar_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d9209edc/logs/2021032412/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_d9209edc/logs/2024022318/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmDA_d9209edc/logs/2021122018/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_d9209edc/logs/2021122012/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem000_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem001_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem002_seg1.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48_S2SWA_gefs_d9209edc

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 13, 2024
@emcbot
Copy link

emcbot commented Nov 13, 2024

CI Failed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3077


Experiment C48mx500_3DVarAOWCDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:45:45 UTC 2024
Experiment C48mx500_3DVarAOWCDA_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d9209edc/logs/2021032412/gdas_fcst_seg0.log
Experiment C96_atm3DVar_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:45:50 UTC 2024
Experiment C96_atm3DVar_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96_atm3DVar_d9209edc/logs/2021122018/gdas_fcst_seg0.log
Experiment C96C48_hybatmDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:56 UTC 2024
Experiment C96C48_hybatmDA_d9209edc Terminated: *FAIL*
Experiment C96C48_ufs_hybatmDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:57 UTC 2024
Experiment C96C48_ufs_hybatmDA_d9209edc Terminated: *FAIL*
Experiment C96C48_hybatmaerosnowDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:59 UTC 2024
Experiment C96C48_hybatmaerosnowDA_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmDA_d9209edc/logs/2021122018/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_d9209edc/logs/2024022318/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_d9209edc/logs/2021122012/gdas_fcst_seg0.log
Experiment C48_S2SWA_gefs_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Wed Nov 13 15:37:41 UTC 2024
Experiment C48_S2SWA_gefs_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem000_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem001_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem002_seg1.log
Experiment C96_S2SWA_gefs_replay_ics_d9209edc Completed 1 Cycles: *SUCCESS* at Wed Nov 13 17:25:52 UTC 2024
Experiment C48_ATM_d9209edc Completed 2 Cycles: *SUCCESS* at Wed Nov 13 18:45:20 UTC 2024
Experiment C48_S2SW_d9209edc Completed 2 Cycles: *SUCCESS* at Wed Nov 13 23:13:39 UTC 2024

The stochastic pattern restart files were not being copied into the
input directory when restarting the model after a segment/failure.
These files are now copied in. Additional changes appear to be
needed for them to be used by UFS.

The files are NOT copied in for non-rerun warm starts.

Refs NOAA-EMC#2937
Adds the namelist flag so the stochastic pattern restart files
get read when doing a RERUN.

Refs NOAA-EMC#2937
The forecast job would attempt to copy the ca_data tiled restart
files even when cellular automata was off. Now it only copies them
if cellular automata is on.
The stage and archive jobs were assuming that the ca_data restart
files will always be present. However, these are only produced
when cellular automata is run. So now they are conditionally
handled based on `DO_CA`.

To accomodate this, the `DO_CA` setting had to be moved out of
`config.fcst`/`config.efcs` and into `config.base`.
Reverts cellular automata to be on for all situations to match
current develop
@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 22, 2024
@@ -448,6 +448,15 @@ export INCVARS_EFOLD="5"
export netcdf_diag=".true."
export binary_diag=".false."

# Cellular automata
case ${RUN} in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is leftover from mixed DO_CA settings, but is the case section necessary now?

@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Nov 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

stage atm_stoch.res.nc and ocn_stoch.res.nc during segment forecasts
6 participants