Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor gridded wave post #3014

Merged
merged 140 commits into from
Nov 27, 2024
Merged

Refactor gridded wave post #3014

merged 140 commits into from
Nov 27, 2024

Conversation

AntonMFernando-NOAA
Copy link
Contributor

Description

The gridded wave post (wavepostsbs) script currently loops over all forecast hours and acts as its own post manager. Other component post and product scripts operate on one forecast time. This PR will update the wave post to match other components, that creates a bunch of short jobs instead of one long job (per member).
Resolves #2290

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

  • Run CI tests in HERA, ORION, and HERCULES

Checklist

  • Gridded wave post scripts that process only one forecast hour specified by the workflow manager (rocoto/ecflow)
  • Rocoto workflow updated to create the corresponding tasks for the new paradigm for both GFS and GEFS
  • Output identical to current system for both GFS and GEFS

AntonMFernando-NOAA and others added 26 commits October 17, 2024 16:56
@AntonMFernando-NOAA
modified gefs_tasks.py
26e06a3
@AntonMFernando-NOAA
Merge branch 'NOAA-EMC:develop' into feature/wave_post_grid
99dea4a
@AntonMFernando-NOAA
modified gefs_tasks.py
97b5932
@AntonMFernando-NOAA
modifed gefs_tasks.py
64552f8
@AntonMFernando-NOAA
modifed gefs_tasks.py
ed3768c
@AntonMFernando-NOAA
modified gefs_tasks.py
9373b88
@AntonMFernando-NOAA
modified gefs_tasks.py
916c1fa
@AntonMFernando-NOAA
modified gefs_tasks.py
f018d2e
@AntonMFernando-NOAA
modify gefs_tasks.py
aede2e9
@AntonMFernando-NOAA
modified exgfs_wave_post_gridded_sbs.sh
f59a6c9
@AntonMFernando-NOAA
modified gefs_tasks.py
f07b0ec
@AntonMFernando-NOAA
modified gefs.py
6261b4f
@AntonMFernando-NOAA
modified gefs_tasks.py
b2b3078
@AntonMFernando-NOAA
modified gefs_tasks.py
ea023be
@AntonMFernando-NOAA
modified gefs_tasks.py
3c69e48
@AntonMFernando-NOAA
modified gefs_task.py
16205f1
@AntonMFernando-NOAA
modified global-workflow\scripts\exgfs_wave_post_gridded_sbs.sh
0c7cb50
@AntonMFernando-NOAA
modified global-workflow\scripts\exgfs_wave_post_gridded_sbs.sh
458b130
@AntonMFernando-NOAA
modifed gefs_tasks.py
027e79f
@AntonMFernando-NOAA
modified global-workflow\scripts\exgfs_wave_post_gridded_sbs.sh
f75bbf3
@AntonMFernando-NOAA
corrected a typo
b408e75
@AntonMFernando-NOAA
mdified global-workflow\scripts\exgfs_wave_post_gridded_sbs.sh
d96631e
@AntonMFernando-NOAA
modified gefs_tasks.py
08c4f12
@AntonMFernando-NOAA
modified gefs_tasks.py
bc74e49
@AntonMFernando-NOAA
updated wavepostsbs.sh
99869ed
@AntonMFernando-NOAA
Merge branch 'develop' into feature/wave_post_grid
8b9d5d1
@AntonMFernando-NOAA AntonMFernando-NOAA marked this pull request as ready for review October 21, 2024 19:54
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Nov 26, 2024
@emcbot
Copy link

emcbot commented Nov 26, 2024

CI Passed on Hercules in Build# 3
Built and ran in directory /work2/noaa/global/CI/HERCULES/3014


Experiment C48_ATM_4b0523b0 Completed 2 Cycles: *SUCCESS* at Mon Nov 25 22:54:44 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_4b0523b0 Completed 1 Cycles: *SUCCESS* at Mon Nov 25 23:55:50 CST 2024
Experiment C96_atm3DVar_4b0523b0 Completed 3 Cycles: *SUCCESS* at Tue Nov 26 00:01:20 CST 2024
Experiment C96C48_hybatmDA_4b0523b0 Completed 3 Cycles: *SUCCESS* at Tue Nov 26 00:07:32 CST 2024
Experiment C48_S2SW_4b0523b0 Completed 2 Cycles: *SUCCESS* at Tue Nov 26 00:32:03 CST 2024
Experiment C48_S2SWA_gefs_4b0523b0 Completed 1 Cycles: *SUCCESS* at Tue Nov 26 02:52:29 CST 2024

@AntonMFernando-NOAA
Copy link
Contributor Author

@WalterKolczynski-NOAA @DavidHuber-NOAA I think all good now.

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Nov 26, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Nov 26, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Nov 26, 2024
@WalterKolczynski-NOAA
Copy link
Contributor

CI Tests set up to run in /lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_3014/RUNTESTS on WCOSS

@emcbot emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Nov 26, 2024
@AntonMFernando-NOAA AntonMFernando-NOAA self-assigned this Nov 26, 2024
@emcbot emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Nov 26, 2024
@emcbot
Copy link

emcbot commented Nov 26, 2024

CI Passed on Hera in Build# 4
Built and ran in directory /scratch1/NCEPDEV/global/CI_dh/3014


Experiment C48_ATM_232027f8 Completed 2 Cycles: *SUCCESS* at Tue Nov 26 19:52:38 UTC 2024
Experiment C48mx500_3DVarAOWCDA_232027f8 Completed 2 Cycles: *SUCCESS* at Tue Nov 26 20:16:59 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_232027f8 Completed 1 Cycles: *SUCCESS* at Tue Nov 26 20:48:26 UTC 2024
Experiment C96C48_hybatmaerosnowDA_232027f8 Completed 3 Cycles: *SUCCESS* at Tue Nov 26 21:48:52 UTC 2024
Experiment C96_atm3DVar_232027f8 Completed 3 Cycles: *SUCCESS* at Tue Nov 26 21:54:57 UTC 2024
Experiment C96C48_hybatmDA_232027f8 Completed 3 Cycles: *SUCCESS* at Tue Nov 26 21:54:58 UTC 2024
Experiment C48_S2SW_232027f8 Completed 2 Cycles: *SUCCESS* at Tue Nov 26 22:20:26 UTC 2024
Experiment C96C48_ufs_hybatmDA_232027f8 Completed 3 Cycles: *SUCCESS* at Tue Nov 26 22:37:31 UTC 2024
Experiment C48_S2SWA_gefs_232027f8 Completed 1 Cycles: *SUCCESS* at Tue Nov 26 23:06:46 UTC 2024

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Nov 27, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit 94f2029 into NOAA-EMC:develop Nov 27, 2024
5 checks passed
danholdaway added a commit to danholdaway/global-workflow that referenced this pull request Jan 27, 2025
@danholdaway
Merge branch 'develop' into feature/jedi_vs_gsi
fd62a43 
* develop:
  Remove WAFS files and references from `develop` (NOAA-EMC#3263)
  fix intel stack version number on c5 (NOAA-EMC#3258)
  Update gsi_monitor and ufs_utils hashes to recent hashes for C5/C6 build and run (NOAA-EMC#3252)
  Enable DA cycling on gaea C5/C6 (NOAA-EMC#3255)
  Copy post-processed sea ice increment for diagnostics (NOAA-EMC#3235)
  Only run METplus in the 3Dvar tests (NOAA-EMC#3245)
  Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 (NOAA-EMC#3106)
  Add echgres as a dependency only for RUN=enkfgdas, not enkfgfs (NOAA-EMC#3246)
  Add domain level to wave gridded COM path (NOAA-EMC#3137)
  CI JJOB Tests using CMake (NOAA-EMC#3214)
  Make assorted updates to waves (NOAA-EMC#3190)
  Move WCOSS2 LD_LIBRARY_PATH patches to load_ufsda_modules.sh (NOAA-EMC#3236)
  Adding a gefs_arch task to GEFS workflow (NOAA-EMC#3211)
  Add additional GEFS variables needed for AI/ML applications  (NOAA-EMC#3221)
  Add bmat task dependency to marine LETKF task (NOAA-EMC#3224)
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
  Separate use of initial increment/perturbation file from REPLAY/+03 ICs  (NOAA-EMC#3119)
  Update gsi_enkf hash and gsi_ver (NOAA-EMC#3207)
  Remove cpus-per-task from APRUN_OCNANALECEN on WCOSS2 (NOAA-EMC#3212)
  Remove 5WAVH from AWIPS GRIB2 parm files (NOAA-EMC#3146)
  Remove multi-grid wave support (NOAA-EMC#3188)
  Add echgres as a dependency for earc (NOAA-EMC#3202)
  Ensure OCNRES and ICERES have 3 digits in the archive script (NOAA-EMC#3199)
  Set runtime shell requirements within Jenkins Pipeline (NOAA-EMC#3171)
  Add efcs and epos to ufs_hybatm xml (NOAA-EMC#3192) (NOAA-EMC#3193)
  Fix GEFS and SFS compile flags in build_all.sh (NOAA-EMC#3197)
  Remove early-cycle EnKF forecast (NOAA-EMC#3185)
  Fix mod_icec bug in atmos_prod (NOAA-EMC#3167)
  Create compute build option (NOAA-EMC#3186)
  Support global-workflow using Rocky 8 on CSPs (NOAA-EMC#2998)
  Change orog gravity wave drag scheme for grid sizes less than 10km (NOAA-EMC#3175)
  Switch snow DA to use 2DVar for deterministic and ensemble mean (NOAA-EMC#3163)
  Update compression options for GEFS history files (NOAA-EMC#3184)
  Update compression options for high res history files (NOAA-EMC#3178)
  Turn DO_TEST_MODE off (NOAA-EMC#3177)
  Hotfix for gdas_arch div/0 (NOAA-EMC#3169)
  Allow building of the ufs-weather-model, WW3 pre/post execs for GFS, GEFS, SFS in the same clone of global-workflow (NOAA-EMC#3098)
  Switch Aerosol DA to use JCB and Jedi class (NOAA-EMC#3125)
  Update ufs-weather-model to 2024-12-06 commit  (NOAA-EMC#3145)
  Enable traditional threading as an option (NOAA-EMC#3149)
  Update HPC_ACCOUNT on Hercules to fv3-cpu (NOAA-EMC#3164)
  Turn C96C48_ufs_hybatmDA and C48mx500_3DVarAOWCDA into a regression test (NOAA-EMC#3120)
  Update GSI analysis jobs to use COMIN/COMOUT (NOAA-EMC#3092)
  Update HPC Tier Definitions (NOAA-EMC#3138)
  Add marine hybrid envar (NOAA-EMC#3041)
  Archive the experiment directory along with git status/diff output (NOAA-EMC#3105)
  Use stochastic restart patterns on rerun (NOAA-EMC#3077)
  Point Jenkinsfile back to CI/ (NOAA-EMC#3139)
  Fix wave restart for cold start and add ic version file (NOAA-EMC#3112)
  Allow users to override the default account at setup time (NOAA-EMC#3127)
  Refactor gridded wave post (NOAA-EMC#3014)
  Update docs related to NOAA CSPs (NOAA-EMC#3043)
  Allow APP to differ between RUNs (NOAA-EMC#2943)
  Run one executable for soca2cice (instead of two) (NOAA-EMC#3118)
  Speed up GSI analysis jobs in CI testing (NOAA-EMC#3115)
  Make aerosol output frequency variable (NOAA-EMC#2982)
  Add new stations to GFS BUFR sounding products (NOAA-EMC#3107)
  JCB-based obs+bias staging, Jedi class updates, and marine B-matrix refactoring (NOAA-EMC#2992)
  Enable tapering of atm ens perts at the model top (NOAA-EMC#3097)
  Update JGDAS ENKF POST  job  (NOAA-EMC#3090)
  SFS Runs at C96mx100  (NOAA-EMC#2960)
  Move machine-based options from config.base to host files (NOAA-EMC#3053)
  Remove RUNDIRS before running CI cases to cover re-run events (NOAA-EMC#3076)
  CI GitHub pipeline (hotfix) update for fetching repo name (NOAA-EMC#3084)
  Update JGDAS ENKF ECEN job  (NOAA-EMC#3050)
  Update snow obs processing job (NOAA-EMC#3055)
  Update to action workflow pipeline in default repo for development  (NOAA-EMC#3062)
  Update to action workflow pipeline in default repo for development (NOAA-EMC#3061)
  Update workflow pipeline (NOAA-EMC#3060)
  PW CI pipeline update5 ready for review so it can be merged and tested (NOAA-EMC#3059)
  Revert "GitHub CI Pipeline update for debugging forked PR support" (NOAA-EMC#3057)
  GitHub CI Pipeline update for debugging forked PR support (NOAA-EMC#3056)
  Add more ocean variables for post-processing in GEFS (NOAA-EMC#2995)
  Auto provisioning of PW clusters from GitHub CI added (NOAA-EMC#3051)
  Fix the name of the TC tracker filenames in archive.py (NOAA-EMC#3030)
  Make wxflow links static instead of from link_workflow (NOAA-EMC#3008)
  Update global jdas enkf diag job with COMIN/COMOUT for COM prefix (NOAA-EMC#2959)
  Add run and finalize methods to marine LETKF task (NOAA-EMC#2944)
  Fix wave restarts and GEFS FHOUT/FHMAX (NOAA-EMC#3009)
  Disabling hyper-threading (NOAA-EMC#2965)
  GitHub Actions Pipeline Updates for Self-Hosted Runners on PW (NOAA-EMC#3018)
  CI jekninsfile update hotfix (NOAA-EMC#3038)
  Update gdas.cd (NOAA-EMC#2978)
  Add ability to add tag to pslots with generate_workflows (NOAA-EMC#3036)
  CI update to shell environment with HOMEgfs to HOME_GFS for systems that need the path (NOAA-EMC#3013)
  Quick updated to Jenkins (health check) launch script (NOAA-EMC#3033)
  Document the generate_workflows.sh script (NOAA-EMC#3028)
  Replace gfs_cyc with an interval (NOAA-EMC#2928)
  Hotfix: Fix generate_workflows.sh optional build flags (NOAA-EMC#3024)
  Add a tool to run multiple YAML cases locally (NOAA-EMC#3004)
  Hotfix: Correctly set overwrite option when specified (NOAA-EMC#3021)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DavidHuber-NOAA DavidHuber-NOAA DavidHuber-NOAA left review comments

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@JessicaMeixner-NOAA JessicaMeixner-NOAA JessicaMeixner-NOAA left review comments

@aerorahul aerorahul Awaiting requested review from aerorahul aerorahul is a code owner

@sbanihash sbanihash Awaiting requested review from sbanihash sbanihash is a code owner

@KateFriedman-NOAA KateFriedman-NOAA Awaiting requested review from KateFriedman-NOAA

Assignees

@AntonMFernando-NOAA AntonMFernando-NOAA

Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Refactor gridded wave post to match other components
10 participants
@AntonMFernando-NOAA @JessicaMeixner-NOAA @sbanihash @MatthewMasarik-NOAA @KateFriedman-NOAA @NeilBarton-NOAA @WalterKolczynski-NOAA @emcbot @aerorahul @DavidHuber-NOAA