GFS v16 workflow development #1
Assignees
Labels
production update
Processing update in production
Comments
RussTreadon-NOAA
added a commit
that referenced
this issue
Dec 16, 2019
* jobs/rocoto/ecen.sh - remove FHSFC_ECEN * ush/rocoto/setup_workflow.py - remove fhrsfc and associated scripting
RussTreadon-NOAA
added a commit
that referenced
this issue
Dec 18, 2019
RussTreadon-NOAA
added a commit
that referenced
this issue
Dec 19, 2019
…og file dependency for post000 job
RussTreadon-NOAA
added a commit
that referenced
this issue
Dec 19, 2019
(1) provide default defintion for CONVNETC (2) update WCOSS_DELL_P3 fitdir
RussTreadon-NOAA
added a commit
that referenced
this issue
Dec 20, 2019
… in exglobal_fcst_nemsfv3gfs.sh
yangfanglin
added a commit
that referenced
this issue
Dec 25, 2019
to save sfcanl for running gldas as ICs
RussTreadon-NOAA
added a commit
that referenced
this issue
Dec 26, 2019
…t DOSFCANL_ENKF in config.esfc
RussTreadon-NOAA
added a commit
that referenced
this issue
Dec 26, 2019
RussTreadon-NOAA
added a commit
that referenced
this issue
Dec 27, 2019
|
After seeing a banding feature in the VSDB analysis increment plots, the temperature increments were examined in greater detail, comparing netCDF increment files, netCDF and nemsio model and utility output, and post output. The results were documented and sent to post and key GFSv16 developers. |
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 6, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 6, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 6, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 7, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 7, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 7, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 7, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 7, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 7, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 7, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 8, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 9, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 10, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 15, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 15, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 15, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 15, 2020
BoiVuong-NOAA
added a commit
that referenced
this issue
Jan 15, 2020
KateFriedman-NOAA
added a commit
that referenced
this issue
Mar 16, 2021
- Final nwprod implementation component tag versions set in Externals.cfg and checkout.sh - GLDAS: roll back to gldas_gfsv16_release.v1.12.0 - EMC_verif-global: roll back to verif_global_v1.11.0 - WAFS: update in Externals.cfg to gfs_wafs.v6.0.19 Notes about local nwprod install changes: - EMC_post: nwprod install shows upp_gfsv16_release.v1.1.1 tag but contains local updates found in upp_gfsv16_release.v1.1.3 tag - WAFS: nwprod install shows gfs_wafs.v6.0.17 tag but contains local updates found in gfs_wafs.v6.0.19 tag Refs: #1,#238
KateFriedman-NOAA
added a commit
that referenced
this issue
Mar 18, 2021
- updates to the transfer parm files were erroneously committed after a diff between nwpara/gfsv16.0.7 and release/gfsv16.0.0 branch; v16.0.7 had been moved to nwprod/gfsv16.0.7 and transfer parm file diffs were not in final install - confirmed with SPA Jen that nwprod/gfsv16.0.7 is finalized (it is) - pulled in transfer parm files from nwprod/gfsv16.0.7 to sync them again Refs: #1,#238
|
Created branch release/gfsv16.0.0_to_ops off of operations branch for merging release/gfsv16.0.0 branch into the operations branch post-GFSv16 implementation on March 22nd 12z. Will use release/gfsv16.0.0_to_ops branch to resolve conflicts before submitting PR of release/gfsv16.0.0_to_ops into operations branch. |
KateFriedman-NOAA
added a commit
that referenced
this issue
Mar 23, 2021
Merge remote-tracking branch 'origin/release/gfsv16.0.0' into release/gfsv16.0.0_to_ops * origin/release/gfsv16.0.0: (1073 commits) Reverting transfer parm file changes committed at 39bab45 Component tag updates for nwprod/gfsv16.0.7 Updated transfer parm files for gdas, enkf, and gfs dissemination Add config.resources.nco.static ecflow forecast job resource updates from NCO v16.0.7 install Add missing symlinks for WAFS source code folders Update EMC_gfs_wafs tag to gfs_wafs.v6.0.19 Remove KEEPDATA from config.base.nco.static Pull in config changes from NCO v16.0.7 install Pull in workflow changes from NCO v16.0.7 install Update EMC_verif-global tag to verif_global_v1.13.4 ecFlow resource adjustments from NCO for forecast and post jobs issue #257 shorten run time on Mars Dell1 file system Update Fit2Obs tag to newm.1.3 for bugfix Update EMC_verif-global tag for Hera bug Pull in nwpara/gfsv16.0.6 updates for parse-storm-type.pl Issue #1 and issue #238 - update Externals.cfg to match checkout.sh updates for v16.0.6 move errchk definition from script to job for wave prdgen and gempak adding definition of errchk which was undefined in these scripts Issue #1 and issue #233 - update GLDAS tag to gldas_gfsv16_release.v1.13.0 Issue #1 and issue #241 - update EMC_verif-global tag Issue #1 and issue #238 - remove ak_10m grid from config.wave and update checkout.sh tags for ufs-weather-model and EMC_gfs_wafs Issue #1 and issue #226 - update UPP tag to upp_gfsv16_release.v1.1.3 in Externals.cfg Issue #1 and issue #226 - update UPP tag to upp_gfsv16_release.v1.1.3 Issue #1 and issue #226 - update exgfs_atmos_grib2_special_npoess.sh for dbn_alert issue #227 reducing output bufr files to 64 levels per NCO request issue #227 reducing output bufr files to 64 levels per NCO request Issue #1: update ecflow to be consistent with NCO's gfs.v16.0.4 and update checkout to bring in new UPP tag (see issue #226) Upaded ush script scale_dec.sh Updated scripts gfs_v16.0 Issue #1 - add grib_util module load to several analysis ecflow scripts Issue #1 - update WAFS tag to gfs_wafs.v6.0.17 for dbn_alert change Issue #1 - correct gridded wave parm files for v16.0.3 Issue #1 - changes from NCO for GFSv16.0.3 Issue #1 - update Externals.cfg with final tags for GFSv16.0.2 Issue #1 - correct permissions on jobs/rocoto/postsnd.sh Issue #1 - update gempak version to 7.3.3 in the ecflow gfs.ver file Issue #1 - update gempak and dumpjb versions to 7.3.3 and 5.1.0 respectively Script alert updates from NCO for wave downstream Add override for COMIN_WAV_RTOFS in emc mode for waveprep job Issue #197: place CDATE specific sections of config.anal and config.prep inside RUN_ENVIR=emc blocks. These sections are use for retrospective parallels and therefore do not need to be executed in operations (NCO). modified: JGLOBAL_FORECAST modified: JGLOBAL_FORECAST to make it work for both emc and nco running environments. modified: JGLOBAL_FORECAST A test showed that jobid is not defined in JGLOBAL_FORECAST running in the Rocoto environment. jobid is defined in ./env files. Defining DATA without sourcing ./env/$nachine.env caused the script to fail. Move the definitin of DATA after sourcing env parameters modified: checkout.sh to use WAFS tag gfs_wafs.v6.0.16 modified: checkout.sh to update UPP to upp_gfsv16_release.v1.1.1, a minor syntax bug fix modified: link_fv3gfs.sh to 1) use hard copies of external fix fields and executable for NCO installation 2) use soft links for all other files and directories for both NCO and EMC installations Compared local files in NCO implementation directory with release/gfs.v16.0.0 branch, changes made by NCO(Jen Yang) in the following files are either accepted or rejected. create a new branch release/gfsv16.0.0.nco to merge changes made by NCO in /gpfs/dell1/nco/ops/nwpara/gfs-v16/gfs.v16.0.1 back to EMC's repository update for the wave parm so that the wave model will look for the correct restart for when gfs is not run every cycle updates for checking if RTOFS files exist and only processing RTOFS files for needed fhr Issue #1 - update WAFS tag to gfs_wafs.v6.0.14 and update dumpjb version to 5.1.0 modified: jobs/JGFS_ATMOS_POSTSND and jobs/rocoto/postsnd.sh to remove redundant variables in the two scripts and make them work for both EMC and NCO parallels. Issue #1 - update WAFS tag to gfs_wafs.v6.0.13 Rename Release_Notes.gfs.v16.0.0.txt to Release_Notes.gfs.v16.0.0.md Issue #1 - update WAFS tag to gfs_wafs.v6.0.12 for removal of in-cloud turbulence per AWC Issue #1 - pull in corrected npe_eobs values in config.resources Issue #1 - update FV3 tag to GFS.v16.0.14 for Hera/Orion build support Issue #1 - adjust WAFS dependencies to wait for f036 post output Issue #1 - adding release notes for GFSv16 Issue #94 producing awips files with masks and deleting wmo headers for arctic ocean updates to add glo_30m to the created grib files for waves for awips processing Modify gfs/gdas post job to 20 minutes in wall clock. Issue #1 - update link_fv3gfs.sh to point to newly frozen fix_nco_gfsv16 FIX_DIR Modify two wafs jobs trigger as: jgfs_atmos_wafs_grib2 trigger ../../post/jgfs_atmos_post_f000 == complete jgfs_atmos_wafs_grib2_0p25 trigger ../../post/jgfs_atmos_post_f036 == complete Issue #1 - update WAFS tag to gfs_wafs.v6.0.10 and change WAFS job dependencies Issue #1 - update config.fv3 and config.resources with v16rt2 values Issue #1 - update gfs_util modulefiles Change config.resources for eobs for low resolution cases Issue #1 - update to fbwndgfs modulefiles for WCOSS-Dell and WCOSS-Cray updates to parm to reduce the number of wave variables changes to the config so that wave models are interpolated to the multi_1 masked files for the regional output grids Modify module for each job to match implementation package change Modify two wafs jobs trigger Modify wall clock and resource for running jobs in NCO Modify obsproc package location add a dependency for the wavepostpnt on wavepostbndpnt for just gfs as this job does not exist for gdas Issue #1 - update WAFS tag to gfs_wafs.v6.0.9 Issue #1 - remove POE/BACK block from config.prep and set POE=YES/BACK=off as defaults in env/WCOSS_DELL_P3.env prep section Issue #1 - remove unneeded DMPDIR and ICSDIR from config.base.nco.static for rocoto add a dependency to wavepostpnt job on wavepostbndpnt so that both jobs will not run at the same time which will slow both jobs down. This is the reason for the dependency, otherwise there is not a "true" dependency between the jobs Issue #1 - return POE=YES and BACK=off setting for prep on WCOSS_DELL_P3 Issue #1 - remove hardcoded POE and BACK values from config.prep Issue #1 - move ABIBF, AHIBF, and HDOB pointers into RUN_ENVIR=emc block Issue #1 - update config.fv3 based on real-time parallel Issue #1 - update prep job resources Issue #1 - update g2tmpl module load in modulefiles/module_base.wcoss_dell_p3 Issue #1 - config updates from real-time parallel Issue #1 - remove unneeded line in vrfy.sh and update link_fv3gfs.sh for UFS_UTILS execs Issue #1 - update GSI tag to gfsda.v16.0.0 Issue #1 - update component tags and modulefiles for nwtest lib updates, remove unneeded module load and modulefile from downstream wave job rocoto scripts Issue #1 - adjust error handling in wave rocoto job scripts issue #142 generate station i,j grid issue #142 generate station i,j grid issue #142 generate station i,j grid issue #142 add 6 bufr station data issue #142 add 6 bufr stations for Thailand TMD issue #145 change dev path to prod for parallel netcdf modules issue #145 change dev path to prod for parallel netcdf modules Issue #1 - updates for modules and small fixes reverting changes to configs that were not intented to be committed fix resource time estimates bug fix in exgfs_wave_post_pnt.sh update resources and trigger from 192->180 last of EE2 changes updates for EE2 from waves Issue #94 fix for failing silently Issue #1 - update WAFS tag to gfs_wafs.v6.0.8 Issue #94 add native grids as default grids Issue #1: update name of ncdiag executable and source code directory to be conistent cleaning up the rearranged scripts Jobs were tested with PDY 20200925, code managers from post, gempak, wave, and post process certified the test run result. Issue #1: update parm/config.vrfy to define VSDBJOBSH (used by jobs/rocoto/vrfy.sh) Code manager indicated all wafs jobs wall clock is 30 mins. Code manager indicated job card for scripts/gfs/atmos/gempak/jgfs_atmos_pgrb2_spec_gempak.ecf need to be changed The EMC realtime parallel does not use operational job settings. Ecflow job card roll back the setting from module_used_gfs-16_job google sheet document. Update GLDAS tag to gldas_gfsv16_release.v1.10.0 Update gfswafs job to run with loop over fcsthrs Issue #1: update vrfy.sh to submit vsdb processing as separate job (only on WCOSS_DELL_P3) Issue #94 add /fakedbn to run DBN_alerts Update config.awips for newly named JJOB scripts Update WAFS jobs/rocoto scripts to use new JJOB names Making J-Job naming change accourding to code manager. Remove temp files Update post.sh UPP JJOB script name to submit Modify each ecflow script with old j-job name for test. Update config.base.nco.static with config.base.emc.dyn changes Script name updates for sfc_prep and tracker Name change for tropcy scripts and update WAFS tag Fixing spelling mistake in config.gldas Updated drivers and release notes Updated scripts Update Externals.cfg with new UPP tag upp_gfsv16_release.v1.0.16. 1)Update sorc/checkout.sh with new UPP tag upp_gfsv16_release.v1.0.16. 2)Update sorc/link_fv3gfs.sh with new file name convention for jjob and ex-script of post processing part. Updated jobs name Update EMC_verif-global tag to verif_global_v1.11.0 Rename scripts to match ecf script naming convention. Add SENDDBN and DBNROOT. Update GLDAS tag. In anticipating changes from the GLDAS repo : renaming JGDAS_GLDAS to JGDAS_ATMOS_GLDAS, and exgdas_gldas.sh to exgdas_atmos_gldas.sh renamed: jobs/JGFS_POSTSND -> jobs/JGFS_ATMOS_POSTSND renamed: scripts/exgfs_postsnd.sh -> scripts/exgfs_atmos_postsnd.sh modified: docs/archive/README_bufr driver/product/run_postsnd.sh driver/product/run_postsnd.sh.cray driver/product/run_postsnd.sh.dell driver/product/run_postsnd.sh.hera driver/product/run_postsnd.sh.jet parm/config/config.postsnd renamed: scripts/exglobal_fcst_nemsfv3gfs.sh -> scripts/exglobal_forecast.sh and modified jobs/JGLOBAL_FORECAST parm/config/config.fcst updates for optimizing point jobs Issue #1 - update SEND variables and add DBNROOT to base configs and add check to build_enkf_chgres_recenter_nc.sh for GSI build Issue #1 - update to UFS_UTILS ops-gfsv16.0.0 tag Issue #1 - fix to link_fv3gfs.sh for new GLDAS tag Issue #131 Unify dbn_alert path Issue #1: update to UPP tag "upp_gfsv16_release.v1.0.15" Fix for running prep on Hera ecflow full day cycle included Update WAFS tag to gfs_wafs.v6.0.6 Small updates: - new UPP tag - new GLDAS tag - new WAFS tag - new module for WAFS - EE2 updates to awips scripts - added WAFS to archival - break downstream and WAFS archival into separate gfs_downstream tarball - update gfsarch dependencies to wait for all wavepost jobs to complete Issue #131 reduce scripts output to logfile Issue #131 added a path to DBNROOT Adding wafs wave and downstream jobs updates for by hour post Issue #1: add fhrgrp and fhrlst back to gfsawips in setup_workflow.py (bugfix) adding the line to go back a day for RTOFS for the if not NCO section because RTOFS will not be available until 06 cycle Restructured ecflow - up to post step deleted relocate_mv_nvortex.fd since storm relocation is no longer needed. modified build_tropcy_NEMS.sh to remove references to relocate_mv_nvortex Issue #1: correct DA typos in sorc/link_fv3gfs.sh Increase walltime for new wavepost jobs Issue #1: update name of DA jobs and scripts in accordance with WCOSS Implementation Standards Issue #94 add waves-prdgen, ICE->ICEC, Sleep in gempak script add gfs gempak downstream jobs into def file worked on wcoss ecflow script rename after redesign approved - not including all wave jobs Issue #1: clean up DA sections of link_fv3gfs.sh fix from Bhavani for having first wave grib file be set as a forecast instead of analysis fix from Bhavani for having first wave grib file be set as a forecast instead of analysis fix from Bhavani for having first wave grib file be set as a forecast instead of analysis ecflow gfs v16 nco review 3 updates to split boundary points plus saving config file updates Adding missing space to if-block in env files to resolve runtime failure Issue #1: remove pgrb2b.0p25 dependency from gfsawips in setup_workflow.py ecflow gfsv16 redesign 2 Issue #1 - update WAFS tag to gfs_wafs.v6.0.4 and remove HOURLY variable from WAFS configs Issue #1 - change wavegempak and waveawipsgridded dependency to match waveawipsbulls and start when wavepostsbs is complete Issue #1: set n_sponge=42 in gfs section of config.fcst adding pnt jobs as seperate jobs for env moving definitions of wavempexec and wave_mpmd from jobs to env Remove unneeded settings from config.post Added null DBNROOT to wave awips configs add extra script for by hour points for waves updates for boundary points by hour parallelization lowering the resource requirement for wave prep job Issue #1: rename enkf_chgres_recenter executables in accordance with WCOSS Implementation Standards modified: checkout.sh to use gldas_gfsv16_release.v1.6.0 update module for cdo Update to WAFS tag and added SENDDBN_NTC to both base configs modified: link_fv3gfs.sh to not link or copy 0readme fix_chem fix_fv3 fix_sfc_climo which are not used by GFS.v16 and are of large size modified: link_fv3gfs.sh to remove chgres_cube.fd and chgres_cube.fd in sorc/link_fv3gfs.sh modified: link_fv3gfs.sh to allow "fix" directories to be removed before rerunning link_fv3gfs.sh for RUN_ENVIR=nco case Added WAFS jobs to free-forecast mode, updates for extending WAFS to fh120, and two bug fixes in link_fv3gfs.sh and hpssarch_gen.sh Renamed global-workflow-owned ex-scripts to remove ecf extension and updated other scripts which call those ex-scripts Remove UFS_UTILS ecf extensions Issue #1: (1) update earc.sh directory removal to be consistent with arch.sh, (2) update config files to be consistent with EMC real-time GFS v16 parallel Remove ecf script name extensions from downstream wave scripts Add new downstream wave jobs to workflow Add new downstream WAFS jobs Issue #1 - update GLDAS and UPP workflow files for removal of ecf script extension Issue #1: remove ".ecf" suffix from DA scripts referenced in sorc/link_fv3gfs.sh Issue #1: remove ".ecf" extension from DA exscripts (as per WCOSS Implementation Standards) referenced from parm/config files updates to resources for wave jobs Issue #1: update name of DA enkf chgres script in config.echgres Issue #1: Rename DA enkf chgres job and script as per EE2 guidance Workflow changes for wave gempak and awips downstream jobs Fixing wavepostbndpnt dependency in setup_workflow_fcstonly.py ... * conflicts resolved Refs: #1
|
Performed merge of release/gfsv16.0.0 branch into release/gfsv16.0.0_to_ops branch. Resolved conflicts in following files: Committed merge to release/gfsv16.0.0_to_ops at de7b54e. |
This was referenced Mar 23, 2021
Merged
KateFriedman-NOAA
added a commit
that referenced
this issue
Mar 25, 2021
Merge remote-tracking branch 'origin/release/gfsv16.0.0' into release/gfsv16.0.0_to_ops * origin/release/gfsv16.0.0: Fix cycle date in bull and cbull wave files Refs: #1
KateFriedman-NOAA
added a commit
that referenced
this issue
Mar 30, 2021
Tag updates:
1. Change WAFS 1.25 products (except for icing & turbulence) on model pressure levels.
2. Add CAT and MWT back to 0.25 WAFS product
Refs: #1
KateFriedman-NOAA
added a commit
that referenced
this issue
Mar 30, 2021
KateFriedman-NOAA
added a commit
that referenced
this issue
Mar 30, 2021
KateFriedman-NOAA
added a commit
that referenced
this issue
Mar 30, 2021
KateFriedman-NOAA
added a commit
that referenced
this issue
Mar 30, 2021
Will retain in other release branch. Removing to keep continuity with nwprod for GFSv16 post-implementation fix updates. Refs: #1
KateFriedman-NOAA
added a commit
that referenced
this issue
Mar 30, 2021
KateFriedman-NOAA
added a commit
that referenced
this issue
Mar 30, 2021
KateFriedman-NOAA
added a commit
that referenced
this issue
Apr 13, 2021
- Remove wavepostbndpntbll trigger from prod def file for each cycle Refs: #1
|
PR #306 merged into operations branch. Operations branch tagged for NCO-v16.0.9 and released. |
KateFriedman-NOAA
added a commit
that referenced
this issue
May 6, 2021
KateFriedman-NOAA
added a commit
that referenced
this issue
May 6, 2021
|
Now that GFSv16 is in operations I will close this issue. Future updates to the v16 system will be documented in their own issues. Thanks for all your hard work everyone! |
KateFriedman-NOAA
pushed a commit
that referenced
this issue
Nov 30, 2021
9 tasks
4 tasks
This was referenced Apr 7, 2024
Closed
Closed
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This issue is being opened to track workflow development for GFS v16.
Branch for work: feature/gfsv16b
Please see prior issue comments and commits:
https://vlab.ncep.noaa.gov/redmine/issues/65358
The text was updated successfully, but these errors were encountered: