Merge branch 'develop' into feature/version-files

* develop:
  Correct issue in linking final restart files (NOAA-EMC#1285)
  Remove execute permissions from config files (NOAA-EMC#1281)
  Make needed updates to run forecast from GEFS (NOAA-EMC#1203)
  Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259)
  Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237)
  Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280)
  More marine DA j-jobs (NOAA-EMC#1270)
  Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265)
  Add new jjob for ocean analysis bmat (NOAA-EMC#1239)
  Retire ecf/versions in develop (NOAA-EMC#1267)
  Deploy documentation to RTD (NOAA-EMC#1264)
  Temporarily disable failing pytest (NOAA-EMC#1263)
  Remove incorrect/misleading comments in config.base (NOAA-EMC#1261)
  Add initial Sphinx documentation (NOAA-EMC#1258)
  Remove nemsio support (NOAA-EMC#1255)
  Increase wallclock for diag jobs (NOAA-EMC#1216)
  Use correct resources for GFS gempak (NOAA-EMC#1214)
  Abstract common j-job tasks (NOAA-EMC#1230)
  Add missing mkgfsawps.x link (NOAA-EMC#1218)
  Fix post sounding job (NOAA-EMC#1212)
  Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240)
  Use fracoro data for all new UFS applications (NOAA-EMC#1182)
  Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238)
  Add more user defined parameters for the marine DA (NOAA-EMC#1235)
  Update pytests action version and run sequentially (NOAA-EMC#1236)
  Add utility to compare Fortran namelists (NOAA-EMC#1234)
  Updates for pygw (NOAA-EMC#1231)
  Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)
  Move member up in directory hierarchy (NOAA-EMC#1201)
  Enable staging ics for cycled experiments. (NOAA-EMC#1199)
  Add tests for configuration.py (NOAA-EMC#1192)
  Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191)
  define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193)
  Fix checking for restart files (NOAA-EMC#1186)
  Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188)
  Update archive job memory request value for R&Ds (NOAA-EMC#1183)
  Reorder post so all flux files are generated when running offline (NOAA-EMC#1181)
  Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179)
  Add missing jobids in some pre-job scripts (NOAA-EMC#1176)
  Remove existing directory if it exists when getic runs (NOAA-EMC#1165)
  Add logging decorator, test and test for yaml_file (NOAA-EMC#1178)
  fix coding norm check in `hosts.py` (NOAA-EMC#1174)
  Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172)
  Support for the GDASApp testing in containers (NOAA-EMC#1151)
  ATM 3DVAR with and without IAU (NOAA-EMC#1113)
  Enable checking for python norms and fix violating code (NOAA-EMC#1168)
  Enforce decimal math in atmos post (NOAA-EMC#1171)
  Update marine DA j-jobs to new format (NOAA-EMC#1149)
  Add utility to manipulate files en masse  (NOAA-EMC#1166)
  add action to run pytests (NOAA-EMC#1167)
  Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162)
  Add a task base class and basic logger (NOAA-EMC#1160)
  Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154)
  move configuration.py to pygw. Use it from there.  return AttrDict after sourcing configs (NOAA-EMC#1153)
  JEDI based Marine DA tasks (NOAA-EMC#1134)
  Allow customizations based on user/configuration (NOAA-EMC#1146)
  First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120)
  enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142)
  Fix typo in .shellcheckrc
  Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138)
  Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132)
  Address shellcheck warnings in env files (NOAA-EMC#1136)
  Adds group size and nmem for GEFS (NOAA-EMC#1127)
  Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133)
  Convert archive jobs to proper j-jobs (NOAA-EMC#1115)
  Update C48 forecast to run with one thread (NOAA-EMC#1131)
  Improved error messages from atmos analysis (NOAA-EMC#1125)
  Update MODULEPATH for Orion (NOAA-EMC#1126)
  MPMD variable updates and fix (NOAA-EMC#1124)
  Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122)
  Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112)
  Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111)
  Correct syntax in remaining sorc scripts (NOAA-EMC#1105)
  Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104)
  Add Early Cycle EnKF workflow (NOAA-EMC#1022)
  Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101)
  Fixed gfs-utils links (NOAA-EMC#1099)
  Fix build scripts and bring into compliance (NOAA-EMC#1096)
  Feature/updates for gdas app (NOAA-EMC#1091)
  Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094)
  Resource updates to support WCOSS2 (NOAA-EMC#1070)
  Set COMPILER in link for detect machine (NOAA-EMC#1092)
  gfs utils update (NOAA-EMC#1088)
  GFS-UTILS update for build and ush scripts (NOAA-EMC#1082)
  Update UFS version to 2022 Oct 19 (NOAA-EMC#1083)
  Use more cycledefs for task control (NOAA-EMC#1078)
  removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079)
  Update UFS to Sept 9 version (NOAA-EMC#1073)
  Modify default file location for monitor data when using rocoto (NOAA-EMC#1065)
  Fix companion ocean resolution for C48 (NOAA-EMC#1066)
  Add trailing slash for gldas topo path (NOAA-EMC#1064)
  Limit number of CPU for post (NOAA-EMC#1061)
  Fix eupd trace (NOAA-EMC#1057)
  Port to S4 (NOAA-EMC#1023)
  Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049)
  Add GDAS to the partial build list (NOAA-EMC#1050)
  Fix group number being treated as octal in gdas arch (NOAA-EMC#1053)
  Remove trace from link script (NOAA-EMC#1046)
  Update gfs-utils hash to 3a609ea (NOAA-EMC#1048)
  Fix link script usage statement (NOAA-EMC#1045)
  Replace preamble variable commands with functions (NOAA-EMC#1012)
  Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009)
  Rename post scripts (NOAA-EMC#1038)
  Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039)
  WCOSS2 run support and script/config updates (NOAA-EMC#1030)
  Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036)
  initial commit for incoming yaml work (NOAA-EMC#1029)
  Fix radiance verification failing to find diag files (NOAA-EMC#1031)
  Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026)
  Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018)
  Update GSI Monitor for radmon fix
  Correct shell linter config (NOAA-EMC#1013)
  Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016)
  Add shell linter Github action for pull requests (NOAA-EMC#1007)
  Build updates for WCOSS2 (NOAA-EMC#1002)
  Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001)
  Fix preamble id (NOAA-EMC#996)
  Add missing "atmos" into job dependencies (NOAA-EMC#998)
  Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992)
  Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990)
  Add GSI monitor scripts (NOAA-EMC#969)
  Fix product generation at some fcst hrs (NOAA-EMC#988)
  Add initial config files for global aerosol DA (NOAA-EMC#986)
  Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979)
  use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970)
  Externals.cfg was stale and had drifted off (NOAA-EMC#965)
  Fix post comparison with zero-padded numbers (NOAA-EMC#964)

.github/scripts/build_docs.sh

@@ -0,0 +1,31 @@
+#! /bin/bash
+
+set -eux
+
+# path to docs directory relative to top level of repository
+# $GITHUB_WORKSPACE is set if the actions/checkout@v3 action is run first
+
+cwd=$(pwd)
+DOCS_DIR="${GITHUB_WORKSPACE}/docs"
+
+# run Make to build the documentation and return to previous directory
+cd "${DOCS_DIR}"
+make clean html
+cd "${cwd}"
+
+# copy HTML output into directory to create an artifact
+mkdir -p artifact/documentation
+cp -R "${DOCS_DIR}/build/html/." artifact/documentation
+
+# check if the warnings.log file is empty
+# Copy it into the artifact and documeentation directories
+# so it will be available in the artifacts
+warning_file="${DOCS_DIR}/build/warnings.log"
+if [[ -s ${warning_file} ]]; then
+  cp -r "${DOCS_DIR}/build/warnings.log" artifact/doc_warnings.log
+  cp artifact/doc_warnings.log artifact/documentation
+  echo "Warnings were encountered while building documentation."
+  echo "========== Begin warnings =========="
+  cat artifact/doc_warnings.log
+  echo "=========== End warnings ==========="
+fi

.github/workflows/docs.yaml

@@ -0,0 +1,51 @@
+name: Build and Deploy Documentation
+on:
+  push:
+    branches:
+      - develop
+      - feature/*
+      - main/*
+      - bugfix/*
+      - release/*
+    paths:
+      - docs/**
+  pull_request:
+    types: [opened, reopened, synchronize]
+
+jobs:
+  documentation:
+    runs-on: ubuntu-latest
+    name: Build and deploy documentation
+
+    steps:
+    - name: Setup Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: "3.9"
+
+    - name: Install (upgrade) python dependencies
+      run: |
+        pip install --upgrade pip sphinx sphinx-gallery sphinx_rtd_theme sphinxcontrib-bibtex
+
+    - name: Checkout
+      uses: actions/checkout@v3
+
+    - name: Build documentation
+      run: |
+        ./.github/scripts/build_docs.sh
+
+    - name: Upload documentation (on success)
+      uses: actions/upload-artifact@v3
+      if: always()
+      with:
+        name: documentation
+        path: artifact/documentation
+
+    - name: Upload warnings (on failure)
+      uses: actions/upload-artifact@v3
+      if: failure()
+      with:
+        name: documentation_warnings.log
+        path: artifact/doc_warnings.log
+        if-no-files-found: ignore
+

.github/workflows/linters.yaml

@@ -0,0 +1,26 @@
+#
+name: linters
+on:
+  pull_request:
+
+permissions:
+  contents: read
+
+defaults:
+  run:
+    shell: bash -o pipefail {0}
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+
+      - name: Lint shell scripts
+        uses: redhat-plumbers-in-action/differential-shellcheck@v3
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/pynorms.yaml

@@ -0,0 +1,24 @@
+name: Check Norms
+on: [push, pull_request]
+
+jobs:
+  check_norms:
+    runs-on: ubuntu-latest
+    name: Check Python coding norms with pycodestyle
+
+    steps:
+
+    - name: Install dependencies
+      run: |
+        pip install --upgrade pip
+        pip install pycodestyle
+
+    - name: Checkout
+      uses: actions/checkout@v3
+      with:
+        path: global-workflow
+
+    - name: Run pycodestyle
+      run: |
+        cd $GITHUB_WORKSPACE/global-workflow
+        pycodestyle -v --config ./.pycodestyle --exclude='.git,.github' ./

.github/workflows/pytests.yaml

@@ -0,0 +1,36 @@
+name: Run pytests
+on: [push, pull_request]
+
+jobs:
+  run_pytests:
+    runs-on: ubuntu-latest
+    name: Install pygw and run tests with pytests
+    strategy:
+      max-parallel: 1
+      matrix:
+        python: ["3.7", "3.8", "3.9", "3.10"]
+
+    steps:
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: Install (upgrade) python dependencies
+        run: |
+          pip install --upgrade pip
+
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          path: global-workflow
+
+      - name: Install pygw
+        run: |
+          cd $GITHUB_WORKSPACE/global-workflow/ush/python/pygw
+          pip install .[dev]
+
+      - name: Run pytests
+        run: |
+          cd $GITHUB_WORKSPACE/global-workflow/ush/python/pygw
+          pytest -v src/tests

.gitignore

@@ -5,9 +5,14 @@ __pycache__
 *.[aox]
 *.mod
 *.sw[a-p]
+._*
 .DS_Store
 .idea/
+.vscode/
 
+# Ignore editor generated backup files
+#-------------------------------------
+*~
 # Ignore folders
 #-------------------
 exec/
@@ -17,8 +22,23 @@ install*/
 # Ignore fix directory symlinks
 #------------------------------
 fix/0readme
-fix/fix_*
-fix/gdas/
+fix/aer
+fix/am
+fix/chem
+fix/cice
+fix/cpl
+fix/datm
+fix/gdas
+fix/gldas
+fix/gsi
+fix/lut
+fix/mom6
+fix/orog
+fix/reg2grb2
+fix/sfc_climo
+fix/ugwd
+fix/verif
+fix/wave
 fix/wafs
 
 # Ignore parm file symlinks
@@ -75,71 +95,12 @@ parm/wafs
 #--------------------------------------------
 sorc/*log
 sorc/logs
-sorc/ufs_model.fd
-sorc/gfs_post.fd
-sorc/gfs_wafs.fd
-sorc/gldas.fd
-sorc/gsi_enkf.fd
-sorc/gsi.fd
-sorc/enkf.fd
-sorc/gdas.cd
-sorc/gsi_utils.fd
-sorc/gsi_monitor.fd
-sorc/ufs_utils.fd
-sorc/verif-global.fd
-
-# Ignore sorc symlinks
-#---------------------
-sorc/calc_analysis.fd
-sorc/calc_increment_ens.fd
-sorc/calc_increment_ens_ncio.fd
-sorc/emcsfc_ice_blend.fd
-sorc/emcsfc_snow2mdl.fd
-sorc/fregrid.fd
-sorc/gdas2gldas.fd
-sorc/getsfcensmeanp.fd
-sorc/getsigensmeanp_smooth.fd
-sorc/getsigensstatp.fd
-sorc/gfs_ncep_post.fd
-sorc/gldas2gdas.fd
-sorc/gldas_forcing.fd
-sorc/gldas_model.fd
-sorc/gldas_post.fd
-sorc/gldas_rst.fd
-sorc/global_chgres.fd
-sorc/global_cycle.fd
-sorc/global_enkf.fd
-sorc/global_gsi.fd
-sorc/interp_inc.fd
-sorc/make_hgrid.fd
-sorc/make_solo_mosaic.fd
-sorc/ncdiag_cat.fd
-sorc/nst_tf_chg.fd
-sorc/oznmon_horiz.fd
-sorc/oznmon_time.fd
-sorc/radmon_angle.fd
-sorc/radmon_bcoef.fd
-sorc/radmon_bcor.fd
-sorc/radmon_time.fd
-sorc/recentersigp.fd
-sorc/upp.fd
-sorc/wafs_awc_wafavn.fd
-sorc/wafs_blending.fd
-sorc/wafs_blending_0p25.fd
-sorc/wafs_cnvgrib2.fd
-sorc/wafs_gcip.fd
-sorc/wafs_grib2_0p25.fd
-sorc/wafs_makewafs.fd
-sorc/wafs_setmissing.fd
+sorc/*.cd
+sorc/*.fd
 
 # Ignore scripts from externals
 #------------------------------
 # jobs symlinks
-jobs/JGDAS_ATMOS_GLDAS
-jobs/JGDAS_ATMOS_VERFOZN
-jobs/JGDAS_ATMOS_VERFRAD
-jobs/JGDAS_ATMOS_VMINMON
-jobs/JGFS_ATMOS_VMINMON
 jobs/JGFS_ATMOS_WAFS
 jobs/JGFS_ATMOS_WAFS_BLENDING
 jobs/JGFS_ATMOS_WAFS_BLENDING_0P25
@@ -148,11 +109,6 @@ jobs/JGFS_ATMOS_WAFS_GRIB2
 jobs/JGFS_ATMOS_WAFS_GRIB2_0P25
 # scripts symlinks
 scripts/exemcsfc_global_sfc_prep.sh
-scripts/exgdas_atmos_gldas.sh
-scripts/exgdas_atmos_verfozn.sh
-scripts/exgdas_atmos_verfrad.sh
-scripts/exgdas_atmos_vminmon.sh
-scripts/exgfs_atmos_vminmon.sh
 scripts/exgfs_atmos_wafs_blending.sh
 scripts/exgfs_atmos_wafs_blending_0p25.sh
 scripts/exgfs_atmos_wafs_gcip.sh
@@ -168,29 +124,17 @@ ush/fv3gfs_driver_grid.sh
 ush/fv3gfs_filter_topo.sh
 ush/fv3gfs_make_grid.sh
 ush/fv3gfs_make_orog.sh
-ush/gldas_archive.sh
-ush/gldas_forcing.sh
-ush/gldas_get_data.sh
-ush/gldas_liscrd.sh
-ush/gldas_post.sh
-ush/gldas_process_data.sh
 ush/global_chgres.sh
 ush/global_chgres_driver.sh
 ush/global_cycle.sh
 ush/global_cycle_driver.sh
-ush/minmon_xtrct_costs.pl
-ush/minmon_xtrct_gnorms.pl
-ush/minmon_xtrct_reduct.pl
 ush/mkwfsgbl.sh
-ush/ozn_xtrct.sh
-ush/radmon_ck_stdout.sh
-ush/radmon_err_rpt.sh
-ush/radmon_verf_angle.sh
-ush/radmon_verf_bcoef.sh
-ush/radmon_verf_bcor.sh
-ush/radmon_verf_time.sh
 ush/ufsda
-ush/rstprod.sh
 ush/wafs_blending.sh
 ush/wafs_grib2.regrid.sh
 ush/wafs_intdsk.sh
+ush/finddate.sh
+ush/make_NTC_file.pl
+ush/make_ntc_bull.pl
+ush/make_tif.sh
+ush/month_name.sh

.pycodestyle

@@ -0,0 +1,6 @@
+[pycodestyle]
+count = False
+ignore = E402,W504
+max-line-length = 160
+statistics = True
+exclude = Experimental

.shellcheckrc

@@ -0,0 +1,13 @@
+# Global settings for Shellcheck (https://github.com/koalaman/shellcheck)
+enable=all
+
+external-sources=false
+
+# Disable variable referenced but not assigned
+disable=SC2154
+
+# Disable following non-constant source
+disable=SC1090
+
+# Disable non-existent binary
+disable=SC1091