Feature: UFS Replay

.github/workflows/ci_run_scm_DEPHY.yml

@@ -1,6 +1,6 @@
 name: CI test to run the SCM with DEPHY v1 data
 
-on: [push, pull_request]
+on: [pull_request,workflow_dispatch]
 
 jobs:
   run-scm-DEPHY:

.github/workflows/ci_run_scm_ufs_replay.yml

@@ -0,0 +1,75 @@
+name: CI test to create SCM UFS-replay cases from UWM regression tests
+
+on: [pull_request,workflow_dispatch]
+
+jobs:
+  run_scm_ufs_replay:
+
+    # The type of runner that the job will run on
+    runs-on: ubuntu-22.04
+    defaults:
+      run:
+        shell: bash -el {0}
+
+    # Environmental variables
+    env:
+      dir_rt_cache: /home/runner/ufs_rts
+      SCM_ROOT:     ${{ github.workspace }}
+
+    steps:
+
+    #######################################################################################
+    # Checkout SCM code, setup python.
+    #######################################################################################
+
+    - name: Checkout SCM.
+      uses: actions/checkout@v3
+
+    - name: Initialize SCM submodules.
+      run: git submodule update --init --recursive
+
+    - name: Update system packages.
+      run: sudo apt-get update
+
+    - name: Cache conda
+      uses: actions/cache@v3
+      with:
+        path: ~/conda_pkgs_dir
+        key: conda-pkgs
+
+    - name: Setup python.
+      uses: conda-incubator/setup-miniconda@v2
+      with:
+        activate-environment: env_ufsreplay
+        environment-file: environment-ufsreplay.yml
+        use-only-tar-bz2: true
+        auto-activate-base: true
+        auto-update-conda: true
+
+    #######################################################################################
+    # Create UFS-replay case for SCM using UWM Regression Tests
+    #######################################################################################
+
+    - name: Cache UWM regression test output.
+      uses: actions/cache@v3
+      with:
+        path: ${dir_rt_cache}
+        key: ufs-rt-files
+
+    - name: Download UWM regression test output from NCAR-DTC FTP site, if not cached.
+      run: |
+        if test ! -d "${dir_rt_cache}"; then
+          mkdir -p ${dir_rt_cache} && cd ${dir_rt_cache}
+          wget -q ftp://ftp.rap.ucar.edu:/pub/ccpp-scm/ufs_rts_scmreplay_ci.tar
+          tar -xvf ufs_rts_scmreplay_ci.tar
+          ls ${dir_rt_cache}
+        fi
+
+    - name: Create UFS-replay case.
+      run: |
+        cd ${SCM_ROOT}/scm/etc/scripts/
+        ./UFS_forcing_ensemble_generator.py -d ${dir_rt_cache}/ --C_RES 192 -dt 360  -n control_c192 -lons 300 -lats 34 -sc
+
+    #######################################################################################
+    # Done
+    #######################################################################################

.github/workflows/ci_scm_ccpp_prebuild.yml

@@ -1,6 +1,6 @@
 name: CI test to run SCM ccpp_prebuild script
 
-on: [push, pull_request]
+on: [push, pull_request, workflow_dispatch]
 
 jobs:
   build-linux:

environment-ufsreplay.yml

@@ -0,0 +1,10 @@
+name: env_ufsreplay
+
+dependencies:
+  - conda-forge::python=3.8.5
+  - conda-forge::netcdf4
+  - conda-forge::f90nml
+  - conda-forge::xarray
+  - conda-forge::numpy
+  - conda-forge::shapely
+  - conda-forge::xesmf

environment.yml

@@ -0,0 +1,6 @@
+name: scm_py37
+
+dependencies:
+  - conda-forge::python=3.7
+  - conda-forge::netcdf4
+  - conda-forge::f90nml

scm/doc/TechGuide/chap_cases.tex

@@ -132,7 +132,7 @@ \section{Included Cases}
 \item UFS initial conditions for 38.1 N, 98.5 W (central Kansas) for 00Z on Oct. 3, 2016 with Noah variables on the C96 FV3 grid (\execout{fv3\_model\_point\_noah.nc})
 \item UFS initial conditions for 38.1 N, 98.5 W (central Kansas) for 00Z on Oct. 3, 2016 with NoahMP variables on the C96 FV3 grid (\execout{fv3\_model\_point\_noahmp.nc})
 \end{itemize}
-See \ref{sec:UFS ICs} for information on how to generate these files for other locations and dates, given appropriate UFS Atmosphere initial conditions.
+See \ref{sec:UFSreplay} for information on how to generate these files for other locations and dates, given appropriate UFS Atmosphere initial conditions.
 
 \section{How to set up new cases}
 
@@ -196,15 +196,36 @@ \section{Using other LASSO cases}
 \item Create a new case configuration file (or copy and modify an existing one) in \execout{ccpp-scm/scm/etc/case\_config}. Be sure that the \execout{case\_name} variable points to the newly created/processed case input file from above.
 \end{enumerate}
 
-\section{Using UFS Initial Conditions}
-\label{sec:UFS ICs}
+\section{Using UFS Output to Create SCM Cases: UFS-Replay}
+\label{sec:UFSreplay}
+
+
+\subsection{Python Dependencies}
+\label{subsection: pydepend}
+The scripts here require a few python packages that may not be found by default in all python installations. There is a YAML file with the python environment needed to run the script in \execout{ccpp-scm/environment-ufsreplay.yml}. To create and activate this environment using conda:
+
+Create environment (only once):
+
+\execout{> conda env create -f environment-ufsreplay.yml}
+
+This will create the conda environment \execout{env\_ufsreplay}
+
+
+Activate environment:
+
+\execout{> conda activate env\_ufsreplay}
+
+
+\subsection{UFS\_IC\_generator.py}
+\label{subsection: ufsicgenerator}
+A script exists in \execout{scm/etc/scripts/UFS\_IC\_generator.py} to read in UFS history (output) files and their initial conditions to generate a SCM case input data file, in DEPHY format. 
 
-A script exists in \execout{scm/etc/scripts/UFS\_IC\_generator.py} to read in UFS Atmosphere cold start initial conditions and generate a case input data file that the SCM can use. Note that the script requires a few python packages that may not be found by default in all python installations: \exec{argparse}, \exec{fnmatch}, \exec{logging}, \exec{NetCDF4}, \exec{numpy}, \exec{shapely}, \exec{f90nml}, and \exec{re}. 
 
 \begin{lstlisting}[language=bash]
 ./UFS_IC_generator.py [-h] (-l LOCATION LOCATION | -ij INDEX INDEX) -d
-DATE -i IN_DIR -g GRID_DIR [-t {1,2,3,4,5,6}]
-[-a AREA] -n CASE_NAME [-oc]
+DATE -i IN_DIR -g GRID_DIR -f FORCING_DIR -n
+CASE_NAME [-t {1,2,3,4,5,6,7}] [-a AREA] [-oc]
+[-lam] [-sc] [-near]
 \end{lstlisting}
 
 Mandatory arguments:
@@ -214,29 +235,130 @@ \section{Using UFS Initial Conditions}
 		\item -l 261.51 38.2 (two floating point values separated by a space)
 		\item -ij 8 49 (two integer values separated by a space; this option must also use the \exec{-{}-tile (-t)} argument to specify the tile number)
 	\end{itemize}
-\item \exec{-{}-date (-d)} YYYYMMDDHHMM: date corresponding to the UFS initial conditions
-\item \exec{-{}-in\_dir (-i)}: path to the directory containing UFS initial conditions
+\item \exec{-{}-date (-d)} YYYYMMDDHHMMSS: date corresponding to the UFS initial conditions
+\item \exec{-{}-in\_dir (-i)}: path to the directory containing the UFS initial conditions
 \item \exec{-{}-grid\_dir (-g)}: path to the directory containing the UFS supergrid files (AKA "fix" directory)
-\item \exec{-{}-case\_name (-n)}: what to call the output NetCDF file
+\item \exec{-{}-forcing\_dir (-f)}: path to the directory containing the UFS history files
+\item \exec{-{}-case\_name (-n)}: name of case
 \end{enumerate}
 
 Optional arguments:
 \begin{enumerate}
 \item \exec{-{}-tile (-t)}: if one already knows the correct tile for the given longitude and latitude OR one is specifying the UFS grid index (\exec{-{}-index} argument)
 \item \exec{-{}-area (-a)}: area of grid cell in $m^2$ (if known or different than the value calculated from the supergrid file)
 \item \exec{-{}-old\_chgres (-oc)}: flag if UFS initial conditions were generated using older version of chgres (global\_chgres); might be the case for pre-2018 data
+\item \exec{-{}-lam (-lam)}: flag to signal that the ICs and forcing is from a limited-area model run
+\item \exec{-{}-save\_comp (-sc)}: flag to create UFS reference file for comparison
+\item \exec{-{}-use\_nearest (-near)}: flag to indicate using the nearest UFS history file gridpoint
+\end{enumerate}
+
+\subsection{UFS\_forcing\_ensemble\_generator.py}
+\label{subsection: ufsforcingensemblegenerator}
+There is an additional script in \execout{scm/etc/scripts/UFS\_forcing\_ensemble\_generator.py} to create UFS-replay case(s) starting with output from UFS Weather Model (UWM) Regression Tests (RTs).
+
+\begin{lstlisting}[language=bash]
+UFS_forcing_ensemble_generator.py [-h] -d DIR -n CASE_NAME
+(-lonl LON_1 LON_2 -latl LAT_1 LAT_2 -nens NENSMEMBERS |
+-lons [LON_LIST] -lats [LAT_LIST])
+[-dt TIMESTEP] [-cres C_RES] [-sdf SUITE] [-sc] [-near]
+\end{lstlisting}
+
+Mandatory arguments:
+\begin{enumerate}
+\item \exec{-{}-dir (-d)}: path to UFS Regression Test output
+\item \exec{-{}-case\_name (-n)}: name of cases
+\item Either: (see examples bewlow) 
+      \begin{itemize}
+           \item \exec{-{}-lon\_limits (-lonl)} AND \exec{-{}-lat\_limits (-latl)} AND \exec{-{}-nensmembers (-nens)}: longitude range, latitude range, and number of cases to create
+           \item \exec{-{}-lon\_list (-lons)} AND \exec{-{}-lat\_list (-lats)}: longitude and latitude of cases
+      \end{itemize}
 \end{enumerate}
 
-The following commands were used from within the \exec{scm/etc/scripts} directory to generate the example UFS Atmosphere initial condition case input file:
+Optional arguments:
+\begin{enumerate}
+\item \exec{-{}-timestep (-dt)}: SCM timestep, in seconds
+\item \exec{-{}-C\_res (-cres)}: UFS spatial resolution
+\item \exec{-{}-suite (-sdf)}: CCPP suite definition file to use for ensemble
+\item \exec{-{}-save\_comp (-sc)}: flag to create UFS reference file for comparison
+\item \exec{-{}-use\_nearest (-near)}: flag to indicate using the nearest UFS history file gridpoint
+\end{enumerate}
+
+Examples to run from within the \exec{scm/etc/scripts} directory to create SCM cases starting with the output from a UFS Weather Model regression test(s):
+
+On the supported platforms Cheyenne (NCAR) and Hera (NOAA), there are staged UWM RTs located at:
+\begin{itemize}
+\item \execout{Cheyenne /glade/scratch/epicufsrt/GMTB/CCPP-SCM/UFS\_RTs}
+\item \execout{Hera /scratch1/BMC/gmtb/CCPP-SCM/UFS\_RTs}
+\end{itemize}
+
+\subsection{Example 1: UFS-replay for single point}
+\label{subsection: example1}
+
+UFS regression test, \execout{control\_c192}, for single point.
+\begin{lstlisting}[language=bash]
+./UFS_forcing_ensemble_generator.py -d /glade/scratch/epicufsrt/GMTB/CCPP-SCM/UFS_RTs/control_c192/ -sc --C_RES 192 -dt 360  -n control_c192 -lons 300 -lats 34
+\end{lstlisting}
+
+Upon successful completion of the script, the command to run the case(s) will print to the screen. For example,
+
+\begin{lstlisting}[language=bash]
+./run_scm.py --npz_type gfs --file scm_ufsens_control_c192.py --timestep 360
+\end{lstlisting}
+
+The file \exec{scm\_ufsens\_control\_c192.py} is created in \exec{ccpp-scm/scm/bin/}, where the SCM run script is to be exectued.
+
+\subsection{Example 2: UFS-replay for list of points}
+\label{subsection: example2}
+
+UFS regression test, \execout{control\_c384}, for multiple points.
+\begin{lstlisting}[language=bash]
+./UFS_forcing_ensemble_generator.py -d /glade/scratch/epicufsrt/GMTB/CCPP-SCM/UFS_RTs/control_c384/ -sc --C_RES 384 -dt 225 -n control_c384 -lons 300 300 300 300 -lats 34 35 35 37
+\end{lstlisting}
+
+Upon successful completion of the script, the command to run the case(s) will print to the screen. For example,
+
+\begin{lstlisting}[language=bash]
+./run_scm.py --npz_type gfs --file scm_ufsens_control_c384.py --timestep 225
+\end{lstlisting}
+
+The file \exec{scm\_ufsens\_control\_c384.py} contains \exec{ALL} of the cases created. Each case created will have the naming convention \exec{case\_name\_nXXX}, where the suffix \exec{XXX} is the case number from 0 to the number of points provided. The contents of the file should look like:
 \begin{lstlisting}[language=bash]
-./UFS_IC_generator.py -l 261.51 38.2 -d 201610030000 -i ../../data/raw_case_input/FV3_C96_example_ICs -g ../../data/raw_case_input/FV3_C96_example_ICs -n fv3_model_point_noah -oc
+run_list = [{"case": "control_c384_n000", "suite": "SCM_GFS_v16"},
+            {"case": "control_c384_n001", "suite": "SCM_GFS_v16"},
+            {"case": "control_c384_n002", "suite": "SCM_GFS_v16"},
+            {"case": "control_c384_n003", "suite": "SCM_GFS_v16"}]
 \end{lstlisting}
 
-Note that the \exec{-{}-in\_dir (-i)} and \exec{-{}-grid\_dir (-g)} arguments are the same in this case (since the supergrid files were copied to the same directory as the initial conditions files for point of example), but they will not in general be the same. Also note that the default behavior of the script is to expect that the NetCDF initial condition files were generated from \execout{chgres\_cube} and not the older \execout{global\_chgres}. If they were generated from the older version (which is likely for pre-2018 data), they will have a slightly different format requiring the \exec{-{}-old\_chgres (-oc)} option to be set in order for the files to be read properly by the script. If you try without the \exec{-{}-old\_chgres (-oc)} flag and receive a ``IndexError: t not found'' error, try the script again with the flag.
+\subsection{Example 3: UFS-replay for an ensemble of points}
+\label{subsection: example3}
+
+UFS regression test, \execout{control\_p8}, for an ensemble (10) of randomly selected points over a specified longitude ($300-320^oW$) and latitude ($40-50^oN$) range
+
+But first, to use the \execout{control\_p8} test we need to rerun the regression test to generate UFS history files with a denser and constant output interval. First, in \execout{control\_p8/model\_configure}, change \exec{-{}-output\_fh} to \execout{"interval -1"}, where \execout{interval} is the UFS history file output frequency (in hours), see \href{https://ufs-weather-model.readthedocs.io/en/latest/InputsOutputs.html}{UFS Weather Model Users Guide} for more details.
+
+For the purposes of this example the \execout{control\_p8} test has already been rerun, but if starting from your own UWM RTs, you can rerun the UWM regression test, on Cheyenne for example, by running the following command in the RT directory: \execout{qsub job\_card}
 
-In addition to the case input files generated by this script, one will need appropriate case configuration files. Make sure that the \exec{model\_ics} variable is set to \exec{.true.} and that the \exec{C\_RES}, \exec{year}, \exec{month}, \exec{day}, and \exec{hour} are all set to the appropriate values that match the UFS Atmosphere initial conditions used. See \execout{scm/etc/case\_config/fv3\_model\_point\_noah.nml} for an example.
+Now the cases can be generated with the following command:
+\begin{lstlisting}[language=bash]
+./UFS_forcing_ensemble_generator.py -d /glade/scratch/epicufsrt/GMTB/CCPP-SCM/UFS_RTs/control_p8/ -sc --C_RES 96 -dt 720 -n control_p8 -lonl 300 320 -latl 40 50 -nens 10 -sdf SCM_GFS_v17_p8
+\end{lstlisting}
+
+Upon successful completion of the script, the command to run the case(s) will print to the screen. For example,
+
+\begin{lstlisting}[language=bash]
+./run_scm.py --npz_type gfs --file scm_ufsens_control_p8.py --timestep 720
+\end{lstlisting}
 
-Running the model is the same as for observational field campaign cases:
+The file \exec{scm\_ufsens\_control\_p8.py} contains ten cases (n000-n009) to be run. The contents of the file should look like:
 \begin{lstlisting}[language=bash]
-./run_scm.py -c fv3_model_point_noah -s SCM_GFS_v16
+run_list = [{"case": "control_p8_n000", "suite": "SCM_GFS_v17_p8"},
+            {"case": "control_p8_n001", "suite": "SCM_GFS_v17_p8"},
+            {"case": "control_p8_n002", "suite": "SCM_GFS_v17_p8"},
+            {"case": "control_p8_n003", "suite": "SCM_GFS_v17_p8"},
+            {"case": "control_p8_n004", "suite": "SCM_GFS_v17_p8"},
+            {"case": "control_p8_n005", "suite": "SCM_GFS_v17_p8"},
+            {"case": "control_p8_n006", "suite": "SCM_GFS_v17_p8"},
+            {"case": "control_p8_n007", "suite": "SCM_GFS_v17_p8"},
+            {"case": "control_p8_n008", "suite": "SCM_GFS_v17_p8"},
+            {"case": "control_p8_n009", "suite": "SCM_GFS_v17_p8"}]
 \end{lstlisting}