Development of a mesh-based cap for UWM S2SW and HAFSW applications

[DeniseWorthen]

Introduction

A new mesh-based cap compatible with the CMEPS mediator has been developed for use in UFS-weather model applications (S2SW, HAFSW). The new cap contains the following features:

1. WW3 runs on the MOM6 grid for S2SW. It is restart reproducible for C96mx100, C192mx050 and C384mx025 configurations.
2. WW3 has the ability to run as shel or multigrid via configuration namelist. Both options are restart reproducible and are B4B when cross-verified (shel vs a multi baseline and vice versa).
3. WW3 can be placed in the slow coupling loop for S2SW.
4. The allocation of global fields from coupling (e.g. WX0, UX0) in WW3 are limited to water points only (allocated to (nsea,1)) instead of land and sea points (nx,ny). Note that MOM6 water points represent ~65% of the total points.
5. WW3 can be compiled and run in debug mode (using method similar to that described in WW3 issue XX)
6. WW3 can optionally produce NetCDF files containing the gridded fields using configuration settings in nems.configure. Gridded fields written to the netCDF file are:

Mean wind, x-component
Mean wind, y-component
Significant wave height
Peak frequency
Peak direction
Wave height of partitions
Peak wave period of partitions
Skin friction velocity, x-component (HAFS only)
Skin friction velocity, y-component (HAFS only)

Limitations to the above are:

1. UWM is currently set up to run the new mesh cap on Cheyenne only. Extension to other platforms is awaiting the next input data update when the required files can be added.
2. Only a single grid has been tested using the multigrid configuration.
3. The restart file for WW3 itself does not reproduce when run as shel for C192 and C384.
4. Binary output for WW3 (e.g. points, grd) is not B4B when comparing a shel run vs a multigrid run
5. The c384 configuration fails on Cheyenne.intel in multigrid mode with a error message indicating an issue with memory use. Running the existing develop branch but substituting in the mod_def for the mx025 grid results in the same error.
6. The oRTs have not been run.

Requirements

In order to run the mesh cap in UWM, the following files are required:

1. A mesh file describing the model grid.

• For HAFSW, the mesh.hafs.nc file can be produced using NCO (netCDF operators) and the ESMF command line tool ESMF_Scrip2Unstruct. Note the mesh for any rectilinear grid can be created in a similar manner.

ncremap -g hafswav.SCRIP.nc -G latlon=441,901#snwe=1.45,45.55,-98.05,-7.95#lat_typ=uni#lat_drc=s2n
ESMF_Scrip2Unstruct hafswav.SCRIP.nc mesh.hafs.nc 0

The lat/lon points for a HAFS mesh created in this manner are slightly different (O~10-6) compared to those present in the mod_def file for the HAFS application. The following shows the difference in latitude and longitude values across the mid-point of the HAFS domain:

• For S2SW, the existing mesh for MOM6 and CICE6 is also used for WW3. The mesh file is produced using the cpld_gridgen utility. This utility is being added to UFS_UTILS (see tracking issue XX)

2. A mod_def file for WW3.

• For HAFSW, the existing mod_def.natl_6m is used
• For S2SW, the creategridfiles.sh utility is used to create the required files (mod_def.mx100,mod_def.mx050, mod_def.mx025). In each case, the required lat,lon,mask and depth arrays required in the corresponding grid.inp files have been created using the cpld_gridgen utility. The obstruction arrays are currently set to 0.

Next Steps

1. Testing with a true multi-grid configuration is required. The only multi-grid application currently in the UWM regression tests is the control_c384gdas_wav test, which is a high-resource test. This test can be converted to running with the new cap, but it is anticipated that a low resolution multi-grid test case will be required to debug the mesh cap's multigrid capability.

   An initial attempt was made to create multiple grids from the 1-deg MOM6 tripole grid analogous to the 3 grids used in the gdas test. One grid was south of 50S, another between 50S and 50N, and the third domain was the MOM6 polar domain represented as a rectangle, as shown here:

   The WW3 model failed to initialize. Assistance will be required from someone with familiarity with the ww3 grid utility to pursue this configuration.

2. The possible advantages of implementing a multi-grid capability using nested states within the mesh cap can also be explored. This capability is already present in CMEPS and the GLIMMER (Land Ice) Cap. In this case, the GLIMMER cap carries a nested state with two distinct domains (Greenland and Antarctica).

3. In UWM, if the existing tests utilizing the connectors (control_c384gdas_wav, control_atmwav) are retained, then the build system will need be able to build both the new cap and code as well as the existing cap.

4. Initial evaluation and verification of the output using the mesh cap in the S2SW and HAFS applications have been shown previously in Issue XX and XX. Full verification has yet to be done.