Consistent time-stamps in restarts across various ufs-weather-model components.

[aerorahul]

Description

The coupled ufs-weather-model produces restarts for its components at a user chosen frequency. The restart filenames have the following formats:

Component	Intermediate restart	Endpoint restart
FV3	YYYYMMDD.HHMMSS.file_type.tile{n}.nc	file_type.tile{n}.nc
MOM6	MOM.res.YYYY-MM-DD-HH-MM-SS.nc	MOM.res.nc
CICE6	iced.YYYY-MM-DD-SECONDS_FROM_00000.nc	N/A
WW3	YYYYMMDD.HHMMSS.restart.${ww3grid}	N/A
CMEPS	ufs.cpld.cpl.r.YYYY-MM-DD-SECONDS_FROM_00000.nc	N/A

Other components could be added to this table where relevant.

All these files are moved around in their respective application workflows between experiment and run directories.
Having a consistent datetime format makes handling of these files easier and less prone to errors.

This feature requests:

• writing out the restarts across the UFS-weather-model components in a consistent and unified format.
• writing out the restarts at the endpoints in FV3 and MOM6 with a time-stamp.

For more information on the files needed to restart a coupled model, please see this wiki link.

Solution

The restarts would look like the following (assuming the adopted format is from FV3 and WW3):

Component	Intermediate restart	Endpoint restart
FV3	YYYYMMDD.HHMMSS.file_type.tile{n}.nc	YYYYMMDD.HHMMSS.file_type.tile{n}.nc
MOM6	YYYYMMDD.HHMMSS.MOM.res.nc	YYYYMMDD.HHMMSS.MOM.res.nc
CICE6	YYYYMMDD.HHMMSS.cice_model.res.nc	YYYYMMDD.HHMMSS.cice_model.res.nc
WW3	YYYYMMDD.HHMMSS.restart.${ww3grid}	YYYYMMDD.HHMMSS.restart.${ww3grid}
CMEPS	YYYYMMDD.HHMMSS.ufs.cpld.cpl.r.nc	YYYYMMDD.HHMMSS.ufs.cpld.cpl.r.nc

Alternatives

Related to

[DeniseWorthen]

Currently we have three format types for restart files. Most components use one of two types (YYYYMMDD.HHMMSS or YYYY-MM-DD-SSSSS). MOM is the outlier in using neither of these two formats, which does add needless complexity.

The name format for MOM is already w/in a system dependent construct (cesm vs ufs) and is straightforward to change to either of the two existing formats, including writing the final restart w/ a timestamp. This could be an issue in future though, as other coupled systems get implemented using the unified NUOPC cap and want their own naming convention.

For UFS (ie, the mesh cap), WW3 restart file names can be written in either format (the default or using seconds-in-day) using a configuration parameter. When this optional naming is chosen, the presence of the restart file is checked to be present, which should resolve issue #1590. Also, the restart filename in this case includes an application level prefix, eg ufs.hafs.ww3.r, ufs.cpld.ww3.r. It could be extended to include a ww3grid descriptor and location (restart_dir).

CICE has the option to set the restart filename prefix. Currently it is set in the namelist as starting with iced and could easily be changed. CICE already uses a restart_dir, which is set as RESTART.

NG-GODAS applications also set the restart file name using a prefix (set via configuration) and write the file using seconds-in-day for the CDEPS and CMEPS restarts.

For Xgrid applications, the required restart file for CCPP also is written in seconds-in-day format.

I would argue that the preponderance of components we use in UFS are either already consistent w/ the seconds-in-day format or can be switched via configuration settings. Implementing an alternate naming structure w/in the code for all components except the ATM seems to be an unnecessary burden.

If MOM were converted to either of the other two formats, we would have reduced our problem by 1/3, no on-going code maintenance would be required. All components except ATM could use seconds-in-day. The workflow would need to recognize either of the two formats. This makes more sense to me than implementing a code-based UFS-centric naming convention in all these different components.

[junwang-noaa]

So we could have: FV3, WW3, CICE6 with YYYYMMDD.HHMMSS. Now we need to discuss format issue with MOM6, NG-GODAS (CDEPS/CMEPS). My understanding is that it's our first time to use CDEPS/CMEPS restart files in operational, it might be good to use consistent YYYYMMDD.HHMMSS format. Also since we hope to have asynchronized IO (write grid comp) to write both history and restart files for all the components, I'd suggest using the consistent data format for UFS now, rather than change the data format later.

[junwang-noaa]

The code changes for fv3atm to have YYYYMMDD.HHMMSS.file_type.tile{n}.nc are ready at:

https://github.com/junwang-noaa/ufs-weather-model/tree/restart_filename

I tested global fv3 cases. I am going to test other applications.

[DeniseWorthen]

@junwang-noaa No component that I work can write restart names w/ a non-standard format w/o making code changes. The CICE namelist only allows the prefix to be changed, not the timestamp format. All other components will need to have the format specified w/ some sort ifdef option or flag to create a ufs-centric name (this capability is in WW3 only). That means maintaining these special code blocks over time.

[junwang-noaa]

@DeniseWorthen We may need to temporarily maintain the code, once the asynchronous IO is available through the write grid component, we will have a consistent way for file names for UFS on write grid comp without impacting the current file style that are used by other earth systems.

[DeniseWorthen]

@junwang-noaa I must not understand. You want the ATM to write restarts using the write grid component. When that is functional, UFS will be able to chose the filename format for restarts. Then we can switch back to using the default format file name for all the other components with ATM using the same 'seconds-in-day' format?

[junwang-noaa]

Sorry for the confusion, what I mean is that there is no impact to the current filename format when we write data on the write grid comp. We will have one the filename format on the write grid comp, that is what we want to maintain in the future once the write grid comp is available.

[DeniseWorthen]

OK, but that doesn't address the issue of maintaining UFS name styles in all the other components. That will be an on-going maintenance task.

If we accepted having the system using just two file name formats (by switching the odd MOM6 format) we'd still simplify the workflow but not have the code maintenance issue.

[junwang-noaa]

Let's take a look at the MOM6 filename format.

[DeniseWorthen]

I've verified the behavior of the ww3 component.

Using the default restart filename, ww3 will start "cold" if for some reason the ww3 restart is not available (issue #1590).

If we make use of the 'user restart name' configuration variable, ww3 will fail to restart if it does not have the restart available (ie, it fails the "exists" check).

Using the 'user restart name' configuration variable will produce a restart file using 'seconds in day' format, the same as both CICE and CMEPS, for example, ufs.cpld.ww3.r.2021-03-22-64800. For the hafs application, the filename would start with ufs.hafs.ww3.r with a trailing timestamp.