Reading and interpolating WOA monthly climatology files for Grid_Stat

[GwenChen-NOAA]

I used the following GRID_STAT_CLIMO_MEAN setup in my METplus 4.0.0 configuration file to read in WOA monthly climatology files, regrid them to match OBS grids, and interpolate them to the VALID date:

GRID_STAT_CLIMO_MEAN_FIELD = {name = "t_an"; level = "(0,0,,)";} --- please note star,star may not show up in the text
GRID_STAT_CLIMO_MEAN_REGRID_METHOD = BILIN
GRID_STAT_CLIMO_MEAN_REGRID_WIDTH = 2
GRID_STAT_CLIMO_MEAN_REGRID_VLD_THRESH = 0.5
GRID_STAT_CLIMO_MEAN_TIME_INTERP_METHOD = DW_MEAN
GRID_STAT_CLIMO_MEAN_DAY_INTERVAL = 31
GRID_STAT_CLIMO_MEAN_HOUR_INTERVAL = 6
GRID_STAT_CLIMO_MEAN_INPUT_DIR = {ENV[CLIMO_PATH]}/WOA18
GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = woa18_decav_t{valid?fmt=%m}_04.nc

When I ran METplus, it pointed to the correct climatology file, but it can't read the WOA file with the following error message in the log file:

ERROR : DictionaryEntry::dict_value() -> bad type

The CLIMO_MEAN dictionary is outputted in the log file as:

metplus INFO: METPLUS_CLIMO_MEAN_DICT=climo_mean = {file_name = ["/lfs/h2/emc/vpppg/noscrub/Lichuan.Chen/Global/climo/WOA18/woa18_decav_t05_04.nc"];field = ["{name="t_an"; level="(0,0,,)";}"];regrid = {method = BILIN;width = 2;vld_thresh = 0.5;}time_interp_method = DW_MEAN;day_interval = 31;hour_interval = 6;}

--- I just noticed that the field didn't show up the same as I posted. There are backslashes before the " in the log text, so it's field = ["{name= backslash"t_an backslash"; level=backslash"(0,0,star,star)backslash";}"];

In which, the syntax for the field looks a bit strange to me. Is this correct? My FCST and OBS are both netCDF files with the following fields:

metplus INFO: METPLUS_FCST_FIELD=field = [{ name="sst"; level="(0,,)"; }]; --- it should be "(0,star,star)"
metplus INFO: METPLUS_OBS_FIELD=field = [{ name="analysed_sst"; level="(0,,)"; convert(x) = K_to_C(x); }];

WOA SST data are 4-dimensional: (time, depth, lat, lon), different from FCST and OBS, which are 3-dimensional: (time, lat, lon). Is it a problem? I can run Grid_Stat and generate CNT stat files using FCST and OBS without CLIMO_MEAN. I also can read WOA climatology files using plot_data_plane with 'name = "t_an"; level = "(0,0,,)";' to create SST maps. Is there anything I missed in setting up CLIMO_MEAN? Thanks for your help!

[georgemccabe]

Hi @GwenChen-NOAA,

You have configured METplus correctly to read the climo mean field. However, enhancements made for METplus 4.1.0 fix the incorrect formatting that you are seeing with the quotation marks in the climo_mean field value. If you are able to upgrade to using version 4.1.0, I recommend making that switch to remedy this issue. If you must continue to use 4.0.0, then you should be able to get around this issue by using the GRID_STAT_MET_CONFIG_OVERRIDES variable instead of GRID_STAT_CLIMO_MEAN_FIELD. You would set something like this:

GRID_STAT_MET_CONFIG_OVERRIDES = climo_mean = { field = [{name = "t_an"; level = "(0,0,*,*)";}];}

This will add this text to the end of the MET configuration file and override the value that way.

On a related note, there is an open GitHub issue #1599 that is slated to be completed in the next development cycle (METplus 5.0.0-beta1 tentatively scheduled for 6/22) that will allow you set the field information this way:

GRID_STAT_CLIMO_MEAN_VAR1_NAME = t_an
GRID_STAT_CLIMO_MEAN_VAR1_LEVELS = "(0,0,*,*)"

The METplus wrapper logic will properly format the values into the correct format that is expected by the MET config file.

Please let me know if you have any other issues or questions. Thanks.

[GwenChen-NOAA]

George,

Thanks for the quick response and suggestions! The GRID_STAT_MET_CONFIG_OVERRIDES works and Grid_Stat is reading the WOA file. However, the time interpolation doesn't work and no anomalies were calculated. I got the following error message in the log file:

DEBUG 4: Data plane information:
DEBUG 4: plane min: -2.08301
DEBUG 4: plane max: 31.2889
DEBUG 4: valid time: 19700101_000000
DEBUG 4: lead time: 000000
DEBUG 4: init time: 19700101_000000
DEBUG 4: accum time: 000000
DEBUG 4:
DEBUG 3: Skipping the 19700101_000000 "t_an(0,0,,)" climatology field since the time offset (10627200 seconds) >= the "day_interval" entry (2678400 seconds) from file: /lfs/h2/emc/vpppg/noscrub/Lichuan.Chen/Global/climo/WOA18/woa18_decav_t05_04.nc
DEBUG 3: Found 0 climatology fields.
DEBUG 3: Found 0 climatology mean and 0 climatology standard deviation field(s) for forecast sst(0,,).

My valid time for FCST is 20220504_000000. Any suggestions how to fix it? Thanks!

[georgemccabe]

Is the file you are trying to read a CF-compliant NetCDF file? It looks like the time information in the file is not being read by MET properly. Could you send me the file? There are instructions to upload the file to our FTP server here: #954

[GwenChen-NOAA]

George,

I have uploaded my data to the ftp server under the chen_data directory. WOA climatology files are CF-compliant NetCDF files, but the time stamp in the files may not be meaningful because they are monthly mean averaged over a period of time. The represented averaging period is denoted in the filename. For example, the woa18_decav_t05_04.nc file is for May averaged over 1981 – 2010 for quarter-degree grid resolution. I also uploaded the WOA documentation in the chen_data directory. The file naming convention is on Page 12. Thanks!

[georgemccabe]

Hi @GwenChen-NOAA, thank you for sending your data and the documentation. It looks like the units for the time variable is "months since 1955-01-01 00:00:00" and while that is technically a valid time unit for CF-compliance, the MET tools do not support it. We often recommend using the MET plot_data_plane tool to check if MET can read new input files properly. Running this command:

plot_data_plane woa18_decav_t05_04.nc out.nc 'name="t_an"; level="(0,0,*,*)";'

outputs this warning:

WARNING: parse_cf_time_string() -> Unsupported time step in the CF convention time unit "months since 1955-01-01 00:00:00"

You can also run plot_data_plane with the "-v 4" argument to raise the logging verbosity. The grid and time information that is parsed from the file is output at the DEBUG 4 level.

Fortunately you can override the attributes that are read parsed by MET using the set_attr_* variables. In this case, you would want to set the valid time using set_attr_valid. You can use the filename template syntax to get the current valid time you are processing:

GRID_STAT_MET_CONFIG_OVERRIDES = climo_mean = { field = [{name = "t_an"; level = "(0,0,*,*)"; set_attr_valid="{valid?fmt=%Y%m%d_%H%M%S}";}];}

Search for "set_attr" on this page to see all of the available attributes that you can override.

Let me know if that works for you.

[JohnHalleyGotway]

@GwenChen-NOAA I also wanted to point out this warning message I see in the output you sent:

WARNING: process_scores() -> Forecast and observation valid times do not match 20220509_000000 != 20220509_120000 for sst(0,*,*) versus analysed_sst(0,*,*).

Are you aware of the discrepancy in the timestamps of the fcst and obs data?

[GwenChen-NOAA]

@JohnHalleyGotway , Thanks for the suggestion. I changed the setting to GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = woa18_decav_t{valid?fmt=%m?shift=-30D}_04.newtime.nc, woa18_decav_t{valid?fmt=%m}_04.newtime.nc, woa18_decav_t{valid?fmt=%m?shift=30D}_04.newtime.nc in my configuration file, but I got fatal error when running METplus. I think METplus cannot recognize 3 files in the template. I uploaded my log file (metplus.log.20220516215705) in the ftp site under chen_data. I also put a RTOFS forecast file, a GHRSST data file, and 3 WOA climatology files in the same directory. Is it possible to specify 3 files in GRID_STAT_CLIMO_MEAN_FILE_NAME? If so, I can set up the valid month and the months before and after as environment variables in my scripts and then pass them into the Grid_Stat configuration file to see if it works.

For the discrepancy in the timestamps of the fcst and obs data, I did see the warning. It is a known issue in the current RTOFS Satellite Verification system. Ideally, we would like the hours matched for both fcst and obs data, but it is not an easy task computationally. Fortunately, ocean variables evolve slowly, so it is safe to assume that the differences between 00Z and 12Z are negligible. In the current system, we simply shift the time of the obs data to 00Z to match the valid time. METplus seems to be tolerant of the discrepancy and still outputs numbers in the CNT stat file despite the warning, so I leave it as is for now. Thanks!

[JohnHalleyGotway]

I suspect I gave you bad advice. Please try changing the capital 'D' to a lowercase one 'd'. I searched the METplus repo and found an example of shifting by a number of days, like this {valid?fmt=%Y%m%d_%H?shift=1d} with a lowercase 'd'.

So please try using this GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = woa18_decav_t{valid?fmt=%m?shift=-30d}_04.newtime.nc, woa18_decav_t{valid?fmt=%m}_04.newtime.nc, woa18_decav_t{valid?fmt=%m?shift=30d}_04.newtime.nc

If that doesn't solve it, I'll ask @georgemccabe to weigh in.

[GwenChen-NOAA]

@JohnHalleyGotway , Good news! This one works! METplus is able to grab the correct monthly files and interpolate to the valid date. The log file has the following outputs:

DEBUG 2: Regridding the 19860615_000000 "t_an(0,0,,)" climatology field to the verification grid.
DEBUG 3: For forecast valid at 20220517_000000, found 2 climatology field(s) with valid time(s): 20220515_000000, 20220615_000000
DEBUG 3: Interpolating climatology fields at 20220515_000000 and 20220615_000000 to 20220517_000000 using the DW_MEAN interpolation method.
DEBUG 3: Found 1 climatology fields.
DEBUG 3: Found 1 climatology mean and 0 climatology standard deviation field(s) for forecast sst(0,,).
DEBUG 2: Processing masking regions.

For other people's reference, below is my climo_mean setup in the Grid_Stat configuration file for METplus 4.0.0:

GRID_STAT_MET_CONFIG_OVERRIDES = climo_mean = {field = [{name="t_an"; level="(0,0,,)";}];}
GRID_STAT_CLIMO_MEAN_REGRID_METHOD = BILIN
GRID_STAT_CLIMO_MEAN_REGRID_WIDTH = 2
GRID_STAT_CLIMO_MEAN_REGRID_VLD_THRESH = 0.5
GRID_STAT_CLIMO_MEAN_REGRID_SHAPE = SQUARE
GRID_STAT_CLIMO_MEAN_TIME_INTERP_METHOD = DW_MEAN
GRID_STAT_CLIMO_MEAN_DAY_INTERVAL = 31
GRID_STAT_CLIMO_MEAN_HOUR_INTERVAL = 6
GRID_STAT_CLIMO_MEAN_INPUT_DIR = {ENV[CLIMO_PATH]}/WOA18_newtime
GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = woa18_decav_t{valid?fmt=%m?shift=-31d}_04.newtime.nc, woa18_decav_t{valid?fmt=%m}_04.newtime.nc, woa18_decav_t{valid?fmt=%m?shift=28d}_04.newtime.nc

Please note that I changed the shift days in the template for the month before from 30 days to 31days, so for valid date on the 31th (e.g., 20220531) it will grab the previous month (e.g., April), and for the month after to 28 days, so for valid date on Jan 31 it will grab February. This setup requires adjustment of the time unit from months to days in the WOA18 climatology files.

Thank you so much for all your help! I'm very happy we work this out. Thanks!

[JohnHalleyGotway]

That's great news! Thanks for confirming!

Can you please select one of the responses in this thread as a good answer and click the "Mark As Answer" button? Then I'll lock discussion to prevent future posts.

Thanks