SeriesAnalysis - Could not find data

[haksulee]

Hello

Could you check what I did wrong in the configuration copied below ?

SeriesAnalysis.gefsv12.conf and related files are available at https://drive.google.com/drive/folders/1XjYk0BrfyAiIj8NtddtMIt-2PZOSHzYu

I am trying to compute statistics using SeriesAnalysis with the GEFSv12 daily precip ensemble mean grid (=output from GenEnsProd) and the AORC data.

In the example file used in the configuration below, VALID time should be 2018010500 = INIT_BEG 2018010400 + 24 LEAD HOUR
However, the error message indicates 20180101 and I have no idea where 20180101 is from (??)

ERROR : get_series_entry() -> Could not find data for tp(20180101_000000,,) in file list:

ncdump -h gen_ens_prod_20180104I_20180105V_ens.nc showed 2018010500_000000, not 20180101_000000

I tried to plot the map using plot_data_plane but failed:
plot_data_plane gen_ens_prod_20180104I_20180105V_ens.nc testmem.ps 'name="tp"; level="(20180105_000000,,)";' -v 5
ERROR : plot_data_plane -> trouble getting field "name="tp"; level="(20180105_000000,,)";" from file "gen_ens_prod_20180104I_20180105V_ens.nc"

plot_data_plane gen_ens_prod_20180104I_20180105V_ens.nc testmem.ps 'name="tp"; level="(0,,)";'
ERROR : plot_data_plane -> trouble getting field "name="tp"; level="(0,,)";" from file "gen_ens_prod_20180104I_20180105V_ens.nc"

plot_data_plane gen_ens_prod_20180104I_20180105V_ens.nc testmem.ps 'name="tp"; level="(20180101_000000,,)";' -v 5
plot_data_plane -> trouble getting field "name="tp"; level="(20180101_000000,,)";" from file "gen_ens_prod_20180104I_20180105V_ens.nc"

Thank you
Haksu

[config]
PROCESS_LIST = SeriesAnalysis

#SERIES_ANALYSIS_RUNTIME_FREQ = RUN_ONCE_PER_INIT_OR_VALID
SERIES_ANALYSIS_RUNTIME_FREQ = RUN_ONCE_PER_LEAD

LOOP_BY = INIT
INIT_TIME_FMT = %Y%m%d%H
#INIT_BEG=2005080700
#INIT_END=2005080700
#INIT_INCREMENT = 12H

INIT_BEG=2018010400
INIT_END=2018010400
#INIT_INCREMENT=604800

#INIT_BEG=2017100500
#INIT_END=2019092600
INIT_INCREMENT=604800
#604800 = 7 day *24 h * 3600 min

#SERIES_ANALYSIS_CUSTOM_LOOP_LIST = c00, p01, p02, p03, p04

#LEAD_SEQ = 12, 9, 6
#LEAD_SEQ = begin_end_incr(6,120,6)
LEAD_SEQ = begin_end_incr(24,120,24)
#LEAD_SEQ = 24

SERIES_ANALYSIS_RUN_ONCE_PER_STORM_ID = False

SERIES_ANALYSIS_CUSTOM_LOOP_LIST =

File I/O

#FCST_SERIES_ANALYSIS_INPUT_DIR = {INPUT_BASE}/met_test/data/sample_fcst
#FCST_SERIES_ANALYSIS_INPUT_DIR = {INPUT_BASE}/gefsv12
#FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = {init?fmt=%Y%m%d%H}/wrfprs_ruc13_{lead?fmt=%2H}.tm00_G212
#FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = {init?fmt=%Y%m%d}/p25/ge???.t00z.pgrb2.0p25.f{lead?fmt=%3H}
#FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = {init?fmt=%Y%m%d}/p25/ge{custom?fmt=%s}.t00z.pgrb2.0p25.f{lead?fmt=%3H}

#FCST_SERIES_ANALYSIS_INPUT_DIR = {INPUT_BASE}/aggregated_precip_v12_1-5day
#FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = {init?fmt=%Y%m%d}/precip_{init?fmt=%Y%m%d}member??day1_5.nc
FCST_SERIES_ANALYSIS_INPUT_DIR = {INPUT_BASE}/aggregated_precip_v12_1-5day_ens_mean
FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = gen_ens_prod{init?fmt=%Y%m%d}I{valid?fmt=%Y%m%d}V_ens.nc
#FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = gen_ens_prod_{init?fmt=%Y%m%d}I_{lead?fmt=%3H}L_{valid?fmt=%Y%m%d}V_ens.nc

#gec00.t00z.pgrb2.0p25.f006
#../../EnsembleStat/gefsv13EP5r2/input/gefsv12/20180104/p25/gep01.t00z.pgrb2.0p25.f006
#../../EnsembleStat/gefsv13EP5r2/input/gefsv12/20180104/p25/gep02.t00z.pgrb2.0p25.f006
#../../EnsembleStat/gefsv13EP5r2/input/gefsv12/20180104/p25/gep03.t00z.pgrb2.0p25.f006
#../../EnsembleStat/gefsv13EP5r2/input/gefsv12/20180104/p25/gep04.t00z.pgrb2.0p25.f006

#OBS_SERIES_ANALYSIS_INPUT_DIR = {INPUT_BASE}/met_test/new
#OBS_SERIES_ANALYSIS_INPUT_TEMPLATE = ST2ml{valid?fmt=%Y%m%d%H}_A03h.nc

#OBS_SERIES_ANALYSIS_INPUT_DIR = {INPUT_BASE}/aorc_regridded/{valid?fmt=%Y%m}
#OBS_SERIES_ANALYSIS_INPUT_TEMPLATE = AORC-OWP_{valid?fmt=%Y%m%d%H}z.nc4

OBS_SERIES_ANALYSIS_INPUT_DIR = {INPUT_BASE}/aggregated_precip_AORC_1day/{valid?fmt=%Y%m}
OBS_SERIES_ANALYSIS_INPUT_TEMPLATE = precip_{valid?fmt=%Y%m}_1day.nc

SERIES_ANALYSIS_TC_STAT_INPUT_DIR =
SERIES_ANALYSIS_TC_STAT_INPUT_TEMPLATE =

SERIES_ANALYSIS_OUTPUT_DIR = {OUTPUT_BASE}/SeriesAnalysis
#SERIES_ANALYSIS_OUTPUT_TEMPLATE = {init?fmt=%Y%m%d%H}sa.nc
SERIES_ANALYSIS_OUTPUT_TEMPLATE = {init?fmt=%Y%m%d%H}I{lead?fmt=%3H}L_{valid?fmt=%Y%m%d}V_sa.nc

SERIES_ANALYSIS_CLIMO_MEAN_INPUT_DIR =
SERIES_ANALYSIS_CLIMO_MEAN_INPUT_TEMPLATE =

SERIES_ANALYSIS_CLIMO_STDEV_INPUT_DIR =
SERIES_ANALYSIS_CLIMO_STDEV_INPUT_TEMPLATE =

Field Info

#FCST_CAT_THRESH =
#OBS_CAT_THRESH =

#FCST_VAR1_NAME = APCP
#FCST_VAR1_LEVELS = A03
#FCST_VAR1_LEVELS = Z0

FCST_VAR1_NAME = tp
FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},,)"
#FCST_VAR1_LEVELS = "(0,,)"

#SERIES_ANALYSIS_MET_CONFIG_OVERRIDES = fcst = {file_type = NETCDF_NCCF;}

#OBS_VAR1_NAME = APCP_03
#OBS_VAR1_LEVELS = "(,)"

OBS_VAR1_NAME = APCP_surface
#OBS_VAR1_LEVELS = (,)
OBS_VAR1_LEVELS = "(0,,)"

BOTH_VAR1_THRESH = gt12.7, gt25.4, gt50.8, gt76.2

SeriesAnalysis Settings

#MODEL = WRF
MODEL = GEFS
OBTYPE = MC_PCP

LOG_SERIES_ANALYSIS_VERBOSITY = 5

SERIES_ANALYSIS_CONFIG_FILE = {PARM_BASE}/met_config/SeriesAnalysisConfig_wrapped

SERIES_ANALYSIS_IS_PAIRED = False

#SERIES_ANALYSIS_DESC =

#SERIES_ANALYSIS_CAT_THRESH =

#SERIES_ANALYSIS_VLD_THRESH =

#SERIES_ANALYSIS_BLOCK_SIZE =

#SERIES_ANALYSIS_REGRID_TO_GRID =
#SERIES_ANALYSIS_REGRID_METHOD =
#SERIES_ANALYSIS_REGRID_WIDTH =
#SERIES_ANALYSIS_REGRID_VLD_THRESH =
#SERIES_ANALYSIS_REGRID_SHAPE =
#SERIES_ANALYSIS_REGRID_CONVERT =
#SERIES_ANALYSIS_REGRID_CENSOR_THRESH =
#SERIES_ANALYSIS_REGRID_CENSOR_VAL =

#SERIES_ANALYSIS_CLIMO_MEAN_FILE_NAME =
#SERIES_ANALYSIS_CLIMO_MEAN_FIELD =
#SERIES_ANALYSIS_CLIMO_MEAN_REGRID_METHOD =
#SERIES_ANALYSIS_CLIMO_MEAN_REGRID_WIDTH =
#SERIES_ANALYSIS_CLIMO_MEAN_REGRID_VLD_THRESH =
#SERIES_ANALYSIS_CLIMO_MEAN_REGRID_SHAPE =
#SERIES_ANALYSIS_CLIMO_MEAN_TIME_INTERP_METHOD =
#SERIES_ANALYSIS_CLIMO_MEAN_MATCH_MONTH =
#SERIES_ANALYSIS_CLIMO_MEAN_DAY_INTERVAL =
#SERIES_ANALYSIS_CLIMO_MEAN_HOUR_INTERVAL =
#SERIES_ANALYSIS_CLIMO_MEAN_FILE_TYPE =

#SERIES_ANALYSIS_CLIMO_STDEV_FILE_NAME =
#SERIES_ANALYSIS_CLIMO_STDEV_FIELD =
#SERIES_ANALYSIS_CLIMO_STDEV_REGRID_METHOD =
#SERIES_ANALYSIS_CLIMO_STDEV_REGRID_WIDTH =
#SERIES_ANALYSIS_CLIMO_STDEV_REGRID_VLD_THRESH =
#SERIES_ANALYSIS_CLIMO_STDEV_REGRID_SHAPE =
#SERIES_ANALYSIS_CLIMO_STDEV_TIME_INTERP_METHOD =
#SERIES_ANALYSIS_CLIMO_STDEV_MATCH_MONTH =
#SERIES_ANALYSIS_CLIMO_STDEV_DAY_INTERVAL =
#SERIES_ANALYSIS_CLIMO_STDEV_HOUR_INTERVAL =
#SERIES_ANALYSIS_CLIMO_STDEV_FILE_TYPE =

#SERIES_ANALYSIS_CLIMO_CDF_BINS =
#SERIES_ANALYSIS_CLIMO_CDF_CENTER_BINS =
#SERIES_ANALYSIS_CLIMO_CDF_DIRECT_PROB =

#SERIES_ANALYSIS_HSS_EC_VALUE =

#FCST_SERIES_ANALYSIS_PROB_THRESH =

#SERIES_ANALYSIS_OUTPUT_STATS_FHO =
#SERIES_ANALYSIS_OUTPUT_STATS_CTC =
#SERIES_ANALYSIS_OUTPUT_STATS_CTS = BASER, CSI, GSS
#SERIES_ANALYSIS_OUTPUT_STATS_MCTC =
#SERIES_ANALYSIS_OUTPUT_STATS_MCTS =

#SERIES_ANALYSIS_OUTPUT_STATS_CNT = TOTAL, RMSE, FBAR, OBAR, PR_CORR
SERIES_ANALYSIS_OUTPUT_STATS_CNT = TOTAL, RMSE, FBAR, OBAR, PR_CORR, ME, MAE, ANOM_CORR, MSE

#SERIES_ANALYSIS_OUTPUT_STATS_SL1L2 =
#SERIES_ANALYSIS_OUTPUT_STATS_SAL1L2 =
#SERIES_ANALYSIS_OUTPUT_STATS_PCT =
#SERIES_ANALYSIS_OUTPUT_STATS_PSTD =
#SERIES_ANALYSIS_OUTPUT_STATS_PJC =
#SERIES_ANALYSIS_OUTPUT_STATS_PRC =

#SERIES_ANALYSIS_MASK_GRID =
#SERIES_ANALYSIS_MASK_POLY =

SeriesAnalysis Plotting

SERIES_ANALYSIS_GENERATE_PLOTS = no

PLOT_DATA_PLANE_TITLE =

SERIES_ANALYSIS_GENERATE_ANIMATIONS = no

[haksulee]

I am trying to compute gridded statistics including MAE, MSE, correlation, etc for each lead time separately (day1, day2, ..., day5) by pairing GEFSv12 ensemble mean grid and AORC daily precip observation.

[CPKalb]

Hi:

The output files you are using from Gen-Ens-Prod do not have a time dimension for the variables. What I mean by this is that the netCDF header shows:

dimensions:
lat = 241 ;
lon = 441 ;

And for the ensemble mean variable:
float tp_20180105_000000_all_all_ENS_MEAN(lat, lon) ;
tp_20180105_000000_all_all_ENS_MEAN:coordinates = "lat lon" ;
tp_20180105_000000_all_all_ENS_MEAN:name = "tp" ;
tp_20180105_000000_all_all_ENS_MEAN:long_name = "tp at 20180105_000000,, Ensemble Mean" ;
tp_20180105_000000_all_all_ENS_MEAN:level = "20180105_000000,," ;
tp_20180105_000000_all_all_ENS_MEAN:units = "kg m**-2" ;
tp_20180105_000000_all_all_ENS_MEAN:_FillValue = -9999.f ;
tp_20180105_000000_all_all_ENS_MEAN:init_time = "20180105_000000" ;
tp_20180105_000000_all_all_ENS_MEAN:init_time_ut = "1515110400" ;
tp_20180105_000000_all_all_ENS_MEAN:valid_time = "20180105_000000" ;
tp_20180105_000000_all_all_ENS_MEAN:valid_time_ut = "1515110400" ;

So, the variable is in 2 dimensions (lat and lon), and does not have a third dimension or a time dimension. When setting the level for netCDF variables, the values inside the parentheses should directly correspond to the dimensions of the netCDF file. For your level, you have:

FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},*,*)"

This would indicate 3 dimensions with the first dimension being a time and then 2 more dimensions. However, all variables in that file, only have 2 dimensions, and none of those dimensions are time. So, the level needs to be changed to the following, to indicate that your data is 2 dimensions and does not have a time dimension:

FCST_VAR1_LEVELS = "(*,*)"

The second problem I see is that the Gen-Ens-Prod output doesn't have a variable called tp. The ensemble mean is called tp_20180105_000000_all_all_ENS_MEAN. If you want to run the statistics on the ensemble mean, you would need

FCST_VAR1_NAME =  tp_{valid?fmt=%Y%m%d_%H%M%S}_all_all_ENS_MEAN

I tried the following command in Plot-Data-Plane, and it produced output as expected. Note that time templates don't work as input to plot-data-plane so I have to list the actual date here:

plot_data_plane gen_ens_prod_20180104I_20180105V_ens.nc testmem.ps 'name="tp_20180105_000000_all_all_ENS_MEAN"; level="(*,*)";' -v 5

Does your Series-Analysis data run when you make the above changes?

Christina

[haksulee]

Hi Christina

Thanks, plot_data_plane worked, but SeriesAnalysis still errored out.
ERROR: Cannot specify both valid and init to time utility
ERROR : get_series_entry() -> Could not find data for tp__000000_all_all_ENS_MEAN(,*) in file list:

I used FCST_VAR1_NAME = tp_{valid?fmt=%Y%m%d}_000000_all_all_ENS_MEAN in the config file, based on FCST_VARNAME at https://metplus.readthedocs.io/en/main_v6.0/Users_Guide/glossary.html
but the log message shows tp*_000000_all_all_ENS_MEAN, not tp_20180105_000000_all_all_ENS_MEAN
Please, let me know what I did wrong.

log messages and the config file used are available at https://drive.google.com/drive/folders/1URAa0l8f5ba-KcItvxC22yjRDTbaWffG

[CPKalb]

The problem here is that support for time templates as part of variable names does not seem to be working in series analysis at this time (though it does work in other MET tools). This means that the {valid?fmt=%Y%m%d} that is specified in the FCST_VAR1_NAME unfortunately will not work. The path forward will be to remove the date from the variable names in the netCDF files, so that your variable names look like tp_all_all_ENS_MEAN, rather than having the date stamp. I'm investigating the best way to do this and will get back to you on that as soon as I have more information.

Christina

[haksulee]

Based on your comment "The output files you are using from Gen-Ens-Prod do not have a time dimension for the variables. ", I guess I have to add a time dimension to the output files from Gen-Ens-Prod. How can I do that?

[CPKalb]

Adding a time dimension will not work in this case. A time dimension is not needed to read the data into MET. The problem above was that your level specification indicated that you had a time dimension when you did not. The level specification for netCDF files needs to indicate the dimensions of your variable.

An additional problem here is with the date stamp that is part of the variable name. This date stamp is not handled by Series-Analysis. I'm investigating how to handle this and will get back to you.

[CPKalb]

Hi Haksulee:

After talking to the engineers here, we determined we need a fix so that Series-Analysis can read data with time templates specified in the FCST_VAR1_NAME. We are working on merging this fix into METplus at the moment. If you are able to download the most recent develop version of METplus and run using that, it will be the easiest solution. As I said above, that fix is not in place yet, but we are close and it should be ready by the end of the day. I can post here when the fix is available as well as which branch to use. If you are able to wait for the fix and use this version of METplus, then it should work with the specifications we have above.

If you are not able to download and use this updated version of METplus (when it becomes available), or if you need to move forward with verification rather than waiting for this change, then you will need to rename your variables inside the netCDF files to omit the date and time stamp. I can think of two ways to accomplish this. The first would be to use NCO tools to edit the variable names. The second would be to use MET's PCP-Combine. You can rename a variable by running the PCP-Combine tool with the add option, inputting a single file and then using FCST_PCP_COMBINE_OUTPUT_NAME to rename the output variable.

I can provide more information on downloading the new version of METplus or on running PCP-Combine to rename variables if you want to proceed with either of those directions. Questions on NCO would need to be directed to their discussion forum.

Christina

[haksulee]

Hi Christina

Thanks for letting me know the options available.
If the new version of METplus is available by the end of the day, I can wait for the new version and try it.
I must use a docker/container version of metplus, because I cannot install softwares on my office machines.
I am currently using the docker image dtcenter/metplus: 5.1-latest
Is the fix also available in the docker/container version of metplus?
If not, I will need PCP-Combine to rename variables.

Thanks
Haksu

[CPKalb]

The fix is now available in the develop version of METplus (https://github.com/dtcenter/METplus/tree/develop). I don't think there is a docker version of this, as I think we only do that for official releases.

Christina

[haksulee]

Hi Christina

Ok, in that case, could you teach me how to use PCP-Combine to rename variables?

Haksu

[CPKalb]

For PCP-Combine, you want to use the add command, but instead of inputting 2 files to add, you will only have one. Then, easiest way to do this is through a user-defined command in METplus. I'm linking some examples of user defined commands in METplus below. To do this, you will want to have set:

FCST_PCP_COMBINE_COMMAND = -add {FCST_PCP_COMBINE_INPUT_DIR}/{FCST_PCP_COMBINE_INPUT_TEMPLATE}

Here is a METplus configuration for the user defined PCP-Combine options:
https://metplus.readthedocs.io/en/latest/generated/met_tool_wrapper/PCPCombine/PCPCombine_user_defined.html#sphx-glr-generated-met-tool-wrapper-pcpcombine-pcpcombine-user-defined-py

As I mentioned above, you will also need to set FCST_PCP_COMBINE_OUTPUT_NAME. This variable isn't in the example configuration, but will be the new output variable name

Christina

[haksulee]

Hi Christina

Thank you for the instructions.
I will test it and let you know how it goes.

Have a good weekend!
Haksu

[haksulee]

Hi Christina

I tried PCP-Combine but got the following error:
ERROR : yyerror() -> syntax error in file "/test/PCPCombine/gefsv12_1dpcp/output_GenEnsProd/tmp/met_config_868_1"
ERROR : line = 1
ERROR : column = 0
ERROR : text = "/"
ERROR : /test/PCPCombine/gefsv12_1dpcp/output_GenEnsProd/ens_mean_20171005I_20171006V_ens.nc

Could you let me know what I can try?

I don't understand why "/" can cause the error.
Related files are at https://drive.google.com/drive/folders/1xK396aWlKtN0GgJB7ET02Iug8UPhqn-p

[georgemccabe]

Hi @haksulee,

The fix is available in the metplus-dev/develop Docker image on DockerHub. You can use this version to test. However, we recommend against using the develop version long term because it changes often and has potential to break things. When a stable release is available, you should switch to using that version instead of develop.

I have not used podman personally, but from what I understand you should be able to use podman to pull images from DockerHub. Something like this may work: podman pull docker.io/metplus-dev/develop.

I hope that helps!

[haksulee]

Hi George McCabe

Thank you.
Let me download and test the development version.

Haksu

[haksulee]

Hi George McCabe

It appears the docker version of metplus also works with podman.
The fix added to the development version of metplus works ok without issues. Let me continue testing the series analysis module.

Thanks
Haksu

[georgemccabe]

Great to hear!