ROC_AUC use in Series Analysis

[jhicks14]

Good afternoon,

I have a configuration file that uses Series Analysis so that I can use the NetCDF output to plot a spatial map. I am using GEFS probabilities >=35% as the "hit" threshold to verify heat index values >=100F. I would like the "ROC_AUC" statistic to be output to the NetCDF file, but I seem to be having trouble with this. I have used the Series Analysis overrides to get skill metrics (such as HSS) to output to the NetCDF file:

SERIES_ANALYSIS_MET_CONFIG_OVERRIDES = output_stats = {fho = [ "H_RATE" ]; ctc = [ "FY_OY", "FY_ON", "FN_OY", "FN_ON" ]; cts = [ "SEDI" ]; pstd = [ "ROC_AUC" ];} fcst = {cat_thresh = [ ge35 ];}; obs = {cat_thresh = [ ge100 ];}

Everything is this override statement works except for the addition of ROC_AUC to the NetCDF file. I should also mention that I used GridStat to produce the spatially-averaged ROC_AUC score successfully, which required "FCST_IS_PROB = True"

I'll attach the config file for reference. Thanks!

Heat_Index_GEFS_Forecast_Prob_Greater_100F_Hit_gte35.txt

[georgemccabe]

Hi @jhicks14, could you send me the METplus log file for this run?

[jhicks14]

Hi @georgemccabe ,

Sure thing! I have it attached. Let me know if you'd like me to re-run at a higher (or lower) verbosity (should be at 4)
metplus.log.20220504172030.txt
.

[georgemccabe]

Hi @jhicks14, thank you for sending the log file. I did some testing locally and determined that METplus 4.0 does not read FCST_IS_PROB in the SeriesAnalysis wrapper. Your log output confirms this since the fcst field info is set properly for GridStat but not for SeriesAnalysis. This has since been fixed, so upgrading to METplus 4.1.0 should resolve this issue.

If you must continue to use 4.0.0, then you should be able to get around this by setting prob=TRUE in the fcst dictionary via the MET config overrides:

SERIES_ANALYSIS_MET_CONFIG_OVERRIDES = output_stats = {fho = [ "H_RATE" ]; ctc = [ "FY_OY", "FY_ON", "FN_OY", "FN_ON" ]; cts = [ "SEDI" ]; pstd = [ "ROC_AUC" ];} fcst = {cat_thresh = [ ge35 ]; prob=TRUE;}; obs = {cat_thresh = [ ge100 ];}

Please keep in mind that if that data contains scalar values, it can compute SL1L2 partial sums or CNT stats or CTC contingency table or CTS contingency table stats, and if it contains probabilities, it can compute PCT, PSTD, PRC, and/or PJC outputs. It isn't setup to process the same input data as being both probabilities and non-probabilities in the same run. This means that if you configure it to mark the forecast data as probabilistic, then you will no longer see the CNT, CTC, or CTS values in the NetCDF output.

[jhicks14]

Thanks, @georgemccabe !

I thought the "FCST_IS_PROB" I had set to "TRUE" would carry over to both the grid_stat and series_analysis tools. The override seems to work, but I'm left with a new error that I cannot seem to fix:

"ERROR : check_prob_thresh() -> When verifying a probability field, you must select at lea
st 3 thresholds beginning with 0.0 and ending with 1.0."

I tried adding the following override to my config file:

SERIES_ANALYSIS_MET_CONFIG_OVERRIDES = output_stats = {fho = [ "H_RATE" ]; ctc = [ "FY_OY", "FY_ON", "FN_OY", "FN_ON" ]; cts = [ "SEDI" ]; pstd = [ "ROC_AUC" ];} fcst = {cat_thresh = [ ge1 ]; censor_thresh = [ ge35 ]; censor_val = [ 1 ]; censor_thresh = [ lt35 ]; censor_val = [ 0 ]; prob=TRUE; prob_thresh = [ ge0,ge0.5,ge1 ];}; obs = {cat_thresh = [ ge100 ];}

I want forecast probabilities >=35% to act as the positive class, so I tried setting the censor thresh >=35 and the corresponding censor_val to 1. I then used the same logic to set values less than 35% to 0. In setting the "prob_thresh" value arbitrarily to [ ge0,ge0.5,ge1 ], I thought that series_analysis would run successfully, but it did not.

Is there a way around this error? I've tried setting:

FCST_VAR1_OPTIONS = convert(x) = x/100 as well and then thresholding, but I still receive the same error.

[georgemccabe]

Hi @jhicks14, I was advised by a colleague that the threshold values for evaluating probabilities in this case should actually be set in the cat_thresh variable instead of prob_thresh.

[jhicks14]

Thanks, @georgemccabe ! I now have it working.

I appreciate the help!