Feature 1843 scatter index (#1863)

* Added scatter index variables: SI, SI_BCL, SI_BCU to cnt columns. SL

* Per issue #1843 Added Scatter Index variables (SI, SI_BCL, SI_BCU) to the CNT stat type. SL

* Added Scatter Index (SI, SI_BCL, SI_BCU) to cnt stat type header. SL

* Per issue #1843, added Scatter Index variables to write_cnt_cols(). SL

* Per issue #1843 added Scatter Index (si) to CNTinfo class. SL

* Per issue #1843 added Scatter Index (SI, si) to CNTInfo::get_stat(). SL

* Per issue #1843 added Scatter Index (SI, si) calculation to both versions of compute_cntinfo(). SL

* Per issue #1843, forgot semi-colon after SI calculation, fixed. SL

* Added Scatter Index (si) to some of the clear() and allocate() functions. SL

* Per issue #1843, added Scatter Index (si) to compute_cnt_mean(). SL

* Per issue #1843, added Scatter Index (si) to bootstrap interval calculation and to the CNT write function. SL

* Per issue #1843 added Scatter Index (SI, SI_BCL, SI_BCU) to store_stat_cnt(). SL

* Per issue #1843, in get_stat() moved Scatterd Index (SI) line to be below RMSE for consitency. SL

* Added Scatter Index (SI) to the CNT output format table. SL

* Added Scatter Index (SI) to CNT statistics definitions. SL

* Update appendixC.rst

* Update stat_columns.cc

* Update compute_ci.cc

* Update met_stats.cc

* Per #1843, update write_cntinfo() function. Since we're READING the SI value immediately after the RMSE value when computing CI's, we also need to write SI immediately after RMSE. Otherwise, we'll be computing CI's using the wrong statistics replicates.

* Per #1843, check for divide by zero when computing SI... this could happen if comparing the same input file to itself with Grid-Stat.

* Per #1843, correct definition of SI from RMSE/ME to RMSE/OBAR.

* Per #1843, updated the SI definition to divide by OBAR instead of ME, but forgot to update the divide-by-zero check.

Co-authored-by: Seth Linden <linden@kiowa.rap.ucar.edu>
Co-authored-by: johnhg <johnhg@ucar.edu>

met/data/table_files/met_header_columns_V10.1.txt

@@ -1,4 +1,4 @@
-V10.1 : STAT : CNT      : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU E25 E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL E90_BCU EIQR EIQR_BCL EIQR_BCU MAD MAD_BCL MAD_BCU ANOM_CORR ANOM_CORR_NCL ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCU ME2 ME2_BCL ME2_BCU MSESS MSESS_BCL MSESS_BCU RMSFA RMSFA_BCL RMSFA_BCU RMSOA RMSOA_BCL RMSOA_BCU ANOM_CORR_UNCNTR ANOM_CORR_UNCNTR_BCL ANOM_CORR_UNCNTR_BCU
+V10.1 : STAT : CNT      : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU E25 E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL E90_BCU EIQR EIQR_BCL EIQR_BCU MAD MAD_BCL MAD_BCU ANOM_CORR ANOM_CORR_NCL ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCU ME2 ME2_BCL ME2_BCU MSESS MSESS_BCL MSESS_BCU RMSFA RMSFA_BCL RMSFA_BCU RMSOA RMSOA_BCL RMSOA_BCU ANOM_CORR_UNCNTR ANOM_CORR_UNCNTR_BCL ANOM_CORR_UNCNTR_BCU SI SI_BCL SI_BCU
 V10.1 : STAT : CTC      : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FY_OY FY_ON FN_OY FN_ON
 V10.1 : STAT : CTS      : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL BASER BASER_NCL BASER_NCU BASER_BCL BASER_BCU FMEAN FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCU ACC ACC_NCL ACC_NCU ACC_BCL ACC_BCU FBIAS FBIAS_BCL FBIAS_BCU PODY PODY_NCL PODY_NCU PODY_BCL PODY_BCU PODN PODN_NCL PODN_NCU PODN_BCL PODN_BCU POFD POFD_NCL POFD_NCU POFD_BCL POFD_BCU FAR FAR_NCL FAR_NCU FAR_BCL FAR_BCU CSI CSI_NCL CSI_NCU CSI_BCL CSI_BCU GSS GSS_BCL GSS_BCU HK HK_NCL HK_NCU HK_BCL HK_BCU HSS HSS_BCL HSS_BCU ODDS ODDS_NCL ODDS_NCU ODDS_BCL ODDS_BCU LODDS LODDS_NCL LODDS_NCU LODDS_BCL LODDS_BCU ORSS ORSS_NCL ORSS_NCU ORSS_BCL ORSS_BCU EDS EDS_NCL EDS_NCU EDS_BCL EDS_BCU SEDS SEDS_NCL SEDS_NCU SEDS_BCL SEDS_BCU EDI EDI_NCL EDI_NCU EDI_BCL EDI_BCU SEDI SEDI_NCL SEDI_NCU SEDI_BCL SEDI_BCU BAGSS BAGSS_BCL BAGSS_BCU
 V10.1 : STAT : FHO      : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL F_RATE H_RATE O_RATE

met/docs/Users_Guide/appendixC.rst

@@ -507,6 +507,15 @@ Called "RMSE" in CNT output :numref:`table_PS_format_info_CNT`
 RMSE is simply the square root of the MSE, :math:`\text{RMSE} = \sqrt{\text{MSE}}`. 
 
 
+Scatter Index (SI)
+~~~~~~~~~~~~~~~~~~
+
+Called "SI" in CNT output :numref:`table_PS_format_info_CNT`
+
+SI is the ratio of the root mean squared error to the average observation value, :math:`\text{SI} = \text{RMSE} / \text{OBAR}`.
+
+Smaller values of SI indicate better agreement between the model and observations (less scatter on scatter plot).
+
 Standard deviation of the error
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

met/docs/Users_Guide/point-stat.rst

@@ -852,6 +852,10 @@ The first set of header columns are common to all of the output files generated
   * - 120-122
     - ANOM_CORR_UNCNTR, :raw-html:`<br />` ANOM_CORR_UNCNTR_BCL, :raw-html:`<br />` ANOM_CORR_UNCNTR_BCU
     - The uncentered Anomaly Correlation excluding mean error including bootstrap upper and lower confidence limits
+  * - 123-125
+    - SI, :raw-html:`<br />` SI_BCL, :raw-html:`<br />` SI_BCU
+    - Scatter Index including bootstrap upper and lower confidence limits
+
 
 .. _table_PS_format_info_MCTC:
 

met/src/basic/vx_util/stat_column_defs.h

@@ -103,7 +103,8 @@ static const char * cnt_columns [] = {
    "MSESS",            "MSESS_BCL",            "MSESS_BCU",
    "RMSFA",            "RMSFA_BCL",            "RMSFA_BCU",
    "RMSOA",            "RMSOA_BCL",            "RMSOA_BCU",
-   "ANOM_CORR_UNCNTR", "ANOM_CORR_UNCNTR_BCL", "ANOM_CORR_UNCNTR_BCU"
+   "ANOM_CORR_UNCNTR", "ANOM_CORR_UNCNTR_BCL", "ANOM_CORR_UNCNTR_BCU",
+   "SI",               "SI_BCL",               "SI_BCL" 
 };
 
 static const char * sl1l2_columns [] = {

met/src/libcode/vx_stat_out/stat_columns.cc

@@ -2341,7 +2341,8 @@ void write_cnt_cols(const CNTInfo &cnt_info, int i,
    //    MSESS,            MSESS_BCL,            MSESS_BCU,
    //    RMSFA,            RMSFA_BCL,            RMSFA_BCU,
    //    RMSOA,            RMSOA_BCL,            RMSOA_BCU,
-   //    ANOM_CORR_UNCNTR, ANOM_CORR_UNCNTR_BCL, ANOM_CORR_UNCNTR_BCU
+   //    ANOM_CORR_UNCNTR, ANOM_CORR_UNCNTR_BCL, ANOM_CORR_UNCNTR_BCU,
+   //    SI,               SI_BCL,               SI_BCU
    //
 
    at.set_entry(r, c+0,  // Total Number of Grid Points
@@ -2635,6 +2636,15 @@ void write_cnt_cols(const CNTInfo &cnt_info, int i,
    at.set_entry(r, c+96, // Anomaly Correlation Uncentered BCU
       cnt_info.anom_corr_uncntr.v_bcu[i]);
 
+   at.set_entry(r, c+97, // Scatter Index
+      cnt_info.si.v);
+
+   at.set_entry(r, c+98, // Scatter Index BCL
+      cnt_info.si.v_bcl[i]);
+
+   at.set_entry(r, c+99, // Scatter Index BCU
+      cnt_info.si.v_bcu[i]);
+
    return;
 }
 

met/src/libcode/vx_statistics/compute_ci.cc

@@ -1265,6 +1265,18 @@ void compute_cnt_stats_ci_bca(const gsl_rng *rng_ptr,
                               cnt_info.rmse.v_bcl[i],
                               cnt_info.rmse.v_bcu[i]);
 
+      //
+      // Compute bootstrap interval for si
+      //
+      s = cnt_info.si.v;
+      read_ldf(cnt_i_file, c,   si_na);
+      read_ldf(cnt_r_file, c++, sr_na);
+      for(i=0; i<cnt_info.n_alpha; i++)
+         compute_bca_interval(s, si_na, sr_na,
+                              cnt_info.alpha[i],
+                              cnt_info.si.v_bcl[i],
+                              cnt_info.si.v_bcu[i]);
+
       //
       // Compute bootstrap interval for e10
       //
@@ -2274,6 +2286,17 @@ void compute_cnt_stats_ci_perc(const gsl_rng *rng_ptr,
                                cnt_info.rmse.v_bcl[i],
                                cnt_info.rmse.v_bcu[i]);
 
+      //
+      // Compute bootstrap interval for si
+      //
+      s = cnt_info.si.v;
+      read_ldf(cnt_r_file, c++, sr_na);
+      for(i=0; i<cnt_info.n_alpha; i++)
+         compute_perc_interval(s, sr_na,
+                               cnt_info.alpha[i],
+                               cnt_info.si.v_bcl[i],
+                               cnt_info.si.v_bcu[i]);
+
       //
       // Compute bootstrap interval for e10
       //
@@ -3879,14 +3902,15 @@ void write_cntinfo(ofstream &tmp_out, const CNTInfo &c) {
    char line[max_line_len];
 
    snprintf(line, max_line_len,
-           "%f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f",
+           "%f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f %f",
            c.fbar.v,    c.fstdev.v,    c.obar.v,          c.ostdev.v,
            c.pr_corr.v, c.anom_corr.v,
            c.rmsfa.v,   c.rmsoa.v,     c.anom_corr_uncntr.v,
            c.me.v,      c.me2.v,       c.estdev.v,        c.mbias.v,
            c.mae.v,     c.mse.v,       c.msess.v,         c.bcmse.v,
-           c.rmse.v,    c.e10.v,       c.e25.v,           c.e50.v,
-           c.e75.v,     c.e90.v,       c.eiqr.v,          c.mad.v);
+           c.rmse.v,    c.si.v,        c.e10.v,           c.e25.v,
+           c.e50.v,     c.e75.v,       c.e90.v,           c.eiqr.v,
+           c.mad.v);
 
    tmp_out << line << "\n";
 

met/src/libcode/vx_statistics/compute_stats.cc

@@ -120,6 +120,14 @@ void compute_cntinfo(const SL1L2Info &s, bool aflag, CNTInfo &cnt_info) {
    // Compute root mean squared error
    cnt_info.rmse.v = sqrt(cnt_info.mse.v);
 
+   // Compute Scatter Index (SI)
+   if(!is_eq(cnt_info.obar.v, 0.0)) {
+      cnt_info.si.v = cnt_info.rmse.v / cnt_info.obar.v;
+   }
+   else {
+      cnt_info.si.v = bad_data_double;
+   }
+
    // Compute normal confidence intervals
    cnt_info.compute_ci();
 
@@ -342,6 +350,17 @@ void compute_cntinfo(const PairDataPoint &pd, const NumArray &i_na,
    //
    cnt_info.rmse.v = sqrt(err_sq_bar);
 
+   //
+   // Compute Scatter Index (SI)
+   //
+   if(!is_eq(cnt_info.obar.v, 0.0)) {
+      cnt_info.si.v = cnt_info.rmse.v / cnt_info.obar.v;
+   }
+   else {
+      cnt_info.si.v = bad_data_double;
+   }
+
+
    //
    // Only compute the Kendall Tau and Spearman's Rank corrleation
    // coefficients if the rank_flag is set.
@@ -1196,6 +1215,9 @@ void compute_cnt_mean(const CNTInfo *cnt_info, int n,
    for(i=0,na.erase(); i<n; i++) na.add(cnt_info[i].rmse.v);
    cnt_mean.rmse.v = na.mean();
 
+   for(i=0,na.erase(); i<n; i++) na.add(cnt_info[i].si.v);
+   cnt_mean.si.v = na.mean();
+
    for(i=0,na.erase(); i<n; i++) na.add(cnt_info[i].e10.v);
    cnt_mean.e10.v = na.mean();
 

met/src/libcode/vx_statistics/met_stats.cc

@@ -735,6 +735,7 @@ void CNTInfo::clear() {
    msess.clear();
    bcmse.clear();
    rmse.clear();
+   si.clear();
    e10.clear();
    e25.clear();
    e50.clear();
@@ -783,6 +784,7 @@ void CNTInfo::assign(const CNTInfo &c) {
    msess            = c.msess;
    bcmse            = c.bcmse;
    rmse             = c.rmse;
+   si               = c.si;
    e10              = c.e10;
    e25              = c.e25;
    e50              = c.e50;
@@ -833,6 +835,7 @@ void CNTInfo::allocate_n_alpha(int i) {
       msess.allocate_n_alpha(n_alpha);
       bcmse.allocate_n_alpha(n_alpha);
       rmse.allocate_n_alpha(n_alpha);
+      si.allocate_n_alpha(n_alpha);
       e10.allocate_n_alpha(n_alpha);
       e25.allocate_n_alpha(n_alpha);
       e50.allocate_n_alpha(n_alpha);
@@ -1016,6 +1019,7 @@ double CNTInfo::get_stat(const char *stat_name) {
    else if(strcmp(stat_name, "MSE"             ) == 0) v = mse.v;
    else if(strcmp(stat_name, "BCMSE"           ) == 0) v = bcmse.v;
    else if(strcmp(stat_name, "RMSE"            ) == 0) v = rmse.v;
+   else if(strcmp(stat_name, "SI"              ) == 0) v = si.v;
    else if(strcmp(stat_name, "E10"             ) == 0) v = e10.v;
    else if(strcmp(stat_name, "E25"             ) == 0) v = e25.v;
    else if(strcmp(stat_name, "E50"             ) == 0) v = e50.v;

met/src/libcode/vx_statistics/met_stats.h

@@ -185,6 +185,9 @@ class CNTInfo {
       CIInfo e10, e25, e50, e75, e90, eiqr;
       CIInfo mad;
 
+      // Scatter Index
+      CIInfo si;
+
       int n_ranks, frank_ties, orank_ties;
 
       void clear();

met/src/tools/core/series_analysis/series_analysis.cc

@@ -1604,6 +1604,9 @@ void store_stat_cnt(int n, const ConcatString &col,
       else if(c == "RMSE")                 { v = cnt_info.rmse.v;                    }
       else if(c == "RMSE_BCL")             { v = cnt_info.rmse.v_bcl[i];             }
       else if(c == "RMSE_BCU")             { v = cnt_info.rmse.v_bcu[i];             }
+      else if(c == "SI")                   { v = cnt_info.si.v;                      }
+      else if(c == "SI_BCL")               { v = cnt_info.si.v_bcl[i];               }
+      else if(c == "SI_BCU")               { v = cnt_info.si.v_bcu[i];               }
       else if(c == "E10")                  { v = cnt_info.e10.v;                     }
       else if(c == "E10_BCL")              { v = cnt_info.e10.v_bcl[i];              }
       else if(c == "E10_BCU")              { v = cnt_info.e10.v_bcu[i];              }

test/hdr/met_10_1.hdr

@@ -1,4 +1,4 @@
-CNT           : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU E25 E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL E90_BCU EIQR EIQR_BCL EIQR_BCU MAD MAD_BCL MAD_BCU ANOM_CORR ANOM_CORR_NCL ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCU ME2 ME2_BCL ME2_BCU MSESS MSESS_BCL MSESS_BCU RMSFA RMSFA_BCL RMSFA_BCU RMSOA RMSOA_BCL RMSOA_BCU ANOM_CORR_UNCNTR ANOM_CORR_UNCNTR_BCL ANOM_CORR_UNCNTR_BCU
+CNT           : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU FSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU E25 E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL E90_BCU EIQR EIQR_BCL EIQR_BCU MAD MAD_BCL MAD_BCU ANOM_CORR ANOM_CORR_NCL ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCU ME2 ME2_BCL ME2_BCU MSESS MSESS_BCL MSESS_BCU RMSFA RMSFA_BCL RMSFA_BCU RMSOA RMSOA_BCL RMSOA_BCU ANOM_CORR_UNCNTR ANOM_CORR_UNCNTR_BCL ANOM_CORR_UNCNTR_BCU SI SI_BCL SI_BCU
 CTC           : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FY_OY FY_ON FN_OY FN_ON
 CTS           : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL BASER BASER_NCL BASER_NCU BASER_BCL BASER_BCU FMEAN FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCU ACC ACC_NCL ACC_NCU ACC_BCL ACC_BCU FBIAS FBIAS_BCL FBIAS_BCU PODY PODY_NCL PODY_NCU PODY_BCL PODY_BCU PODN PODN_NCL PODN_NCU PODN_BCL PODN_BCU POFD POFD_NCL POFD_NCU POFD_BCL POFD_BCU FAR FAR_NCL FAR_NCU FAR_BCL FAR_BCU CSI CSI_NCL CSI_NCU CSI_BCL CSI_BCU GSS GSS_BCL GSS_BCU HK HK_NCL HK_NCU HK_BCL HK_BCU HSS HSS_BCL HSS_BCU ODDS ODDS_NCL ODDS_NCU ODDS_BCL ODDS_BCU LODDS LODDS_NCL LODDS_NCU LODDS_BCL LODDS_BCU ORSS ORSS_NCL ORSS_NCU ORSS_BCL ORSS_BCU EDS EDS_NCL EDS_NCU EDS_BCL EDS_BCU SEDS SEDS_NCL SEDS_NCU SEDS_BCL SEDS_BCU EDI EDI_NCL EDI_NCU EDI_BCL EDI_BCU SEDI SEDI_NCL SEDI_NCU SEDI_BCL SEDI_BCU BAGSS BAGSS_BCL BAGSS_BCU
 FHO           : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL F_RATE H_RATE O_RATE