Feature 1020 set_attrs (#1406)

* Per #1020, update the VarInfo class heirarchy to store the contents of the set_attrs dictionary config entries. Still need to add code to parse those entries and update the tests and documentation.

* Per #1020, update the VarInfo class to parse and store the set_attrs dictionary members.

* Per #1020, update point_stat and ensemble_stat to write the obs_info units to the OBS_UNITS output column instead of writing a constant NA string.

* Per #1020, adds attr access functions to the VarInfo class. VarInfo::name_attr() for example returns SetAttrsName, if set, and simply Name otherwise.

* Per #1020, this is work in progress. Swapping out calls to shc set_fcst_var, set_obs_var, set_fcst_level, set_obs_level, set_fcst_units, and set_obs_units with calls to the corresponding attrs access function. Still many more updates to the code required.

* Per #1020, add support for set_attrs.set_accum to override the accumulation interval read from the data.

* Per #1020, add a set_attrs() utility function to update the metadata in a DataPlane object using the contents of a set_attr Dictionary stored in a VarInfo object.

* Per #1020, add grid.nxy() utility function to return Nx*Ny instead of always having to type out grid.nx()*grid.ny().

* Per #1020, switch VarInfo::SetAttrsGrid from a ConcatString to a Grid object. And parse that grid in the set_dict() function. The huge drawback here is that the grid may be specified as a named grid or using a grid specification string... but it cannot be defined as the path to a gridded data file. The library dependency logic is just too complex, and I'd need to move a lot of code around to make this work. For now, just leave that feature out.

* No real change, just adding a blank line that was missing.

* Per #1020, add Met2dDataFile::set_grid() function to override the grid spec parsed from the file. Also, update the process_data_plane() function to handle updating the metadata and grid defintions.

* Per #1020, had to add -lvx_color to 5 Makefile.am files to get them linking again.

* Per #1020, work on log messages to make the set_attrs parsing more clear.

* Per #1020, update logic for how to parse set_attrs entries. For example, parse them as set_attrs.set_name instead of set_name from the set_attrs dictionary. That makes setting up the config file more flexible.

* Per #1020, update Grid-Stat to write NetCDF output using name_attr, level_attr, and units_attr instead of name(), level_name(), and units(). This change should be made everywhere NetCDF output files are written.

* Per #1020, update the parsing logic to check for embedded whitespace in name, level, and units.

* Per #1020, removed unused SetAttrsEnsemble option.

* Per #1020, add new test in unit_grid_stat.xml to exercise the set_attrs functionality.

* Per #1020, update data/config/README with information about the set_attrs dictionary.

* Per #1020, in Ensemble-Stat, replace calls to name(), level_name(), and units() with calls to name_attr(), level_attr(), and units_attr().

* Per #1020, in Grid-Diag, replace calls to name(), level_name(), and units() with calls to name_attr(), level_attr(), and units_attr().

* Deleting stale, commented out, development code.

* Per #1020, in MODE and MTD replace calls to name(), level_name(), and units() with calls to name_attr(), level_attr(), and units_attr().

* Per #1020, in PCP-Combine, Shift-Data-Plane, and Regrid-Data-Plane, replace calls to name(), level_name(), and units() with calls to name_attr(), level_attr(), and units_attr().

* Per #1020, in TCRMW and Series-Analysis, replace calls to name(), level_name(), and units() with calls to name_attr(), level_attr(), and units_attr().

* Per #1020, in Wavelet-Stat, replace calls to name(), level_name(), and units() with calls to name_attr(), level_attr(), and units_attr().

* src/tools/core/wavelet_stat/wavelet_stat.cc

* src/tools/tc_utils/tc_rmw/tc_rmw.cc

* Adding a couple of spaces to unit_grid_stat.xml to be safe.

* Per #1020, update call to set_obs_units() in point-stat to avoid a warning about setting a header column to a null string.

* Per #1020, changing the order of parsing to see if this fixes the runtime issues I found on dakota.

* Per #1020, so this represents a course correction. Rather than grouping all of these metadata modifiers into a set_attrs dictionary, I'm now parsing them all as individual entries. See #1020 issue comments for more details.

* Per #1020, respond to PR comments by updating Grid-Stat to include the user-specified long_name string in the NetCDF matched pairs file from Grid-Stat.

Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu>

met/data/config/README

@@ -538,6 +538,35 @@ regrid = {
 //          censor_thresh = [ >12000 ];
 //          censor_val    = [ 12000 ];
 //
+//      - Several configuration options are provided to override and correct the
+//        metadata read from the input file. The supported options are listed
+//        below:
+//
+//        // Data attributes
+//        set_attr_name      = "string";
+//        set_attr_level     = "string";
+//        set_attr_units     = "string";
+//        set_attr_long_name = "string";
+//
+//        // Time attributes
+//        set_attr_init  = "YYYYMMDD[_HH[MMSS]]";
+//        set_attr_valid = "YYYYMMDD[_HH[MMSS]]";
+//        set_attr_lead  = "HH[MMSS]";
+//        set_attr_accum = "HH[MMSS]";
+//
+//        // Grid definition (must match the actual data dimensions)
+//        set_attr_grid  = "named grid or grid specification string";
+//
+//        // Flags
+//        is_precipitation     = boolean;
+//        is_specific_humidity = boolean;
+//        is_u_wind            = boolean;
+//        is_v_wind            = boolean;
+//        is_grid_relative     = boolean;
+//        is_wind_speed        = boolean;
+//        is_wind_direction    = boolean;
+//        is_prob              = boolean;
+//
 //      - The "cat_thresh" entry is an array of thresholds to be used when
 //        computing categorical statistics.
 //

met/internal_tests/libcode/vx_data2d_factory/Makefile.am

@@ -38,6 +38,7 @@ test_factory_LDADD = -lvx_data2d_factory \
 	-lvx_gsl_prob \
 	-lvx_util \
 	-lvx_math \
+	-lvx_color \
 	-lvx_cal \
 	-lvx_log \
 	-lm -lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas

met/internal_tests/libcode/vx_data2d_grib/Makefile.am

@@ -22,6 +22,7 @@ test_read_grib1_LDADD = -lvx_data2d_grib \
 	-lvx_gsl_prob \
 	-lvx_util \
 	-lvx_math \
+	-lvx_color \
 	-lvx_cal \
 	-lvx_log \
 	-lgsl -lgslcblas

met/internal_tests/libcode/vx_data2d_nc_met/Makefile.am

@@ -23,6 +23,7 @@ test_read_nc_met_LDADD = -lvx_data2d_nc_met \
 	-lvx_gsl_prob \
 	-lvx_util \
 	-lvx_math \
+	-lvx_color \
 	-lvx_cal \
 	-lvx_log \
 	-lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas

met/internal_tests/libcode/vx_data2d_nccf/Makefile.am

@@ -23,6 +23,7 @@ test_read_nccf_LDADD = -lvx_data2d_nccf \
 	-lvx_gsl_prob \
 	-lvx_util \
 	-lvx_math \
+	-lvx_color \
 	-lvx_cal \
 	-lvx_log \
 	-lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas

met/src/basic/vx_config/config_constants.h

@@ -617,6 +617,27 @@ static const char conf_key_trunc_factor[]      = "gaussian_trunc_factor";
 static const char conf_key_eclv_points[]       = "eclv_points";
 static const char conf_key_var_name_map[]      = "var_name_map";
 
+//
+// Entries to override file metadata 
+//
+static const char conf_key_set_attr_name[]        = "set_attr_name";
+static const char conf_key_set_attr_units[]       = "set_attr_units";
+static const char conf_key_set_attr_level[]       = "set_attr_level";
+static const char conf_key_set_attr_long_name[]   = "set_attr_long_name";
+static const char conf_key_set_attr_grid[]        = "set_attr_grid";
+static const char conf_key_set_attr_init[]        = "set_attr_init";
+static const char conf_key_set_attr_valid[]       = "set_attr_valid";
+static const char conf_key_set_attr_lead[]        = "set_attr_lead";
+static const char conf_key_set_attr_accum[]       = "set_attr_accum";
+static const char conf_key_is_precipitation[]     = "is_precipitation";
+static const char conf_key_is_specific_humidity[] = "is_specific_humidity";
+static const char conf_key_is_u_wind[]            = "is_u_wind";
+static const char conf_key_is_v_wind[]            = "is_v_wind";
+static const char conf_key_is_grid_relative[]     = "is_grid_relative";
+static const char conf_key_is_wind_speed[]        = "is_wind_speed";
+static const char conf_key_is_wind_direction[]    = "is_wind_direction";
+static const char conf_key_is_prob[]              = "is_prob";
+
 //
 // Climatology parameter key names
 //

met/src/libcode/vx_data2d/data2d_utils.cc

@@ -33,7 +33,7 @@ bool derive_wdir(const DataPlane &u2d, const DataPlane &v2d,
 
    mlog << Debug(3)
         << "Deriving wind direction from U and V wind components.\n";
-   
+
    //
    // Check that the dimensions match
    //
@@ -214,7 +214,7 @@ bool rotate_uv_grid_to_earth(const DataPlane &u2d, const DataPlane &v2d,
 
    mlog << Debug(3)
         << "Rotating U and V wind components from grid-relative to earth-relative.\n";
-   
+
    //
    // Check that the dimensions match
    //
@@ -277,3 +277,48 @@ bool rotate_uv_grid_to_earth(const DataPlane &u2d, const DataPlane &v2d,
 }
 
 ////////////////////////////////////////////////////////////////////////
+
+void set_attrs(const VarInfo *info, DataPlane &dp) {
+
+   if(!info) return;
+
+   //
+   // Update attributes, if requested
+   //
+
+   // init_time
+   if(info->init_attr() != (unixtime) 0) {
+      mlog << Debug(3) << "Resetting initialization time from "
+           << unix_to_yyyymmdd_hhmmss(dp.init()) << " to "
+           << unix_to_yyyymmdd_hhmmss(info->init_attr()) << ".\n";
+      dp.set_init(info->init_attr());
+   }
+
+   // valid_time
+   if(info->valid_attr() != (unixtime) 0) {
+      mlog << Debug(3) << "Resetting valid time from "
+           << unix_to_yyyymmdd_hhmmss(dp.valid()) << " to "
+           << unix_to_yyyymmdd_hhmmss(info->valid_attr()) << ".\n";
+      dp.set_valid(info->valid_attr());
+   }
+
+   // lead_time
+   if(!is_bad_data(info->lead_attr())) {
+      mlog << Debug(3) << "Resetting lead time from "
+           << sec_to_hhmmss(dp.lead()) << " to "
+           << sec_to_hhmmss(info->lead_attr()) << ".\n";
+      dp.set_lead(info->lead_attr());
+   }
+
+   // accum_time
+   if(!is_bad_data(info->accum_attr())) {
+      mlog << Debug(3) << "Resetting accumulation interval from "
+           << sec_to_hhmmss(dp.accum()) << " to "
+           << sec_to_hhmmss(info->accum_attr()) << ".\n";
+      dp.set_lead(info->accum_attr());
+   }
+
+   return;
+}
+
+////////////////////////////////////////////////////////////////////////

met/src/libcode/vx_data2d/data2d_utils.h

@@ -16,6 +16,7 @@
 
 #include "vx_grid.h"
 #include "data_plane.h"
+#include "var_info.h"
 
 ////////////////////////////////////////////////////////////////////////
 
@@ -33,6 +34,8 @@ extern bool rotate_uv_grid_to_earth(const DataPlane &u2d, const DataPlane &v2d,
                                     const Grid &,
                                     DataPlane &u2d_rot, DataPlane &v2d_rot);
 
+extern void set_attrs(const VarInfo *info, DataPlane &dp);
+
 ////////////////////////////////////////////////////////////////////////
 
 #endif   /*  __DATA_2D_UTILS_H__  */

met/src/libcode/vx_data2d/data_class.cc

@@ -20,6 +20,8 @@ using namespace std;
 #include "vx_log.h"
 #include "indent.h"
 #include "data_class.h"
+#include "data2d_utils.h"
+#include "apply_mask.h"
 
 ////////////////////////////////////////////////////////////////////////
 
@@ -121,8 +123,6 @@ void Met2dDataFile::mtddf_clear()
 
 {
 
-// mttd_clear();
-
 if ( Raw_Grid  )  { delete Raw_Grid;   Raw_Grid  = (Grid *) 0; }
 if ( Dest_Grid )  { delete Dest_Grid;  Dest_Grid = (Grid *) 0; }
 
@@ -192,7 +192,8 @@ const Grid & Met2dDataFile::grid() const
 
 if ( ! Dest_Grid )  {
 
-   mlog << Error << "\nMet2dDataFile::grid() -> no grid defined!\n\n";
+   mlog << Error << "\nMet2dDataFile::grid() -> "
+        << "no grid defined!\n\n";
 
    exit ( 1 );
 
@@ -212,7 +213,8 @@ const Grid & Met2dDataFile::raw_grid() const
 
 if ( ! Raw_Grid )  {
 
-   mlog << Error << "\nMet2dDataFile::raw_grid() -> no raw grid defined!\n\n";
+   mlog << Error << "\nMet2dDataFile::raw_grid() -> "
+        << "no raw grid defined!\n\n";
 
    exit ( 1 );
 
@@ -242,14 +244,52 @@ return;
 ////////////////////////////////////////////////////////////////////////
 
 
+void Met2dDataFile::set_grid(const Grid &grid)
+
+{
+
+mlog << Debug(3) << "Resetting grid definition from \""
+     << Dest_Grid->serialize() << "\" to \"" << grid.serialize()
+     << "\".\n";
+
+     //
+     // Make sure the grid dimensions do not change
+     //
+
+  if ( raw_nx() != grid.nx() || raw_ny() != grid.ny() )  {
+
+     mlog << Error << "\nMet2dDataFile::set_grid() -> "
+          << "When resetting the grid definition to \""
+          << grid.serialize() << "\", the grid dimensions "
+          << "cannot change (" << grid.nx() << ", " << grid.ny()
+          << ") != (" << raw_nx() << ", " << raw_ny() << ").\n\n";
+
+     exit ( 1 );
+
+  }
+
+if ( Dest_Grid )  { delete Dest_Grid;  Dest_Grid = 0; }
+
+Dest_Grid = new Grid;
+
+(*Dest_Grid) = grid;
+
+return;
+
+}
+
+
+////////////////////////////////////////////////////////////////////////
+
+
 void Met2dDataFile::copy_raw_grid_to_dest()
 
 {
 
 if ( ! Raw_Grid )  {
 
-   mlog << Error
-        << "Met2dDataFile::copy_raw_grid_to_dest() -> no raw grid set!\n\n";
+   mlog << Error << "\nMet2dDataFile::copy_raw_grid_to_dest() -> "
+        << "no raw grid set!\n\n";
 
    exit ( 1 );
 
@@ -276,27 +316,42 @@ void Met2dDataFile::process_data_plane(VarInfo *vinfo, DataPlane &dp)
 if ( ! vinfo )  return;
 
    //
-   // Apply shift to the right logic.
+   // Apply shift to the right logic
    //
 
 if ( ShiftRight != 0 )  dp.shift_right(ShiftRight);
 
    //
-   // Apply conversion logic.
+   // Apply conversion logic
    //
 
 dp.convert(vinfo->ConvertFx);
 
    //
-   // Apply censor logic.
+   // Apply censor logic
    //
 
 dp.censor(vinfo->censor_thresh(), vinfo->censor_val());
 
+   //
+   // Update the metadata, if requested
+   //
+
+set_attrs(vinfo, dp);
+
+   //
+   // Update the grid definition, if requested
+   //
+
+if ( vinfo->grid_attr().nxy() > 0 )  {
+
+   set_grid(vinfo->grid_attr());
+
+}
+
 return;
 
 }
 
 
 ////////////////////////////////////////////////////////////////////////
-

met/src/libcode/vx_data2d/data_class.h

@@ -112,6 +112,7 @@ class Met2dDataFile : public Met2dData {
          //
 
       void set_shift_right(int);
+      void set_grid(const Grid &);
 
          //
          //  get stuff