Feature #2609 tc_diag_missing_data (#2680)

Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: Seth Linden <linden@seneca.rap.ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: John and Cindy <halleygotway@Halleys-Mac-mini.local> Co-authored-by: Howard Soh <hsoh@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> Co-authored-by: Seth Linden <linden@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: davidalbo <dave@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Fix Python environment issue (#2407) fix definitions of G172 and G220 based on comments in NOAA-EMC/NCEPLIBS-w3emc#157. (#2406) fix #2380 develop override (#2382) fix #2408 develop empty config (#2410) fix #2390 develop compile zlib (#2404) fix #2412 develop climo (#2422) fix #2437 develop convert (#2439) fix for develop, for #2437, forgot one reference to the search_parent for a dictionary lookup. fix #2452 develop airnow (#2454) fix #2449 develop pdf (#2464) fix #2402 develop sonarqube (#2468) fix #2426 develop buoy (#2475) fix 2518 dtypes appf docs (#2519) fix 2531 compilation errors (#2533) fix #2531 compilation_errors_configure (#2535) fix 2596 main v11.1 rpath compilation (#2614) fix #2514 main_v11.1 clang (#2628)
dtcenter · Sep 13, 2023 · 7063744 · 7063744
1 parent a2f6bbd
commit 7063744
Show file tree

Hide file tree

Showing 21 changed files with 391 additions and 59 deletions.
diff --git a/data/config/TCDiagConfig_default b/data/config/TCDiagConfig_default
@@ -45,9 +45,14 @@ valid_exc = [];
 
 //
 // Subset by the valid hour and lead time.
+// By default, compute diagnostics every 6 hours out to 126 hours.
 //
 valid_hour = [];
-lead       = [];
+lead       = [   "0",   "6",  "12",  "18",  "24",
+                "30",  "36",  "42",  "48",  "54",
+                "60",  "66",  "72",  "78",  "84",
+                "90",  "96", "102", "108", "114",
+               "120", "126" ];
 
 ////////////////////////////////////////////////////////////////////////////////
 
@@ -134,6 +139,11 @@ vortex_removal = FALSE;
 
 ////////////////////////////////////////////////////////////////////////////////
 
+//
+// Flag to control input files
+//
+one_time_per_file_flag = TRUE;
+
 //
 // Flags to control output files
 //

diff --git a/docs/Users_Guide/config_options_tc.rst b/docs/Users_Guide/config_options_tc.rst
@@ -108,7 +108,7 @@ For example:
 valid_beg end inc exc
 ---------------------
 
-Specify a model valid time window YYYYMMDD[_HH[MMSS]] format or provide a
+Specify a model valid time window in YYYYMMDD[_HH[MMSS]] format or provide a
 list of specific valid times to include (inc) or exclude (exc). If a time
 window is specified, only tracks for which all points are contained within
 the window will be used. If valid times to include or exclude are specified,
@@ -1077,7 +1077,7 @@ Statistics will be generated separately for each ATCF ID.
 init_beg, init_end
 ^^^^^^^^^^^^^^^^^^
 
-Forecast and operational initialization time window
+Forecast and operational initialization time window, as strings in YYYYMMDD[_HH[MMSS]] format
 
 .. code-block:: none
 
@@ -1087,7 +1087,7 @@ Forecast and operational initialization time window
 valid_beg, valid_end
 ^^^^^^^^^^^^^^^^^^^^
 
-Forecast, BEST, and operational valid time window
+Forecast, BEST, and operational valid time window, as strings in YYYYMMDD[_HH[MMSS]] format
 
 .. code-block:: none
 		
@@ -1097,7 +1097,7 @@ Forecast, BEST, and operational valid time window
 lead
 ^^^^
 
-Forecast and operational lead times in hours
+Forecast and operational lead times, as strings in HH[MMSS] format
 
 .. code-block:: none
 

diff --git a/docs/Users_Guide/tc-diag.rst b/docs/Users_Guide/tc-diag.rst
@@ -94,12 +94,21 @@ Configuring input tracks and time
   valid_inc = [];
   valid_exc = [];
   valid_hour = [];
-  lead = [];
 
 The TC-Diag tool should be configured to filter the input track data (**-deck**) down to the subset of tracks that correspond to the gridded data files provided (**-data**). The filtered tracks should contain data for only *one initialization time* but may contain tracks for multiple models.
 
 The configuration options listed above are used to filter the input track data down to those that should be processed in the current run. These options are common to multiple MET tools and are described in :numref:`config_options_tc`.
 
+.. code-block:: none
+
+  lead = [   "0",    "6",  "12",  "18",  "24",
+            "30",   "36",  "42",  "48",  "54",
+            "60",   "66",  "72",  "78",  "84",
+            "90",   "96", "102", "108", "114",
+            "120", "126" ];
+
+The **lead** entry is an array of strings specifying lead times in HH[MMSS] format. By default, diagnostics are computed every 6 hours out to 126 hours. Lead times for which no track point or gridded model data exist produce a warning message and diagnostics set to a missing data value.
+
 Configuring domain information
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -204,8 +213,16 @@ The **vortex_removal** flag entry is a boolean specifying whether or not vortex
 
 .. note:: As of MET version 11.1.0, vortex removal logic is not yet supported.
 
-Configuring data output options
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Configuring data input and output options
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: none
+
+  one_time_per_file_flag = TRUE;
+
+The **one_time_per_file_flag** entry controls the logic for reading data from input files. This option describes how data is stored in the gridded input files specified with the **-data** command line option. Set this to true if each input file contains all of the data for a single initialization time and for a single valid time. If the input files contain data for multiple initialization or valid times, or if data for one valid time is spread across multiple files, set this to false.
+
+If true, all input fields are read efficiently from each file in a single call. If false, each field is processed separately in a less efficient manner.
 
 .. code-block:: none
 

diff --git a/internal/test_unit/config/TCDiagConfig_ian b/internal/test_unit/config/TCDiagConfig_ian
@@ -45,9 +45,14 @@ valid_exc = [];
 
 //
 // Subset by the valid hour and lead time.
+// By default, compute diagnostics every 6 hours out to 126 hours.
 //
 valid_hour = [];
-lead       = [ "0", "6", "12", "18", "24" ];
+lead       = [   "0",   "6",  "12",  "18",  "24",
+                "30",  "36",  "42",  "48",  "54",
+                "60",  "66",  "72",  "78",  "84",
+                "90",  "96", "102", "108", "114",
+               "120", "126" ];
 
 ////////////////////////////////////////////////////////////////////////////////
 
@@ -128,6 +133,11 @@ vortex_removal = FALSE;
 
 ////////////////////////////////////////////////////////////////////////////////
 
+//
+// Flag to control input files
+//
+one_time_per_file_flag = TRUE;
+
 //
 // Flags to control output files
 //

diff --git a/src/basic/vx_config/config_constants.h b/src/basic/vx_config/config_constants.h
@@ -1206,18 +1206,19 @@ static const char conf_key_radial_velocity_field_name[] = "radial_velocity_field
 static const char conf_key_tangential_velocity_field_name[] = "tangential_velocity_field_name";
 static const char conf_key_radial_velocity_long_field_name[] = "radial_velocity_long_field_name";
 static const char conf_key_tangential_velocity_long_field_name[] = "tangential_velocity_long_field_name";
-static const char conf_key_vortex_removal[] = "vortex_removal";
 
 //
 // TC-Diag specific parameter key names
 //
 
-static const char conf_key_domain_info[]     = "domain_info";
-static const char conf_key_domain[]          = "domain";
-static const char conf_key_diag_script[]     = "diag_script";
-static const char conf_key_nc_rng_azi_flag[] = "nc_rng_azi_flag";
-static const char conf_key_nc_diag_flag[]    = "nc_diag_flag";
-static const char conf_key_cira_diag_flag[]  = "cira_diag_flag";
+static const char conf_key_domain_info[]            = "domain_info";
+static const char conf_key_domain[]                 = "domain";
+static const char conf_key_diag_script[]            = "diag_script";
+static const char conf_key_vortex_removal[]         = "vortex_removal";
+static const char conf_key_one_time_per_file_flag[] = "one_time_per_file_flag";
+static const char conf_key_nc_rng_azi_flag[]        = "nc_rng_azi_flag";
+static const char conf_key_nc_diag_flag[]           = "nc_diag_flag";
+static const char conf_key_cira_diag_flag[]         = "cira_diag_flag";
 
 //
 // Parameter value names common to multiple tools

diff --git a/src/basic/vx_log/logger.cc b/src/basic/vx_log/logger.cc
@@ -37,8 +37,11 @@ static const int ErrorMessageLevel     = -1;
 
 static const int WarningMessageLevel   =  0;
 
+static const bool DefaultPrintWarning = true;
+
 static const bool DefaultExitOnWarning =  false;
 
+
    //
    //  these need external linkage, do not make static or extern
    //
@@ -217,6 +220,8 @@ LoggerWarning::LoggerWarning()
 {
    warn = WarningMessageLevel;
 
+   PrintWarning = DefaultPrintWarning;
+
    ExitOnWarning = DefaultExitOnWarning;
 
    NeedToExit = false;
@@ -246,6 +251,8 @@ LoggerWarning::LoggerWarning(const LoggerWarning & lw)
 {
    warn = lw.warn;
 
+   PrintWarning = lw.PrintWarning;
+
    ExitOnWarning = lw.ExitOnWarning;
 
    NeedToExit = lw.NeedToExit;
@@ -259,6 +266,8 @@ LoggerWarning & LoggerWarning::operator=(const LoggerWarning & lw)
 {
    warn = lw.warn;
 
+   PrintWarning = lw.PrintWarning;
+
    ExitOnWarning = lw.ExitOnWarning;
 
    NeedToExit = lw.NeedToExit;
@@ -491,6 +500,17 @@ void Logger::dump(ostream & dump_out, int depth) const
 //////////////////////////////////////////////////////////////////
 
 
+bool Logger::print_warning() const
+{
+
+   return (Warning.PrintWarning);
+
+}
+
+
+//////////////////////////////////////////////////////////////////
+
+
 void Logger::set_verbosity_level(const int i)
 {
       //
@@ -509,6 +529,19 @@ void Logger::set_verbosity_level(const int i)
 //////////////////////////////////////////////////////////////////
 
 
+void Logger::set_print_warning(bool b)
+{
+      //
+      // if false, do not print warning messages
+      //
+   Warning.PrintWarning = b;
+
+}
+
+
+//////////////////////////////////////////////////////////////////
+
+
 void Logger::set_exit_on_warning(bool b)
 {
       //
@@ -519,7 +552,6 @@ void Logger::set_exit_on_warning(bool b)
 }
 
 
-
 //////////////////////////////////////////////////////////////////
 
 
@@ -1604,7 +1636,10 @@ Logger & Logger::operator<<(const LoggerError e)
 
 Logger & Logger::operator<<(const LoggerWarning w)
 {  
-   (*this) << level(WarningMessageLevel);
+   if (Warning.PrintWarning)
+   {
+      (*this) << level(WarningMessageLevel);
+   }
 
    if (Warning.ExitOnWarning) Warning.NeedToExit = true;
 

diff --git a/src/basic/vx_log/logger.h b/src/basic/vx_log/logger.h
@@ -115,6 +115,7 @@ class LoggerWarning {
 
    public:
       int warn;
+      bool PrintWarning;
       bool ExitOnWarning;
       bool NeedToExit;
 
@@ -248,6 +249,8 @@ class Logger
 
       int verbosity_level() const;
 
+      bool print_warning() const;
+
       bool is_open() const;
 
          //
@@ -256,6 +259,8 @@ class Logger
 
       void set_verbosity_level(const int);
 
+      void set_print_warning(bool);
+
       void set_exit_on_warning(bool);
 
          //

diff --git a/src/libcode/vx_data2d/data_class.cc b/src/libcode/vx_data2d/data_class.cc
@@ -309,6 +309,49 @@ return;
 ////////////////////////////////////////////////////////////////////////
 
 
+int Met2dDataFile::data_planes(vector<VarInfo*> &vi_list,
+                               vector<DataPlane> &dp_list)
+
+{
+
+int n_valid = 0;
+DataPlane cur_dp;
+
+   //
+   // Loop over requested VarInfo objects
+   //
+
+for(int i=0; i<vi_list.size(); i++) {
+
+   //
+   // Initialize
+   //
+
+   cur_dp.clear();
+
+   //
+   // Read the current DataPlane
+   //
+
+   if(data_plane(*vi_list[i], cur_dp)) n_valid++;
+   else                                cur_dp.clear();
+
+   //
+   // Store the result
+   //
+
+   dp_list.push_back(cur_dp);
+
+}
+
+return ( n_valid );
+
+}
+
+
+////////////////////////////////////////////////////////////////////////
+
+
 bool Met2dDataFile::process_data_plane(VarInfo *vinfo, DataPlane &dp)
 
 {

diff --git a/src/libcode/vx_data2d/data_class.h b/src/libcode/vx_data2d/data_class.h
@@ -153,6 +153,11 @@ class Met2dDataFile : public Met2dData {
 
       virtual int index(VarInfo &) = 0;
 
+         //  retrieve a vector of all requested data planes
+
+      int data_planes(std::vector<VarInfo*>&,
+                      std::vector<DataPlane>&);
+
          //  post-process data after reading it
 
       bool process_data_plane(VarInfo *, DataPlane &);

diff --git a/src/libcode/vx_data2d_grib/data2d_grib.cc b/src/libcode/vx_data2d_grib/data2d_grib.cc
@@ -827,3 +827,5 @@ int MetGrib1DataFile::index(VarInfo &vinfo){
 
    return rec;
 }
+
+////////////////////////////////////////////////////////////////////////
diff --git a/src/libcode/vx_data2d_grib/data2d_grib.h b/src/libcode/vx_data2d_grib/data2d_grib.h
@@ -84,10 +84,16 @@ class MetGrib1DataFile : public Met2dDataFile {
 
       int read_record( VarInfoGrib &);   //  returns match count (>=0), or -1 on error
 
+         //  retrieve the first matching data plane
+
       bool data_plane(VarInfo &, DataPlane &);
 
+         //  retrieve all matching data planes
+
       int data_plane_array(VarInfo &, DataPlaneArray &);
-
+
+         //  retrieve the index of the first matching record
+
       int index(VarInfo &);
 
 };
Original file line number	Diff line number	Diff line change
Expand Up		@@ -827,3 +827,5 @@ int MetGrib1DataFile::index(VarInfo &vinfo){

		return rec;
		}

		////////////////////////////////////////////////////////////////////////