Update develop-ref after #1704 and #1708 (#1712)

* Per #1429, enhance error message from DataLine::get_item(). (#1682)

* Feature 1429 tc_log second try (#1686)

* Per #1429, enhance error message from DataLine::get_item().

* Per #1429, I realize that the line number actually is readily available in the DataLine class... so include it in the error message.

* Feature 1588 ps_log (#1687)

* Per #1588, updated pair_data_point.h/.cc to add detailed Debug(4) log messages, as specified in the GitHub issue. Do still need to test each of these cases to confirm that the log messages look good.

* Per #1588, switch very detailed interpolation details from debug level 4 to 5.

* Per #1588, remove the Debug(4) log message about duplicate obs since it's been moved up to a higher level.

* Per #1588, add/update detailed log messages when processing point observations for bad data, off the grid, bad topo, big topo diffs, bad fcst value, and duplicate obs.

* #1454 Disabled plot_data_plane_CESM_SSMI_microwave and plot_data_plane_CESM_sea_ice_nc becaues of not evenly spaced

* #1454 Moved NC attribute name to nc_utils.h

* #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance

* #1454 Corrected sanity checking for lat/lon projection based on the percentage of the delta instead of fixed tolerance

* #1454 Corrected data.delta_lon

* #1454 Change bact to use diff instead of absolute value of diff

* 454 Deleted instea dof commenting out

* 454 Deleted instea dof commenting out

* Feature 1684 bss and 1685 single reference model (#1689)

* Per #1684, move an instance of the ClimoCDFInfo class into PairBase. Also define derive_climo_vals() and derive_climo_prob() utility functions.

* Add to VxPairDataPoint and VxPairDataEnsemble functions to set the ClimoCDFInfo class.

* Per #1684, update ensemble_stat and point_stat to set the ClimoCDFInfo object based on the contents of the config file.

* Per #1684, update the vx_statistics library and stat_analysis to make calls to the new derive_climo_vals() and derive_climo_prob() functions.

* Per #1684, since cdf_info is a member of PairBase class, need to handle it in the PairDataPoint and PairDataEnsemble assignment and subsetting logic.

* Per #1684, during development, I ran across and then updated this log message.

* Per #1684, working on log messages and figured that the regridding climo data should be moved from Debug(1) to at least Debug(2).

* Per #1684 and #1685, update the logic for the derive_climo_vals() utility function. If only a single climo bin is requested, just return the climo mean. Otherwise, sample the requested number of values.

* Per #1684, just fixing the format of this log message.

* Per #1684, add a STATLine::get_offset() member function.

* Per #1684, update parse_orank_line() logic. Rather than calling NumArray::clear() call NumArray::erase() to preserve allocated memory. Also, instead of parsing ensemble member values by column name, parse them by offset number.

* Per #1684, call EnsemblePairData::extend() when parsing ORANK data to allocate one block of memory instead of bunches of litte ones.

* Per #1684 and #1685, add another call to Ensemble-Stat to test computing the CRPSCL_EMP from a single climo mean instead of using the full climo distribution.

* Per #1684 and #1685, update ensemble-stat docs about computing CRPSS_EMP relative to a single reference model.

* Per #1684, need to update Grid-Stat to store the climo cdf info in the PairDataPoint objects.

* Per #1684, remove debug print statements.

* Per #1684, need to set cdf_info when aggregating MPR lines in Stat-Analysis.

* Per #1684 and #1685, update PairDataEnsemble::compute_pair_vals() to print a log message indicating the climo data being used as reference:

For a climo distribution defined by mean and stdev:
DEBUG 3: Computing ensemble statistics relative to a 9-member climatological ensemble.

For a single deterministic reference:
DEBUG 3: Computing ensemble statistics relative to the climatological mean.

* Per #1691, add met-10.0.0-beta4 release notes. (#1692)

* Updated Python documentation

* Per #1694, add VarInfo::magic_str_attr() to construct a field summary string from the name_attr() and level_attr() functions.

* Per #1694, fixing 2 issues here. There was a bug in the computation of the max value. Had a less-than sign that should have been greater-than. Also, switch from tracking data by it's magic_str() to simply using VAR_i and VAR_j strings. We *could* have just used the i, j integers directly, but constructing the ij joint histogram integer could have been tricky since we start numbering with 0 instead of 1. i=0, j=1 would result in 01 which is the same as integer of 1. If we do want to switch to integers, we just need to make them 1-based and add +1 all over the place.

* Per #1694, just switching to consistent variable name.

* Just consistent spacing.

* Adding files for ReadTheDocs

* Adding .yaml file for ReadTheDocs

* Updated path to requirements.txt file

* Updated path to conf.py file

* Removing ReadTheDocs files and working in separate branch

* Trying different options for formats (#1702)

* Per #1706, add bugfix to the develop branch. Also add a new job to unit_stat_analysis.xml to test out the aggregation of the ECNT line type. This will add new unit test output and cause the NB to fail. (#1708)

* Feature 1471 python_grid (#1704)

* Per #1471, defined a parse_grid_string() function in the vx_statistics library and then updated vx_data2d_python to call that function. However, this creates a circular dependency because vx_data2d_python now depends on vx_statistics.

* Per #1471, because of the change in dependencies, I had to modify many, many Makefile.am files to link to the -lvx_statistics after -lvx_data2d_python. This is not great, but I didn't find a better solution.

* Per #1471, add a sanity check to make sure the grid and data dimensions actually match.

* Per #1471, add 3 new unit tests to demonstrate setting the python grid as a named grid, grid specification string, or a gridded data file.

* Per #1471, document python grid changes in appendix F.

* Per #1471, just spacing.

* Per #1471, lots of Makefile.am changes to get this code to compile on kiowa. Worringly, it compiled and linked fine on my Mac laptop but not on kiowa. Must be some large differences in the linker logic.

Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu>

* Committing a fix for unit_python.xml directly to the develop branch. We referenced  in a place where it's not defined.

* Add *.dSYM to the .gitignore files in the src and internal_tests directories.

Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu>
Co-authored-by: hsoh-u <hsoh@ucar.edu>
Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu>
Co-authored-by: David Fillmore <davidfillmore@users.noreply.github.com>
Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu>
Co-authored-by: MET Tools Test Account <met_test@kiowa.rap.ucar.edu>

.readthedocs.yaml

@@ -0,0 +1,23 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required 
+version: 2
+
+# Build all formats (htmlzip, pdf, epub)
+#formats: all
+formats: []
+
+# Optionally set the version of Python and requirements required to build your
+# docs
+python:
+   version: 3.7
+   install:
+   - requirements: met/docs/requirements.txt
+
+# Configuration for Sphinx documentation (this is the default documentation
+# type)
+sphinx:
+  builder: html
+  configuration: met/docs/conf.py

met/docs/Users_Guide/appendixF.rst

@@ -65,7 +65,9 @@ The data must be loaded into a 2D NumPy array named **met_data**. In addition th
      'long_name': 'FooBar',
      'level':     'Surface',
      'units':     'None',
-  
+ 
+     # Define 'grid' as a string or a dictionary
+ 
      'grid': {
         'type': 'Lambert Conformal',
         'hemisphere': 'N',
@@ -83,12 +85,32 @@ The data must be loaded into a 2D NumPy array named **met_data**. In addition th
         'ny': 129,
       }
   
-     }
+  }
+
+
+In the dictionary, valid time, initialization time, lead time and accumulation time (if any) must be indicated by strings. Valid and initialization times must be given in YYYYMMDD[_HH[MMSS]] format, and lead and accumulation times must be given in HH[MMSS] format, where the square brackets indicate optional elements. The dictionary must also include strings for the name, long_name, level, and units to describe the data. The rest of the **attrs** dictionary gives the grid size and projection information in the same format that is used in the netCDF files written out by the MET tools. Those entries are also listed below. Note that the **grid** entry in the **attrs** dictionary can either be defined as a string or as a dictionary itself.
+
+If specified as a string, **grid** can be defined as follows:
+
+• As a named grid:
+
+.. code-block:: none
 
+  'grid': 'G212'
+
+• As a grid specification string, as described in :ref:`appendixB`:
+
+.. code-block:: none
+
+  'grid': 'lambert 185 129 12.19 -133.459 -95 40.635 6371.2 25 25 N'
+
+• As the path to an existing gridded data file:
+
+.. code-block:: none
 
-In the dictionary, valid time, initialization time, lead time and accumulation time (if any) must be indicated by strings. Valid and initialization times must be given in YYYYMMDD[_HH[MMSS]] format, and lead and accumulation times must be given in HH[MMSS] format, where the square brackets indicate optional elements. The dictionary must also include strings for the name, long_name, level, and units to describe the data. The rest of the **attrs** dictionary gives the grid size and projection information in the same format that is used in the netCDF files written out by the MET tools. Those entries are also listed below. Note that the **grid** entry in the **attrs** dictionary is itself a dictionary.
+  'grid': '/path/to/sample_data.grib'
 
-The supported grid **type** strings are described below:
+When specified as a dictionary, the contents of the **grid** dictionary vary based on the grid **type** string. The entries for the supported grid types are described below:
 
 • **Lambert Conformal** grid dictionary entries:
 

met/docs/requirements.txt

@@ -1,11 +1,10 @@
-sphinx
-sphinx-gallery
-sphinx-rtd-theme
-sphinxcontrib-applehelp
+sphinx==2.4.4
+sphinx-gallery==0.7.0
+sphinx-rtd-theme==0.4.3
+sphinxcontrib-applehelp==1.0.2
 sphinxcontrib-bibtex
-sphinxcontrib-devhelp
-sphinxcontrib-htmlhelp
-sphinxcontrib-jsmath
-sphinxcontrib-qthelp
-sphinxcontrib-serializinghtml
-
+sphinxcontrib-devhelp==1.0.2
+sphinxcontrib-htmlhelp==1.0.3
+sphinxcontrib-jsmath==1.0.1
+sphinxcontrib-qthelp==1.0.3
+sphinxcontrib-serializinghtml==1.1.4

met/internal_tests/.gitignore

@@ -3,3 +3,4 @@
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/basic/.gitignore

@@ -3,3 +3,4 @@
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/basic/vx_config/.gitignore

@@ -12,3 +12,4 @@ test_config
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/basic/vx_log/.gitignore

@@ -5,3 +5,4 @@ test_logger
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/basic/vx_util/.gitignore

@@ -8,3 +8,4 @@ test_add_rows
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/basic/vx_util/Makefile.am

@@ -82,18 +82,30 @@ endif
 test_ascii_header_SOURCES = test_ascii_header.cc
 test_ascii_header_CPPFLAGS = ${MET_CPPFLAGS}
 test_ascii_header_LDFLAGS = -L. ${MET_LDFLAGS}
-test_ascii_header_LDADD = -lvx_util \
+test_ascii_header_LDADD = -lvx_stat_out \
+	-lvx_statistics \
+	-lvx_shapedata \
+	-lvx_gsl_prob \
+	-lvx_analysis_util \
+	-lvx_shapedata \
+	-lvx_util \
+	$(PYTHON_LIBS) \
+	-lvx_statistics \
+	-lvx_data2d_factory \
+	-lvx_data2d_nc_met \
+	-lvx_data2d_nc_pinterp \
+	$(PYTHON_LIBS) \
+	-lvx_data2d_nccf \
+	-lvx_data2d_grib $(GRIB2_LIBS) \
+	-lvx_data2d \
+	-lvx_nc_util \
+	-lvx_regrid \
+	-lvx_grid \
 	-lvx_config \
 	-lvx_gsl_prob \
 	-lvx_cal \
 	-lvx_util \
 	-lvx_math \
+	-lvx_color \
 	-lvx_log \
-	-lgsl -lgslcblas
-
-if ENABLE_PYTHON
-test_ascii_header_LDADD += $(MET_PYTHON_LD) -lvx_data2d_python -lvx_python3_utils -lvx_data2d_python -lvx_python3_utils -lvx_math 
-test_ascii_header_LDADD += -lvx_data2d_python -lvx_python3_utils -lvx_data2d_python -lvx_python3_utils
-test_ascii_header_LDADD += -lvx_math -lvx_grid -lvx_util -lvx_data2d -lvx_config -lvx_gsl_prob -lvx_cal -lvx_math -lvx_util
-endif
-
+	-lm -lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas -lvx_util

met/internal_tests/libcode/.gitignore

@@ -3,3 +3,4 @@
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_data2d/.gitignore

@@ -5,3 +5,4 @@ dump_default_table
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_data2d_factory/.gitignore

@@ -5,3 +5,4 @@ test_factory
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_data2d_factory/Makefile.am

@@ -25,20 +25,27 @@ test_is_grib_LDADD = -lvx_data2d_factory \
 test_factory_SOURCES = test_factory.cc
 test_factory_CPPFLAGS = ${MET_CPPFLAGS}
 test_factory_LDFLAGS = -L. ${MET_LDFLAGS}
-test_factory_LDADD = -lvx_data2d_factory \
+test_factory_LDADD = -lvx_stat_out \
+	-lvx_statistics \
+	-lvx_shapedata \
+	-lvx_gsl_prob \
+	-lvx_analysis_util \
+	-lvx_data2d_factory \
 	-lvx_data2d_nc_met \
 	-lvx_data2d_grib $(GRIB2_LIBS) \
 	-lvx_data2d_nc_pinterp \
 	$(PYTHON_LIBS) \
 	-lvx_data2d_nccf \
+	-lvx_statistics \
 	-lvx_data2d \
 	-lvx_nc_util \
+	-lvx_regrid \
 	-lvx_grid \
 	-lvx_config \
 	-lvx_gsl_prob \
+	-lvx_cal \
 	-lvx_util \
 	-lvx_math \
 	-lvx_color \
-	-lvx_cal \
 	-lvx_log \
 	-lm -lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas

met/internal_tests/libcode/vx_data2d_grib/.gitignore

@@ -4,3 +4,4 @@ test_read_grib1
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_data2d_nc_met/.gitignore

@@ -4,3 +4,4 @@ test_read_nc_met
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_data2d_nccf/.gitignore

@@ -4,3 +4,4 @@ test_read_nccf
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_geodesy/.gitignore

@@ -4,3 +4,4 @@ test_spheroid
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_grid/.gitignore

@@ -4,3 +4,4 @@ test_grid_area
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_nc_util/.gitignore

@@ -4,3 +4,4 @@ test_pressure_levels
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_nc_util/Makefile.am

@@ -19,21 +19,32 @@ noinst_PROGRAMS = test_pressure_levels
 test_pressure_levels_SOURCES = test_pressure_levels.cc
 test_pressure_levels_CPPFLAGS = ${MET_CPPFLAGS}
 test_pressure_levels_LDFLAGS = -L. ${MET_LDFLAGS}
-test_pressure_levels_LDADD = -lvx_tc_util \
+test_pressure_levels_LDADD = -lvx_stat_out \
+	-lvx_statistics \
+	-lvx_shapedata \
+	-lvx_gsl_prob \
+	-lvx_analysis_util \
+	-lvx_tc_util \
+	-lvx_shapedata \
+	-lvx_util \
+	$(PYTHON_LIBS) \
+	-lvx_statistics \
+	-lvx_data2d_factory \
+	-lvx_data2d_nc_met \
+	-lvx_data2d_nc_pinterp \
+	$(PYTHON_LIBS) \
+	-lvx_data2d_nccf \
+	-lvx_data2d_grib $(GRIB2_LIBS) \
 	-lvx_data2d \
 	-lvx_nc_util \
+	-lvx_regrid \
+	-lvx_grid \
 	-lvx_config \
 	-lvx_gsl_prob \
 	-lvx_cal \
 	-lvx_util \
 	-lvx_math \
+	-lvx_color \
 	-lvx_log \
-	-lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas
+	-lm -lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas -lvx_util
 
-if ENABLE_PYTHON
-test_pressure_levels_LDADD += $(MET_PYTHON_LD)
-test_pressure_levels_LDADD += -lvx_data2d_python -lvx_python3_utils
-test_pressure_levels_LDADD += -lvx_data2d_python -lvx_python3_utils
-test_pressure_levels_LDADD += -lvx_grid -lvx_util -lvx_config
-test_pressure_levels_LDADD += -lvx_data2d -lvx_gsl_prob -lvx_util -lvx_math -lvx_cal -lvx_config
-endif

met/internal_tests/libcode/vx_physics/.gitignore

@@ -4,3 +4,4 @@ test_thermo
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_plot_util/.gitignore

@@ -4,3 +4,4 @@ test_map_region
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_ps/.gitignore

@@ -4,3 +4,4 @@ test_ps
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_series_data/.gitignore

@@ -4,3 +4,4 @@ test_series_data
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_solar/.gitignore

@@ -4,3 +4,4 @@ test_ra_dec
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_tc_util/.gitignore

@@ -5,3 +5,4 @@ test_read
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/libcode/vx_tc_util/Makefile.am

@@ -20,39 +20,64 @@ noinst_PROGRAMS = test_read \
 test_read_SOURCES = test_read.cc
 test_read_CPPFLAGS = ${MET_CPPFLAGS}
 test_read_LDFLAGS = -L. ${MET_LDFLAGS}
-test_read_LDADD = -lvx_tc_util \
+test_read_LDADD = -lvx_stat_out \
+	-lvx_statistics \
+	-lvx_shapedata \
+	-lvx_gsl_prob \
+	-lvx_analysis_util \
+	-lvx_tc_util \
+	-lvx_shapedata \
+	-lvx_util \
+	$(PYTHON_LIBS) \
+	-lvx_statistics \
+	-lvx_data2d_factory \
+	-lvx_data2d_nc_met \
+	-lvx_data2d_nc_pinterp \
+	$(PYTHON_LIBS) \
+	-lvx_data2d_nccf \
+	-lvx_data2d_grib $(GRIB2_LIBS) \
+	-lvx_data2d \
+	-lvx_nc_util \
+	-lvx_regrid \
+	-lvx_grid \
 	-lvx_config \
 	-lvx_gsl_prob \
 	-lvx_cal \
 	-lvx_util \
 	-lvx_math \
+	-lvx_color \
 	-lvx_log \
-	-lgsl -lgslcblas
-
-if ENABLE_PYTHON
-test_read_LDADD += $(MET_PYTHON_LD)
-test_read_LDADD += -lvx_data2d_python -lvx_python3_utils
-test_read_LDADD += -lvx_data2d_python -lvx_python3_utils
-test_read_LDADD += -lvx_grid -lvx_util -lvx_config
-test_read_LDADD += -lvx_data2d -lvx_gsl_prob -lvx_util -lvx_math -lvx_cal -lvx_config
-endif
+	-lm -lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas -lvx_util
 
 test_read_prob_SOURCES = test_read_prob.cc
 test_read_prob_CPPFLAGS = ${MET_CPPFLAGS}
 test_read_prob_LDFLAGS = -L. ${MET_LDFLAGS}
-test_read_prob_LDADD = -lvx_tc_util \
+test_read_prob_LDADD = -lvx_stat_out \
+	-lvx_statistics \
+	-lvx_shapedata \
+	-lvx_gsl_prob \
+	-lvx_analysis_util \
+	-lvx_tc_util \
+	-lvx_shapedata \
+	-lvx_util \
+	$(PYTHON_LIBS) \
+	-lvx_statistics \
+	-lvx_data2d_factory \
+	-lvx_data2d_nc_met \
+	-lvx_data2d_nc_pinterp \
+	$(PYTHON_LIBS) \
+	-lvx_data2d_nccf \
+	-lvx_data2d_grib $(GRIB2_LIBS) \
+	-lvx_data2d \
+	-lvx_nc_util \
+	-lvx_regrid \
+	-lvx_grid \
 	-lvx_config \
 	-lvx_gsl_prob \
 	-lvx_cal \
 	-lvx_util \
 	-lvx_math \
+	-lvx_color \
 	-lvx_log \
-	-lgsl -lgslcblas
+	-lm -lnetcdf_c++4 -lnetcdf -lgsl -lgslcblas -lvx_util
 
-if ENABLE_PYTHON
-test_read_prob_LDADD += $(MET_PYTHON_LD)
-test_read_prob_LDADD += -lvx_data2d_python -lvx_python3_utils
-test_read_prob_LDADD += -lvx_data2d_python -lvx_python3_utils
-test_read_prob_LDADD += -lvx_grid -lvx_util -lvx_config
-test_read_prob_LDADD += -lvx_data2d -lvx_gsl_prob -lvx_util -lvx_math -lvx_cal -lvx_config
-endif

met/internal_tests/tools/.gitignore

@@ -3,3 +3,4 @@
 .deps
 Makefile
 Makefile.in
+*.dSYM

met/internal_tests/tools/other/.gitignore

@@ -3,3 +3,4 @@
 .deps
 Makefile
 Makefile.in
+*.dSYM