Merge branch 'master' into yohai-ds_scatter

* master:
  typo in whats_new (pydata#2763)
  Update computation.py to use Python 3 function signatures (pydata#2756)
  add h5netcdf+dask tests (pydata#2737)
  Fix name loss when masking (pydata#2749)
  fix datetime_to_numeric and Variable._to_numeric (pydata#2668)
  Fix mypy errors (pydata#2753)
  enable internal plotting with cftime datetime (pydata#2665)
  remove references to cyordereddict (pydata#2750)
  BUG: Pass kwargs to the FileManager for pynio engine (pydata#2380) (pydata#2732)
  reintroduce pynio/rasterio/iris to py36 test env (pydata#2738)
  Fix CRS being WKT instead of PROJ.4 (pydata#2715)
  Refactor (part of) dataset.py to use explicit indexes (pydata#2696)

Author: dcherian

ci/requirements-py36.yml

@@ -20,16 +20,16 @@ dependencies:
   - scipy
   - seaborn
   - toolz
-  # - rasterio  # xref #2683
+  - rasterio
   - bottleneck
   - zarr
   - pseudonetcdf>=3.0.1
   - eccodes
   - cdms2
-# - pynio  # xref #2683
-# - iris>=1.10  # xref #2683
+  - pynio
+  - iris>=1.10
   - pydap
   - lxml
   - pip:
     - cfgrib>=0.9.2
-    - mypy==0.650
+    - mypy==0.660

ci/requirements-py37-windows.yml

@@ -4,6 +4,7 @@ channels:
 dependencies:
   - python=3.7
   - cftime
+  - nc-time-axis
   - dask
   - distributed
   - h5py

ci/requirements-py37.yml

@@ -4,6 +4,7 @@ channels:
 dependencies:
   - python=3.7
   - cftime
+  - nc-time-axis
   - dask
   - distributed
   - h5py

doc/installing.rst

@@ -45,8 +45,6 @@ For accelerating xarray
 - `bottleneck <https://github.com/kwgoodman/bottleneck>`__: speeds up
   NaN-skipping and rolling window aggregations by a large factor
   (1.1 or later)
-- `cyordereddict <https://github.com/shoyer/cyordereddict>`__: speeds up most
-  internal operations with xarray data structures (for python versions < 3.5)
 
 For parallel computing
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -63,6 +61,8 @@ For plotting
   :ref:`plot-maps`
 - `seaborn <https://stanford.edu/~mwaskom/software/seaborn/>`__: for better
   color palettes
+- `nc-time-axis <https://github.com/SciTools/nc-time-axis>`__: for plotting
+  cftime.datetime objects (1.2.0 or later)
 
 
 Instructions
@@ -111,4 +111,4 @@ To run these benchmark tests in a local machine, first install
 - `airspeed-velocity <https://asv.readthedocs.io/en/latest/>`__: a tool for benchmarking Python packages over their lifetime.
 
 and run
-``asv run  # this will install some conda environments in ./.asv/envs``
+``asv run  # this will install some conda environments in ./.asv/envs``

doc/plotting.rst

@@ -24,6 +24,11 @@ Matplotlib syntax and function names were copied as much as possible, which
 makes for an easy transition between the two.
 Matplotlib must be installed before xarray can plot.
 
+To use xarray's plotting capabilities with time coordinates containing
+``cftime.datetime`` objects
+`nc-time-axis <https://github.com/SciTools/nc-time-axis>`_ v1.2.0 or later
+needs to be installed.
+
 For more extensive plotting applications consider the following projects:
 
 - `Seaborn <http://seaborn.pydata.org/>`_: "provides
@@ -237,7 +242,7 @@ It is also possible to make line plots such that the data are on the x-axis and
 ============
 
 As an alternative, also a step plot similar to matplotlib's ``plt.step`` can be
-made using 1D data. 
+made using 1D data.
 
 .. ipython:: python
 
@@ -259,7 +264,7 @@ when plotting data grouped with :py:func:`xarray.Dataset.groupby_bins`.
     plt.ylim(-20,30)
     @savefig plotting_example_step_groupby.png width=4in
     plt.title('Zonal mean temperature')
-    
+
 In this case, the actual boundaries of the bins are used and the ``where`` argument
 is ignored.
 

doc/time-series.rst

@@ -74,7 +74,7 @@ will be used for indexing.  :py:class:`~xarray.CFTimeIndex` enables a subset of
 the indexing functionality of a :py:class:`pandas.DatetimeIndex` and is only
 fully compatible with the standalone version of ``cftime`` (not the version
 packaged with earlier versions ``netCDF4``).  See :ref:`CFTimeIndex` for more
-information. 
+information.
 
 Datetime indexing
 -----------------
@@ -215,7 +215,7 @@ For more examples of using grouped operations on a time dimension, see
 
 
 .. _CFTimeIndex:
-     
+
 Non-standard calendars and dates outside the Timestamp-valid range
 ------------------------------------------------------------------
 
@@ -224,14 +224,14 @@ Through the standalone ``cftime`` library and a custom subclass of
 functionality enabled through the standard :py:class:`pandas.DatetimeIndex` for
 dates from non-standard calendars commonly used in climate science or dates
 using a standard calendar, but outside the `Timestamp-valid range`_
-(approximately between years 1678 and 2262).  
+(approximately between years 1678 and 2262).
 
 .. note::
 
    As of xarray version 0.11, by default, :py:class:`cftime.datetime` objects
    will be used to represent times (either in indexes, as a
-   :py:class:`~xarray.CFTimeIndex`, or in data arrays with dtype object) if 
-   any of the following are true: 
+   :py:class:`~xarray.CFTimeIndex`, or in data arrays with dtype object) if
+   any of the following are true:
 
    - The dates are from a non-standard calendar
    - Any dates are outside the Timestamp-valid range.
@@ -252,7 +252,7 @@ coordinate with dates from a no-leap calendar and a
    dates = [DatetimeNoLeap(year, month, 1) for year, month in
             product(range(1, 3), range(1, 13))]
    da = xr.DataArray(np.arange(24), coords=[dates], dims=['time'], name='foo')
-                         
+
 xarray also includes a :py:func:`~xarray.cftime_range` function, which enables
 creating a :py:class:`~xarray.CFTimeIndex` with regularly-spaced dates.  For
 instance, we can create the same dates and DataArray we created above using:
@@ -261,20 +261,20 @@ instance, we can create the same dates and DataArray we created above using:
 
    dates = xr.cftime_range(start='0001', periods=24, freq='MS', calendar='noleap')
    da = xr.DataArray(np.arange(24), coords=[dates], dims=['time'], name='foo')
-   
+
 For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports:
 
 - `Partial datetime string indexing`_ using strictly `ISO 8601-format`_ partial
   datetime strings:
-  
+
 .. ipython:: python
 
    da.sel(time='0001')
    da.sel(time=slice('0001-05', '0002-02'))
 
 - Access of basic datetime components via the ``dt`` accessor (in this case
   just "year", "month", "day", "hour", "minute", "second", "microsecond",
-  "season", "dayofyear", and "dayofweek"): 
+  "season", "dayofyear", and "dayofweek"):
 
 .. ipython:: python
 
@@ -323,14 +323,7 @@ For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports:
     da.resample(time='81T', closed='right', label='right', base=3).mean()
 
 .. note::
-   
-   While much of the time series functionality that is possible for standard
-   dates has been implemented for dates from non-standard calendars, there are
-   still some remaining important features that have yet to be implemented,
-   for example:
-
-   - Built-in plotting of data with :py:class:`cftime.datetime` coordinate axes
-     (:issue:`2164`).   
+    
 
    For some use-cases it may still be useful to convert from
    a :py:class:`~xarray.CFTimeIndex` to a :py:class:`pandas.DatetimeIndex`,
@@ -351,8 +344,8 @@ For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports:
    do not depend on differences between dates (e.g. differentiation,
    interpolation, or upsampling with resample), as these could introduce subtle
    and silent errors due to the difference in calendar types between the dates
-   encoded in your data and the dates stored in memory.  
-  
+   encoded in your data and the dates stored in memory.
+
 .. _Timestamp-valid range: https://pandas.pydata.org/pandas-docs/stable/timeseries.html#timestamp-limitations
 .. _ISO 8601-format: https://en.wikipedia.org/wiki/ISO_8601
 .. _partial datetime string indexing: https://pandas.pydata.org/pandas-docs/stable/timeseries.html#partial-string-indexing

doc/whats-new.rst

@@ -24,14 +24,20 @@ Breaking changes
 - Remove support for Python 2. This is the first version of xarray that is
   Python 3 only. (:issue:`1876`).
   By `Joe Hamman <https://github.com/jhamman>`_.
-- The `compat` argument to `Dataset` and the `encoding` argument to 
+- The `compat` argument to `Dataset` and the `encoding` argument to
   `DataArray` are deprecated and will be removed in a future release.
   (:issue:`1188`)
   By `Maximilian Roos <https://github.com/max-sixty>`_.
+- `cyordereddict` is no longer used as an optional dependency (:issue:`2744`).
+  By `Joe Hamman <https://github.com/jhamman>`_.
 
 Enhancements
 ~~~~~~~~~~~~
 
+- Internal plotting now supports ``cftime.datetime`` objects as time series.
+  (:issue:`2164`)
+  By `Julius Busecke <https://github.com/jbusecke>`_ and
+  `Spencer Clark <https://github.com/spencerkclark>`_.
 - Add ``data=False`` option to ``to_dict()`` methods. (:issue:`2656`)
   By `Ryan Abernathey <https://github.com/rabernat>`_
 - :py:meth:`~xarray.DataArray.coarsen` and
@@ -81,7 +87,15 @@ Bug fixes
   :py:class:`CFTimeIndex` now results in a :py:class:`pandas.TimedeltaIndex`
   instead of raising a ``TypeError`` (:issue:`2671`).  By `Spencer Clark
   <https://github.com/spencerkclark>`_.
-
+- backend_kwargs are no longer ignored when using open_dataset with pynio engine
+  (:issue:'2380')
+  By 'Jonathan Joyce <https://github.com/jonmjoyce>'_.
+- Fix ``open_rasterio`` creating a WKT CRS instead of PROJ.4 with
+  ``rasterio`` 1.0.14+ (:issue:`2715`).
+  By `David Hoese <https://github.com/djhoese>`_.
+- Masking data arrays with :py:meth:`xarray.DataArray.where` now returns an
+  array with the name of the original masked array (:issue:`2748` and :issue:`2457`).
+  By `Yohai Bar-Sinai <https://github.com/yohai>`_.
 .. _whats-new.0.11.3:
 
 v0.11.3 (26 January 2019)

setup.cfg

@@ -45,6 +45,8 @@ ignore_missing_imports = True
 ignore_missing_imports = True
 [mypy-Nio.*]
 ignore_missing_imports = True
+[mypy-nc_time_axis.*]
+ignore_missing_imports = True
 [mypy-numpy.*]
 ignore_missing_imports = True
 [mypy-netCDF4.*]

xarray/backends/pynio_.py

@@ -45,13 +45,13 @@ class NioDataStore(AbstractDataStore):
     """Store for accessing datasets via PyNIO
     """
 
-    def __init__(self, filename, mode='r', lock=None):
+    def __init__(self, filename, mode='r', lock=None, **kwargs):
         import Nio
         if lock is None:
             lock = PYNIO_LOCK
         self.lock = ensure_lock(lock)
         self._manager = CachingFileManager(
-            Nio.open_file, filename, lock=lock, mode=mode)
+            Nio.open_file, filename, lock=lock, mode=mode, kwargs=kwargs)
         # xarray provides its own support for FillValue,
         # so turn off PyNIO's support for the same.
         self.ds.set_option('MaskedArrayMode', 'MaskedNever')

xarray/backends/rasterio_.py

@@ -282,7 +282,10 @@ def open_rasterio(filename, parse_coordinates=None, chunks=None, cache=None,
         # CRS is a dict-like object specific to rasterio
         # If CRS is not None, we convert it back to a PROJ4 string using
         # rasterio itself
-        attrs['crs'] = riods.crs.to_string()
+        try:
+            attrs['crs'] = riods.crs.to_proj4()
+        except AttributeError:
+            attrs['crs'] = riods.crs.to_string()
     if hasattr(riods, 'res'):
         # (width, height) tuple of pixels in units of CRS
         attrs['res'] = riods.res

xarray/core/alignment.py

@@ -3,13 +3,15 @@
 import warnings
 from collections import OrderedDict, defaultdict
 from contextlib import suppress
+from typing import Any, Mapping, Optional, Tuple
 
 import numpy as np
+import pandas as pd
 
 from . import utils
 from .indexing import get_indexer_nd
 from .utils import is_dict_like, is_full_slice
-from .variable import IndexVariable
+from .variable import IndexVariable, Variable
 
 
 def _get_joiner(join):
@@ -260,8 +262,15 @@ def reindex_like_indexers(target, other):
     return indexers
 
 
-def reindex_variables(variables, sizes, indexes, indexers, method=None,
-                      tolerance=None, copy=True):
+def reindex_variables(
+    variables: Mapping[Any, Variable],
+    sizes: Mapping[Any, int],
+    indexes: Mapping[Any, pd.Index],
+    indexers: Mapping,
+    method: Optional[str] = None,
+    tolerance: Any = None,
+    copy: bool = True,
+) -> 'Tuple[OrderedDict[Any, Variable], OrderedDict[Any, pd.Index]]':
     """Conform a dictionary of aligned variables onto a new set of variables,
     filling in missing values with NaN.
 
@@ -274,7 +283,7 @@ def reindex_variables(variables, sizes, indexes, indexers, method=None,
     sizes : dict-like
         Dictionary from dimension names to integer sizes.
     indexes : dict-like
-        Dictionary of xarray.IndexVariable objects associated with variables.
+        Dictionary of indexes associated with variables.
     indexers : dict
         Dictionary with keys given by dimension names and values given by
         arrays of coordinates tick labels. Any mis-matched coordinate values
@@ -300,13 +309,15 @@ def reindex_variables(variables, sizes, indexes, indexers, method=None,
     Returns
     -------
     reindexed : OrderedDict
-        Another dict, with the items in variables but replaced indexes.
+        Dict of reindexed variables.
+    new_indexes : OrderedDict
+        Dict of indexes associated with the reindexed variables.
     """
     from .dataarray import DataArray
 
     # build up indexers for assignment along each dimension
     int_indexers = {}
-    targets = {}
+    targets = OrderedDict()  # type: OrderedDict[Any, pd.Index]
     masked_dims = set()
     unchanged_dims = set()
 
@@ -346,7 +357,7 @@ def reindex_variables(variables, sizes, indexes, indexers, method=None,
                     'the new index %r' % (dim, existing_size, new_size))
 
     # create variables for the new dataset
-    reindexed = OrderedDict()
+    reindexed = OrderedDict()  # type: OrderedDict[Any, Variable]
 
     for dim, indexer in indexers.items():
         if isinstance(indexer, DataArray) and indexer.dims != (dim,):
@@ -359,7 +370,7 @@ def reindex_variables(variables, sizes, indexes, indexers, method=None,
 
         if dim in variables:
             var = variables[dim]
-            args = (var.attrs, var.encoding)
+            args = (var.attrs, var.encoding)  # type: tuple
         else:
             args = ()
         reindexed[dim] = IndexVariable((dim,), indexers[dim], *args)
@@ -384,7 +395,10 @@ def reindex_variables(variables, sizes, indexes, indexers, method=None,
 
             reindexed[name] = new_var
 
-    return reindexed
+    new_indexes = OrderedDict(indexes)
+    new_indexes.update(targets)
+
+    return reindexed, new_indexes
 
 
 def broadcast(*args, **kwargs):

xarray/core/common.py

@@ -713,7 +713,7 @@ def resample(self, indexer=None, skipna=None, closed=None, label=None,
         array([ 0.      ,  0.032258,  0.064516, ..., 10.935484, 10.967742, 11.      ])
         Coordinates:
           * time     (time) datetime64[ns] 1999-12-15 1999-12-16 1999-12-17 ...
-        
+
         Limit scope of upsampling method
         >>> da.resample(time='1D').nearest(tolerance='1D')
         <xarray.DataArray (time: 337)>