Marks slow, flaky, and failing tests (#1336)

* Adds marked tests as slow, flakey, and optionalci

Tests are marked using the following py.test options:

 * @slow which is run on default, not run if `--skip-slow`.
 * @optionalci is run on default, not run if `--skip-optional-ci`
 * @flakey which is not run on default, run if `--run-flakey`

Note, the `optionalci` category is needed in order to mark
tests that are environment dependent when run on continuous
integration (CI) hardware, e.g., tests that fail due to
variable loads on travis CI.

* Spelling: flakey -> flaky

* Test suite adjustments

* Typos

* Remove bottleneck from most CI builds

* xfail for cross engine read write netcdf4

* Try fix: Closes open file (for Windows error)

```
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process
```

* Try fix: Closes open file (for Windows error)

```
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process
```

Author: pwolfram

.travis.yml

@@ -20,6 +20,10 @@ matrix:
     env: CONDA_ENV=py35
   - python: 3.6
     env: CONDA_ENV=py36
+  - python: 3.6
+    env:
+    - CONDA_ENV=py36
+    - EXTRA_FLAGS="--run-flaky"
   - python: 3.6
     env: CONDA_ENV=py36-pydap
   - python: 3.6
@@ -36,6 +40,10 @@ matrix:
   - python: 3.6
     env: CONDA_ENV=py36-condaforge-rc
   allow_failures:
+  - python: 3.6
+    env:
+    - CONDA_ENV=py36
+    - EXTRA_FLAGS="--run-flaky"
   - python: 3.6
     env: CONDA_ENV=py36-pydap
   - python: 3.6
@@ -76,7 +84,7 @@ install:
   - python setup.py install
 
 script:
-  - py.test xarray --cov=xarray --cov-report term-missing --verbose
+  - py.test xarray --cov=xarray --cov-report term-missing --verbose $EXTRA_FLAGS
 
 after_success:
   - coveralls

ci/requirements-py27-cdat+pynio.yml

@@ -4,19 +4,20 @@ channels:
 dependencies:
   - python=2.7
   - cdat-lite
+  - cyordereddict
   - dask
   - distributed
-  - pytest
+  - h5py
+  - h5netcdf
+  - matplotlib
+  - netcdf4
   - numpy
-  - pandas>=0.15.0
+  - pandas
   - pynio
-  - netcdf4
+  - pytest
   - scipy
-  - pytest-cov
-  - cyordereddict
-  - h5py
-  - netcdf4
-  - matplotlib
   - seaborn
+  - toolz
   - pip:
     - coveralls
+    - pytest-cov

ci/requirements-py27-windows.yml

@@ -4,10 +4,14 @@ channels:
 dependencies:
   - python=2.7
   - dask
+  - distributed
+  - h5py
+  - h5netcdf
   - matplotlib
-  - netCDF4
+  - netcdf4
+  - pytest
   - numpy
   - pandas
-  - pip
-  - pytest
   - scipy
+  - seaborn
+  - toolz

ci/requirements-py35.yml

@@ -3,18 +3,18 @@ channels:
   - conda-forge
 dependencies:
   - python=3.5
-  - cython
   - dask
   - distributed
   - h5py
+  - h5netcdf
   - matplotlib
   - netcdf4
   - pytest
   - numpy
   - pandas
-  - seaborn
   - scipy
+  - seaborn
+  - toolz
   - pip:
     - coveralls
     - pytest-cov
-    - h5netcdf

ci/requirements-py36-condaforge-rc.yml

@@ -4,18 +4,18 @@ channels:
   - conda-forge
 dependencies:
   - python=3.6
-  - cython
   - dask
   - distributed
   - h5py
+  - h5netcdf
   - matplotlib
   - netcdf4
   - pytest
   - numpy
   - pandas
   - seaborn
   - scipy
+  - toolz
   - pip:
     - coveralls
     - pytest-cov
-    - h5netcdf

ci/requirements-py36-dask-dev.yml

@@ -3,13 +3,18 @@ channels:
   - conda-forge
 dependencies:
   - python=3.6
-  - cython
+  - h5py
+  - h5netcdf
+  - matplotlib
+  - netcdf4
   - pytest
   - numpy
   - pandas
+  - seaborn
   - scipy
   - toolz
   - pip:
     - coveralls
     - pytest-cov
-    - git+https://github.com/blaze/dask.git
+    - git+https://github.com/dask/dask.git
+    - git+https://github.com/dask/distributed.git

ci/requirements-py36-netcdf4-dev.yml

@@ -7,12 +7,14 @@ dependencies:
   - dask
   - distributed
   - h5py
+  - h5netcdf
+  - matplotlib
   - pytest
   - numpy
   - pandas
   - scipy
+  - toolz
   - pip:
     - coveralls
     - pytest-cov
-    - h5netcdf
     - git+https://github.com/Unidata/netcdf4-python.git

ci/requirements-py36-pandas-dev.yml

@@ -3,14 +3,18 @@ channels:
   - conda-forge
 dependencies:
   - python=3.6
-  - cython=0.23.4
+  - cython
+  - dask
+  - distributed
+  - h5py
+  - h5netcdf
+  - matplotlib
+  - netcdf4
   - pytest
-  - numpy=1.10.1
-  - netcdf4=1.1.9
-  - scipy=0.16.0
+  - numpy
+  - scipy
   - toolz
   - pip:
     - coveralls
     - pytest-cov
-    - dask
     - git+https://github.com/pydata/pandas.git

ci/requirements-py36-pydap.yml

@@ -6,11 +6,13 @@ dependencies:
   - dask
   - distributed
   - h5py
+  - h5netcdf
+  - matplotlib
   - netcdf4
   - pytest
   - numpy
-  - pandas
   - scipy
+  - toolz
   - pip:
     - coveralls
     - pytest-cov

ci/requirements-py36-windows.yml

@@ -4,10 +4,14 @@ channels:
 dependencies:
   - python=3.6
   - dask
+  - distributed
+  - h5py
+  - h5netcdf
   - matplotlib
-  - netCDF4
+  - netcdf4
+  - pytest
   - numpy
   - pandas
-  - pip
-  - pytest
   - scipy
+  - seaborn
+  - toolz

ci/requirements-py36.yml

@@ -3,18 +3,18 @@ channels:
   - conda-forge
 dependencies:
   - python=3.6
-  - cython
   - dask
   - distributed
   - h5py
+  - h5netcdf
   - matplotlib
   - netcdf4
   - pytest
   - numpy
   - pandas
-  - seaborn
   - scipy
+  - seaborn
+  - toolz
   - pip:
     - coveralls
     - pytest-cov
-    - h5netcdf

conftest.py

@@ -0,0 +1,9 @@
+"""Configuration for pytest."""
+
+
+def pytest_addoption(parser):
+    """Add command-line flags for pytest."""
+    parser.addoption("--run-flaky", action="store_true",
+                     help="runs flaky tests")
+    parser.addoption("--skip-slow", action="store_true",
+                     help="skips slow tests")

doc/whats-new.rst

@@ -50,6 +50,11 @@ By `Henry S. Harrison <https://hsharrison.github.io>`_.
   cleaner output.  The full metadata is still available as ``data.attrs``.
   By `Zac Hatfield-Dodds <https://github.com/Zac-HD>`_.
 
+- Enhanced tests suite by use of ``@slow`` and ``@flaky`` decorators, which are
+  controlled via ``--run-flaky`` and ``--skip-slow`` command line arguments
+  to ``py.test``.  By `Stephan Hoyer <https://github.com/shoyer>`_ and
+  `Phillip J. Wolfram <https://github.com/pwolfram>`_.
+
 Bug fixes
 ~~~~~~~~~
 - ``rolling`` now keeps its original dimension order (:issue:`1125`).

xarray/tests/__init__.py

@@ -114,6 +114,15 @@ def requires_bottleneck(test):
     return test if has_bottleneck else pytest.mark.skip('requires bottleneck')(test)
 
 
+flaky = pytest.mark.skipif(
+    not pytest.config.getoption("--run-flaky"),
+    reason="set --run-flaky option to run flaky tests")
+
+slow = pytest.mark.skipif(
+    pytest.config.getoption("--skip-slow"),
+    reason="set --skip-slow option to run slow tests")
+
+
 class TestCase(unittest.TestCase):
     if PY3:
         # Python 3 assertCountEqual is roughly equivalent to Python 2