Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a GRIB backend via ECMWF cfgrib / ecCodes #2476

Merged
merged 29 commits into from
Oct 17, 2018
+162 −8
Merged

Add a GRIB backend via ECMWF cfgrib / ecCodes #2476

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
72606f7
Integration of ECMWF cfgrib driver to read GRIB files into xarray.
alexamici Oct 9, 2018
71fcbe7
Remove all coordinate renaming from the cfgrib backend.
alexamici Oct 9, 2018
6faa7b9
Move flavour selection to `cfgrib.Dataset.from_path`.
alexamici Oct 9, 2018
1469a0e
Sync xarray backend import style with xarray.
alexamici Oct 9, 2018
12811e8
Make use of the new xarray.backends.FileCachingManager.
alexamici Oct 9, 2018
a4409b6
Add just-in-case locking for ecCodes.
alexamici Oct 9, 2018
80b8788
Explicitly assign attributes to CfGribArrayWrapper
alexamici Oct 10, 2018
9dfd660
Add missing locking in CfGribArrayWrapper and use explicit_indexing_a…
alexamici Oct 10, 2018
edc4e85
Add a comment about the ugly work-around needed for filter_by_keys.
alexamici Oct 10, 2018
9b5335a
Declare correct indexing support.
alexamici Oct 10, 2018
186a504
Merge branch 'upstream' into feature/grib-support-via-cfgrib
alexamici Oct 14, 2018
485a409
Add TestCfGrib test class.
alexamici Oct 14, 2018
81f18c2
cfgrib doesn't store a file reference so no need for CachingFileManager.
alexamici Oct 14, 2018
5dedb3f
Add cfgrib testing to Travis-CI.
alexamici Oct 14, 2018
831ae4f
Naming.
alexamici Oct 14, 2018
6372e6e
Fix line lengths and get to 100% coverage.
alexamici Oct 14, 2018
8e9b2e3
Add reference to *cfgrib* engine in inline docs.
alexamici Oct 14, 2018
07b9469
First cut of the documentation.
alexamici Oct 14, 2018
340720a
Tentative test cfgrib under dask.distributed.
alexamici Oct 14, 2018
4d84f70
Better integration test.
alexamici Oct 14, 2018
0b027db
Remove explicit copyright and license boilerplate to harmonise with o…
alexamici Oct 15, 2018
a4ead54
Add a usage example.
alexamici Oct 15, 2018
ec80d86
Fix code style.
alexamici Oct 15, 2018
f30b7d0
Fix doc style.
alexamici Oct 16, 2018
223d25c
Fix docs testing. The example.grib file is not accessible.
alexamici Oct 17, 2018
2ef993f
Merge remote-tracking branch 'upstream/master' into feature/grib-supp…
alexamici Oct 17, 2018
bbf01e3
Fix merge in docs.
alexamici Oct 17, 2018
da2b9dd
Fix merge in docs.
alexamici Oct 17, 2018
eda96a4
Fix doc style.
alexamici Oct 17, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion doc/io.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -641,9 +641,13 @@ GRIB format via cfgrib
----------------------

xarray supports reading GRIB files via ECMWF cfgrib_ python driver and ecCodes_
C-library, if they are installed. To open GRIB file supply ``engine='cfgrib'``
C-library, if they are installed. To open a GRIB file supply ``engine='cfgrib'``
to :py:func:`~xarray.open_dataset`:

.. ipython:: python

ds_grib = xr.open_dataset('example.grib', engine='cfgrib')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If example.grib doesn't actually exist when we build the docs, this will give a nasty error message. It would be better to use a :verbatim: directive here -- scroll open to the opendap examples to see what that looks like.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ouch! I noticed that there was no docs testing in Travis-CI and wondered if you where testing the docs at all. I should have built the docs myself. What is the intended way do you build the docs? python sertup.py build_sphinx fails on my setup due to missing numpydoc.

I'm fixing it as you suggest.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the :verbatim: directive, but I was not able to test the build of the documentation due to several errors when building the gallery (a number of core dump in GEOS, maybe due to the setup on my MacOS).

The fix looks trivial enough that it may work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would work:

.. ipython::
    :verbatim:

    In [3]: ds_grib = xr.open_dataset('example.grib', engine='cfgrib')

We actually do test the doc build on Travis-CI, but unfortunately there's no easy way to see generated docs and also errors in ipython directive blocks don't stop the build (there's a bug we filed about this somewhere).

Copy link
Collaborator Author

@alexamici alexamici Oct 17, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. I mis-read the example.


We recommend installing ecCodes via conda::

conda install -c conda-forge eccodes
Expand Down
23 changes: 2 additions & 21 deletions xarray/backends/cfgrib_.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,3 @@
#
# Copyright 2017-2018 European Centre for Medium-Range Weather Forecasts.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Authors:
# Alessandro Amici - B-Open - https://bopen.eu
#

from __future__ import absolute_import, division, print_function

import numpy as np
Expand All @@ -28,8 +9,8 @@
from .locks import ensure_lock, SerializableLock

# FIXME: Add a dedicated lock, even if ecCodes is supposed to be thread-safe
# in most circumstances. See:
# https://confluence.ecmwf.int/display/ECC/Frequently+Asked+Questions
# in most circumstances. See:
# https://confluence.ecmwf.int/display/ECC/Frequently+Asked+Questions
ECCODES_LOCK = SerializableLock()


Expand Down
3 changes: 2 additions & 1 deletion xarray/tests/test_distributed.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
import xarray as xr
from xarray.backends.locks import HDF5_LOCK, CombinedLock
from xarray.tests.test_backends import (ON_WINDOWS, create_tmp_file,
create_tmp_geotiff, open_example_dataset)
create_tmp_geotiff,
open_example_dataset)
from xarray.tests.test_dataset import create_test_data

from . import (
Expand Down