Unable to open soil moisture dataset, key error in lru_cache.py

[elfmanryan]

#help_wanted

Expected behavior

Open dataset saved in local store.

Actual behavior

Traceback (most recent call last):
  File "/envs/venv/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 137, in acquire
    file = self._cache[self._key]
  File "/envs/venv/lib/python3.6/site-packages/xarray/backends/lru_cache.py", line 43, in __getitem__
    value = self._cache[key]
KeyError: [<function _open_netcdf4_group at 0x7f154586f8c8>, ('/home/vagrant/.cate/data_stores/local/local.soil_moisture4/ESACCI-SOILMOISTURE-L3S-SSMV-COMBINED-20101101000000-fv03.2.nc', CombinedLock([<SerializableLock: 6f06d6ce-28de-487a-b3e2-b808240f8c35>, <SerializableLock: 77d806d6-8546-49c1-ad84-c642609164d7>])), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('group', None), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/envs/venv/lib/python3.6/site-packages/cate/ds/local.py", line 183, in open_dataset
    monitor=monitor)
  File "/envs/venv/lib/python3.6/site-packages/cate/core/ds.py", line 648, in open_xarray_dataset
    chunks = get_spatial_ext_chunk_sizes(files[0])
  File "/envs/venv/lib/python3.6/site-packages/cate/core/ds.py", line 689, in get_spatial_ext_chunk_sizes
    ds = xr.open_dataset(ds_or_path, decode_times=False)
  File "/envs/venv/lib/python3.6/site-packages/xarray/backends/api.py", line 320, in open_dataset
    filename_or_obj, group=group, lock=lock, **backend_kwargs)
  File "/envs/venv/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 355, in open
    return cls(manager, lock=lock, autoclose=autoclose)
  File "/envs/venv/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__
    self.format = self.ds.data_model
  File "/envs/venv/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 359, in ds
    return self._manager.acquire().value
  File "/envs/venv/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 143, in acquire
    file = self._opener(*self._args, **kwargs)
  File "/envs/venv/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 247, in _open_netcdf4_group
    ds = nc4.Dataset(filename, mode=mode, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 2135, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1752, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: b'/home/vagrant/.cate/data_stores/local/local.soil_moisture4/ESACCI-SOILMOISTURE-L3S-SSMV-COMBINED-20101101000000-fv03.2.nc'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/vagrant/example_run.py", line 38, in <module>
    open_ds = core.open_dataset(local_path)
  File "/envs/venv/lib/python3.6/site-packages/cate/core/ds.py", line 591, in open_dataset
    return data_source.open_dataset(time_range, region, var_names, monitor=monitor)
  File "/envs/venv/lib/python3.6/site-packages/cate/ds/local.py", line 187, in open_dataset
    .format(e), source=self) from e
cate.core.ds.DataAccessError: Data source "local.soil_moisture4": Cannot open local dataset:
[Errno -101] NetCDF: HDF error: b'/home/vagrant/.cate/data_stores/local/local.soil_moisture4/ESACCI-SOILMOISTURE-L3S-SSMV-COMBINED-20101101000000-fv03.2.nc'

Process finished with exit code 1

Steps to reproduce the problem

1. vagrant up (using the VagrantFile attached, removing the '.txt' needed to allow upload here)
   Vagrantfile.txt

2. try and open the soil moisture dataset:

# -*- coding: utf-8 -*-
from cate.core.ds import DATA_STORE_REGISTRY
from cate import core

monitor = core.ConsoleMonitor()
local_store_name = 'local'
local_file_name = 'soil_moisture'

data_store = DATA_STORE_REGISTRY.get_data_store('esa_cci_odp')
local_store = DATA_STORE_REGISTRY.get_data_store(local_store_name)

ds_query = data_store.query(query_expr='SOILMOISTURE')

try:
    ds_query[2].make_local(local_name=local_file_name, time_range='2010-11-1, 2010-12-1', monitor=monitor)
except ValueError:
    print('already local')

local_path_name = '{}.{}'.format(local_store_name, local_file_name)
open_ds = core.open_dataset(local_path_name)

Specifications

Ubuntu 18.04 bionic, python 3.6, all libraries to specification from environment.yml file in cate --see VagrantFile attached.

[forman]

The following issues seem to have a similar root cause, that in the end let netCDF4._netCDF4.pyx library raise with [Errno -68], maybe OPeNDAP-related:

• Failure downloading data with region constraint #818
• Fire: Copying remote data failed #816
• Landcover: NetCDF access failure #783

[jnhansen]

May be related to pydata/xarray#2560

[elfmanryan]

Thanks, looks very similar indeed

…

On Fri, 14 Dec 2018 at 11:52, Johannes Hansen ***@***.***> wrote: May be related to pydata/xarray#2560 <pydata/xarray#2560> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#822 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AZUeHbs4S_hd3adQRqowJ6hIpk1U8h8dks5u45EbgaJpZM4ZRkro> .

-- *Ryan Elfman* | GIS Analyst in Earth Observation & Mapping *ecometrica* London | Boston | Edinburgh | Montreal *Main: **+44 (0)131 662 4342* *Email: ryan.elfman@ecometrica.com <richard.morley@ecometrica.com>* Web: www.ecometrica.com <http://ecometrica.com/> *Skype: * ryan.elfman