Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add warning for netCDF4 bug #1835

Closed
wants to merge 8 commits into from
Closed
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions doc/whats-new.rst
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@ Enhancements

Bug fixes
~~~~~~~~~
- Added warning in api.py of a netCDF4 bug that occurs when
the filepath has 88 characters (:issue:`1745`).
By `Liam Brannigan <https://github.com/braaannigan>` _.
- Fixed encoding of multi-dimensional coordinates in
:py:meth:`~Dataset.to_netcdf` (:issue:`1763`).
By `Mike Neish <https://github.com/neishm>`_.
Expand Down
10 changes: 9 additions & 1 deletion xarray/backends/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from glob import glob
from io import BytesIO
from numbers import Number

import warnings

import numpy as np

Expand Down Expand Up @@ -281,6 +281,14 @@ def maybe_decode_store(store, lock=False):
engine = _get_default_engine(filename_or_obj,
allow_remote=True)
if engine == 'netcdf4':
import netCDF4
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you accidentally kept your changes in this file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought that I had checked out the original version again to replace all of my edits, but that doesn't seem to have worked. I'm looking for a solution.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can simply revert this file (e.g., copy/paste from master) and then add another commit

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was worried that would cause some new unforeseen problem. I've commited this now.

if len( filename_or_obj ) == 88 and float(netCDF4.__version__[:3]) < 1.3:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove spaces around filename_or_obj, per pep8.

warnings.warn('\nA segmentation fault may occur when the \n'
'file path has exactly 88 characters. The issue is known \n'
'to occur with version 1.2.4 of netCDF4 and can be \n'
'addressed by upgrading netCDF4 to at least version 1.3.1. \n'
'More details can be found here: \n'
'https://github.com/pydata/xarray/issues/1745 \n')
store = backends.NetCDF4DataStore.open(filename_or_obj,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you put this inside NetCDF4DataStore.open() instead? We try to keep the backend specific logic in the backend classes.

group=group,
autoclose=autoclose)
Expand Down
13 changes: 13 additions & 0 deletions xarray/backends/netCDF4_.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
from __future__ import print_function
import functools
import operator
import warnings
from distutils.version import LooseVersion

import numpy as np

Expand Down Expand Up @@ -244,6 +246,17 @@ def __init__(self, netcdf4_dataset, mode='r', writer=None, opener=None,
def open(cls, filename, mode='r', format='NETCDF4', group=None,
writer=None, clobber=True, diskless=False, persist=False,
autoclose=False):
import netCDF4 as nc4
if (len(filename) == 88 and
LooseVersion(nc4.__version__) < "1.3.1"):
warnings.warn(
'\nA segmentation fault may occur when the\n'
'file path has exactly 88 characters as it does.\n'
'in this case. The issue is known to occur with\n'
'version 1.2.4 of netCDF4 and can be addressed by\n'
'upgrading netCDF4 to at least version 1.3.1.\n'
'More details can be found here:\n'
'https://github.com/pydata/xarray/issues/1745 \n')
if format is None:
format = 'NETCDF4'
opener = functools.partial(_open_netcdf4_group, filename, mode=mode,
Expand Down
10 changes: 10 additions & 0 deletions xarray/tests/test_backends.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import tempfile
import unittest
import sys
import warnings

import numpy as np
import pandas as pd
Expand Down Expand Up @@ -1056,6 +1057,15 @@ def test_unsorted_index_raises(self):
except IndexError as err:
self.assertIn('first by calling .load', str(err))

def test_88_character_filename_segmentation_fault(self):
# should be fixed in netcdf4 v1.3.1
with mock.patch('netCDF4.__version__', '1.2.4'):
with warnings.catch_warnings():
warnings.simplefilter("error")
with raises_regex(Warning, 'segmentation fault'):
# Need to construct 88 character filepath
xr.Dataset().to_netcdf('a' * (88 - len(os.getcwd()) - 1))


class NetCDF4DataStoreAutocloseTrue(NetCDF4DataTest):
autoclose = True
Expand Down