Cannot save netcdf files with non-standard calendars

[mcgibbon]

Code Sample, a copy-pastable example if possible

Using noleap.nc from the following zip: noleap.zip

import xarray as xr
ds = xr.open_dataset('noleap.nc')
ds.to_netcdf('noleap_new.nc')

Problem description

A long traceback gets printed out (sorry, I can't copy it properly from my current machine) that ends in

TypeError: float() argument must be a string or a number, not 'netcdftime._netcdftime.DatetimeNoLeap'

Expected Output

Obviously, we expect the file to save. If xarray can decode the times, it should be able to encode them.

Output of xr.show_versions()

AttributeError: module 'xarray' has no attribute 'show_versions'

I'm running xarray 0.9.1, numpy 1.12.1, pandas 0.21.0, and netCDF4 1.2.8.

[spencerkclark]

@mcgibbon yes, work is being done to fix this. #1252 addresses it (among other things).

[mcgibbon]

Great! I've had two people independently come to me with this same problem in the past three weeks, so it's good to see it's being worked on.

[jhamman]

@mcgibbon - would you mind confirming this is happening with a current version of xarray? Also, the full traceback would be appreciated. I guess what we really need is a regression test.

[spencerkclark]

@jhamman with xarray version 0.10.2:

>>> import xarray as xr
>>> xr.__version__
'0.10.2'
>>> from netcdftime import DatetimeNoLeap
>>> da = xr.DataArray([DatetimeNoLeap(1, 1, 1), DatetimeNoLeap(1, 2, 1)], name='noleap')
>>> da.to_netcdf('noleap.nc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "//anaconda/envs/research/lib/python2.7/site-packages/xarray/core/dataarray.py", line 1517, in to_netcdf
    return dataset.to_netcdf(*args, **kwargs)
  File "//anaconda/envs/research/lib/python2.7/site-packages/xarray/core/dataset.py", line 1131, in to_netcdf
    unlimited_dims=unlimited_dims)
  File "//anaconda/envs/research/lib/python2.7/site-packages/xarray/backends/api.py", line 657, in to_netcdf
    unlimited_dims=unlimited_dims)
  File "//anaconda/envs/research/lib/python2.7/site-packages/xarray/core/dataset.py", line 1068, in dump_to_store
    unlimited_dims=unlimited_dims)
  File "//anaconda/envs/research/lib/python2.7/site-packages/xarray/backends/common.py", line 358, in store
    variables, attributes = self.encode(variables, attributes)
  File "//anaconda/envs/research/lib/python2.7/site-packages/xarray/backends/common.py", line 443, in encode
    variables, attributes = cf_encoder(variables, attributes)
  File "//anaconda/envs/research/lib/python2.7/site-packages/xarray/conventions.py", line 747, in cf_encoder
    for k, v in iteritems(variables))
  File "python2/cyordereddict/_cyordereddict.pyx", line 47, in cyordereddict._cyordereddict.OrderedDict.__init__ (python2/cyordereddict/_cyordereddict.c:1225)
  File "//anaconda/envs/research/lib/python2.7/_abcoll.py", line 571, in update
    for key, value in other:
  File "//anaconda/envs/research/lib/python2.7/site-packages/xarray/conventions.py", line 747, in <genexpr>
    for k, v in iteritems(variables))
  File "//anaconda/envs/research/lib/python2.7/site-packages/xarray/conventions.py", line 399, in encode_cf_variable
    var = ensure_dtype_not_object(var, name=name)
  File "//anaconda/envs/research/lib/python2.7/site-packages/xarray/conventions.py", line 361, in ensure_dtype_not_object
    data = data.astype(dtype=_infer_dtype(data, name))
  File "//anaconda/envs/research/lib/python2.7/site-packages/xarray/conventions.py", line 320, in _infer_dtype
    .format(name))
ValueError: unable to infer dtype on variable 'noleap'; xarray cannot serialize arbitrary Python objects

With https://github.com/spencerkclark/xarray/tree/NetCDFTimeIndex (#1252):

>>> import xarray as xr
>>> from netcdftime import DatetimeNoLeap
>>> da = xr.DataArray([DatetimeNoLeap(1, 1, 1), DatetimeNoLeap(1, 2, 1)], name='noleap')
>>> da.to_netcdf('noleap.nc')
>>> xr.open_dataset('noleap.nc').noleap
/Users/spencerclark/xarray-dev/xarray/xarray/coding/times.py:150: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using dummy netCDF4.datetime objects instead, reason: dates out of range
  result = decode_cf_datetime(example_value, units, calendar)
/Users/spencerclark/xarray-dev/xarray/xarray/coding/variables.py:66: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using dummy netCDF4.datetime objects instead, reason: dates out of range
  return self.func(self.array[key])
<xarray.DataArray 'noleap' (dim_0: 2)>
array([netcdftime._netcdftime.DatetimeNoLeap(1, 1, 1, 0, 0, 0, 0, 3, 1),
       netcdftime._netcdftime.DatetimeNoLeap(1, 2, 1, 0, 0, 0, 0, 6, 32)],
      dtype=object)
Dimensions without coordinates: dim_0

(#1252 also adds tests for this)

[mcgibbon]

Thanks @spencerkclark !

[spencerkclark]

I think this can be closed now that #1252 has been merged. Using @mcgibbon's example file the following now works using the master branch:

In [1]: import xarray as xr

In [2]: with xr.set_options(enable_cftimeindex=True):
   ...:     ds = xr.open_dataset('noleap.nc')
   ...:

In [3]: ds.to_netcdf('noleap_new.nc')

In [4]: with xr.set_options(enable_cftimeindex=True):
   ...:     ds2 = xr.open_dataset('noleap_new.nc')
   ...:

In [5]: xr.testing.assert_identical(ds, ds2)