cannot set _FillValue attribute for VLEN or compound variable

[shoyer]

Is this an intrinsic limitation of netCDF-C, or simply something that hasn't been wrapped yet?

I don't need the aspect of filling in default values in a Variable, but I would like to be able to set _FillValue to indicate values that should be treated as missing when read from disk.

My use case is supporting arrays of (unicode) strings with missing values in xarray.

[shoyer]

For example, a good choice for _FillValue could be one of the non-character unicode symbols (e.g., U+FFFF), which are guaranteed not to correspond to valid characters.

[jswhit]

The C-lib doesn't support fill values for non-string vlens and compound variables. It might work for vlen strings (NC_STRING), let me have a look....

[shoyer]

What about setting the _FillValue attribute, rather than the fill_value interpreted by netCDF-C/HDF5 (i.e., by nc_set_fill)?

Currently, overriding _FillValue by setting the attribute is not allowed, but I don't think the attribute is actually directly understood by netCDF-C (though I could be wrong here):

netcdf4-python/netCDF4/_netCDF4.pyx

Lines 3864 to 3867 in 967da7e

	if name == '_FillValue':
	msg='_FillValue attribute must be set when variable is '+\
	'created (using fill_value keyword to createVariable)'
	raise AttributeError(msg)

[dopplershift]

Well, there's this:

Attribute names commencing with underscore ('_') are reserved for use by the netCDF library.

Given the wording in Appendix A I don't think the python wrapper should be playing games here. @WardF anything to add?

[WardF]

You can set _FillValue by hand and it is interpreted by the C library; you don't have to use nc_set_fill. This is in the documentation somewhere, I will see if I can track it down.

[jswhit]

Pull request #732 allows _FillValue to be set for vlen string variables using the fill_value createVariable kwarg. Still cannot be set for non-string vlens and compound variables though.

[shoyer]

@jswhit Awesome, the possibility of setting fill-value for vlen strings would solve the major issue here.

[jswhit]

Can you give it a try and let me know if it does what you need?

[shoyer]

@jswhit I can give it a try, but I would trust a unit test more than my anecdotal experience :)

[shoyer]

Yes, it works -- even with automatic decoding of array values matching the fill-value into the appropriate missing value!

In [4]: ds = netCDF4.Dataset('varlen.nc', 'w')

In [5]: ds.createDimension('x', 2)
Out[5]: <class 'netCDF4._netCDF4.Dimension'>: name = 'x', size = 2

In [6]: var = ds.createVariable('y', str, ('x',), fill_value='<missing>')

In [7]: var[0] = 'first'

In [8]: ds.close()

In [11]: import xarray

In [12]: ds = xarray.open_dataset('varlen.nc')

In [13]: ds.load()
Out[13]:
<xarray.Dataset>
Dimensions:  (x: 2)
Dimensions without coordinates: x
Data variables:
    y        (x) object 'first' nan

[jswhit]

pull request merged.