Behavior of set_fill_off() changed in v1.5.1 #972
Description
netCDF4-1.5.1 (installed via pip 19.1.1)
Python 3.6.2 (virtualenv)
macOS Sierra (10.12.6)
Based on the documentation, my understanding ofDataset.set_fill_off() is that it leaves variables uninitialized upon creation. This is useful in cases were the variable is immediately filled with real data. With earlier versions of netCDF4, this works as expected:
""" Demonstrate fill value bug in netCDF4-1.5.1+
"""
from netCDF4 import Dataset
from numpy.ma import masked_all
# Write masked data to netCDF, which will be replaced by _FillValue.
dataset = Dataset("test.nc", "w")
dataset.set_fill_off() # comment this out and everything works as expected
dim = dataset.createDimension("dim", 10)
var = dataset.createVariable("var1", "f8", (dim.name,))
var[:] = masked_all((10,), "f8")
dataset.close()
# Read variable from netCDF. It should be masked everywhere.
dataset = Dataset("test.nc", "r")
var = dataset.variables["var1"]
assert var[:].mask.all()As of v1.5.1, the final assertion fails. Inspection of the netCDF file using ncdump shows that var1 is set to _FillValue as expected. However, when var1 is read from the file, it is not masked anywhere. The value is _FillValue (9.9692e+36) everywhere, but the mask of the resulting MaskedArray is False.
Somehow, calling set_fill_off() when the variable is created is preventing it from being masked correctly when the dataset is read. This seems contrary to the documentation, and is a new behavior in v1.5.1.
A workaround is to not call set_fill_off().