Usage of valid_min, valid_max attributes for masking data

[cpaulik]

I'm not sure that this is a bug since you indicated in #493 that CF conventions should be handled by downstream applications.

The attached file is slightly wrong since the values outside of the valid range are not the same as the _FillValue but according to CF they should still be masked since they are outside of the defined valid_min - valid_max range.

See the below example on the attached file:

import netCDF4
print netCDF4.__version__
import numpy as np
fname_nc = '/data/C3S/ASCAT-Level1-testdata/netcdf/W_XX-EUMETSAT-Darmstadt,SURFACE+SATELLITE,METOPB+ASCAT_C_EUMP_20170101003900_22253_eps_o_125_l1.nc'
with netCDF4.Dataset(fname_nc) as ds:
    field = 'azi_angle_trip'
    var = ds.variables[field]
    print var
    data_scaled = ds.variables[field][:, :, 2]
    below_neg_180 = np.where(data_scaled < -180)
    print below_neg_180[0].shape
    print data_scaled[below_neg_180]
    var.set_auto_scale(False)
    data_unscaled = var[:, :, 2]
    print data_unscaled[below_neg_180]

outputs:

1.2.8
<type 'netCDF4._netCDF4.Variable'>
>i2 azi_angle_trip(numRows, numCells, numSigma)
    _FillValue: -32767
    valid_min: -18000
    valid_max: 18000
    standard_name: beam_azimuth_angle
    long_name: beam azimuth angle
    units: degrees
    coordinates: lat lon sigma0
    scale_factor: 0.01
    comment: Incidence angle for re-sampled sigma0 tripplet. Values range from -180 to +180, where minus is west and plus is east.
unlimited dimensions: 
current shape = (3264, 82, 3)
filling on
(12,)
[-327.67999268 -327.67999268 -327.67999268 -327.67999268 -327.67999268
 -327.67999268 -327.67999268 -327.67999268 -327.67999268 -327.67999268
 -327.67999268 -327.67999268]
[-32768 -32768 -32768 -32768 -32768 -32768 -32768 -32768 -32768 -32768
 -32768 -32768]

W_XX-EUMETSAT-Darmstadt,SURFACE+SATELLITE,METOPB+ASCAT_C_EUMP_20170101003900_22253_eps_o_125_l1.zip

[jswhit]

The code tries to set values outside the valid range to the missing_value, but this variable has no missing_value. It's not clear to me from the CF docs whether it should use the _FillValue in this case.

[cpaulik]

The CF docs specify

If only one missing value is needed for a variable then we recommend that this value be specified using the _FillValue attribute.

in their section about missing data

[jswhit]

Fix for this in pull request #672

[jswhit]

closed by pull request #672