Problem with loading mask in iris 2 when missing_value is wrong

[valeriupredoi]

Hi guys, here's one obscure one for you: we (the ESMValTool guys) are working with hundreds of models and a lot of them have metadata or data issues and we fix them internally, but we found one that albeit super easy to fix, it may uncover a hidden bug in iris. Here's the synopsys:

• Problematic dataset: BNU-ESM; you can have a look at the full file on Jasmin at /badc/cmip5/data/cmip5/output1/BNU/BNU-ESM/historical/mon/aerosol/aero/r1i1p1/files/od550aer_20120504/od550aer_aero_BNU-ESM_historical_r1i1p1_185001-200512.nc
• Data description: this particular dataset contains data points that should be masked; these have actual values of 1e36;
• Problem at hand: iris 1.13 recognizes these values to be masked and builds the cube.data.mask structure correctly; iris 2.2.0 doesn't, it builds no mask and assigns 1e36 values as valid data points;
• Cause: the dataset file contains both _FillValue and missing_value netcdf attributes both with 1e+20f values (CMOR standard); in effect the missing values' actual numeric values in the data array are 1e36 and are interpreted by iris 2.2.0 as valid values
• Fix: if setting the missing_value to 1e36 in the file (via say cdo setmissval) then iris 2 will create the correct mask; this is a bit of a twisted situation because the _FillValue's will then have to be reverted to 1e+20f at save point to comply with CMOR standards.

My take on this would be if you guys reinstated the same approach to missing data as it was done in iris 1 it would be great! But I understand why the issues due to partial support of dask+masks, so if you could look into this case in particular case and see what can be done, that'd be awesome! Cheers muchly!

[pp-mo]

dataset file contains both _FillValue and missing_value netcdf attributes both with 1e+20f values ...
... the missing values' actual numeric values in the data array are 1e36 ...
... iris 1.13 recognizes these values to be masked

As described this all sounds very odd - Iris 1.13 should on no account ignore _FillValue or missing_value attributes actually recorded in the file. Though I think Iris2 might in some cases.

you can have a look at the full file on Jasmin

Unfortunately I can't get at that file with any particular immediacy ..
Here @ metoffice AVD, we aren't researchers + none of us has a JASMIN account !
( at least yet )

Can you maybe just copy us an ncdump -h of it here ?

[valeriupredoi]

shoot! sorry @pp-mo hadn't realized about Jasmin access, so used to working on it it's like a second nature to me 😁 Here's the ncdump:

netcdf od550aer_aero_BNU-ESM_historical_r1i1p1_185001-200512 {
dimensions:
	time = UNLIMITED ; // (1872 currently)
	lat = 64 ;
	lon = 128 ;
	bnds = 2 ;
variables:
	double time(time) ;
		time:bounds = "time_bnds" ;
		time:units = "days since 1850-01-01 00:00:00" ;
		time:calendar = "noleap" ;
		time:axis = "T" ;
		time:long_name = "time" ;
		time:standard_name = "time" ;
	double time_bnds(time, bnds) ;
	double lat(lat) ;
		lat:bounds = "lat_bnds" ;
		lat:units = "degrees_north" ;
		lat:axis = "Y" ;
		lat:long_name = "latitude" ;
		lat:standard_name = "latitude" ;
	double lat_bnds(lat, bnds) ;
	double lon(lon) ;
		lon:bounds = "lon_bnds" ;
		lon:units = "degrees_east" ;
		lon:axis = "X" ;
		lon:long_name = "longitude" ;
		lon:standard_name = "longitude" ;
	double lon_bnds(lon, bnds) ;
	float od550aer(time, lat, lon) ;
		od550aer:standard_name = "atmosphere_optical_thickness_due_to_ambient_aerosol" ;
		od550aer:long_name = "Ambient Aerosol Optical Thickness at 550 nm" ;
		od550aer:comment = "AOD from the ambient aerosls (i.e., includes aerosol water).  Does not include AOD from stratospheric aerosols if these are prescribed but includes other possible background aerosol types." ;
		od550aer:units = "1" ;
		od550aer:cell_methods = "time: mean" ;
		od550aer:cell_measures = "area: areacella" ;
		od550aer:missing_value = 1.e+20f ;
		od550aer:_FillValue = 1.e+20f ;
		od550aer:associated_files = "baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_aerosol_fx_BNU-ESM_historical_r0i0p0.nc areacella: areacella_fx_BNU-ESM_historical_r0i0p0.nc" ;

// global attributes:
		:institution = "GCESS,BNU,Beijing,China" ;
		:institute_id = "BNU" ;
		:experiment_id = "historical" ;
		:source = "BNU-ESM" ;
		:model_id = "BNU-ESM" ;
		:forcing = "Nat,Ant" ;
		:parent_experiment_id = "piControl" ;
		:parent_experiment_rip = "r1i1p1" ;
		:branch_time = 0. ;
		:contact = "Ji Duoying (duoyingji@bnu.edu.cn)" ;
		:initialization_method = 1 ;
		:physics_version = 1 ;
		:tracking_id = "3044603d-9755-48f9-b6cf-1369d5d45095" ;
		:product = "output" ;
		:experiment = "historical" ;
		:frequency = "mon" ;
		:creation_date = "2012-03-12T07:16:46Z" ;
		:history = "2012-03-12T07:16:46Z CMOR rewrote data to comply with CF standards and CMIP5 requirements." ;
		:Conventions = "CF-1.4" ;
		:project_id = "CMIP5" ;
		:table_id = "Table aero (26 July 2011) aa8461434060b130391069fa7489bcec" ;
		:title = "BNU-ESM model output prepared for CMIP5 historical" ;
		:parent_experiment = "pre-industrial control" ;
		:modeling_realm = "aerosol" ;
		:realization = 1 ;
		:cmor_version = "2.7.1" ;
}

[github-actions]

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

[github-actions]

This stale issue has been automatically closed due to a lack of community activity.

If you still care about this issue, then please either:

• Re-open this issue, if you have sufficient permissions, or
• Add a comment pinging @SciTools/iris-devs who will re-open on your behalf.