Add reading of pixel_quality variable to FCI FDHSI reader

[ameraner]

Feature Request

Is your feature request related to a problem? Please describe.
Currently the FCI FDHSI reader is issuing the warning
[WARNING: 2020-04-30 15:01:51 : satpy.readers.yaml_reader] Can't load ancillary dataset pixel_quality
This is because the reading of that variable is not implemented.

Describe the solution you'd like
Implement the reading of the pixel_quality variable and make it available as a dataset.

Describe any changes to existing user workflow
/

Additional context
/

[gerritholl]

The pixel_quality data variable is in each of the /data/<channel>/measured groups (source: MTG FCI L1 Product User Guide, appendix A.2.4, page 81), so the reader needs to expose this as a dataset for each channel, probably called pixel_quality_vis_04 or vis_04_pixel_quality. By searching for quality in existing satpy readers, the latter seems to be the standard approach. So far quality flags are only exposed by satpy for L2 datasets.

[gerritholl]

To reproduce:

from satpy import Scene
from satpy.utils import debug_on
debug_on()
sc = Scene(["/media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115845_GTT_DEV_20130804115330_20130804115345_N__C_0072_0035.nc"], reader="fci_l1c_fdhsi")
sc.load(["nir_22"])

Result:

[DEBUG: 2020-05-04 12:16:06 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/'
[DEBUG: 2020-05-04 12:16:06 : satpy.readers] Reading ['/home/gholl/checkouts/satpy/satpy/etc/readers/fci_l1c_fdhsi.yaml']
[DEBUG: 2020-05-04 12:16:06 : satpy.readers.yaml_reader] Assigning to fci_l1c_fdhsi: ['/media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115845_GTT_DEV_20130804115330_20130804115345_N__C_0072_0035.nc']
[DEBUG: 2020-05-04 12:16:06 : satpy.readers.fci_l1c_fdhsi] Reading: /media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115845_GTT_DEV_20130804115330_20130804115345_N__C_0072_0035.nc
[DEBUG: 2020-05-04 12:16:06 : satpy.readers.fci_l1c_fdhsi] Start: 2013-08-04 11:53:30
[DEBUG: 2020-05-04 12:16:06 : satpy.readers.fci_l1c_fdhsi] End: 2013-08-04 11:53:45
[DEBUG: 2020-05-04 12:16:06 : satpy.composites] Looking for composites config file fci.yaml
[DEBUG: 2020-05-04 12:16:06 : satpy.composites] Looking for composites config file visir.yaml
[DEBUG: 2020-05-04 12:16:06 : satpy.readers.fci_l1c_fdhsi] Channel nir_22 resolution: 11136
[DEBUG: 2020-05-04 12:16:06 : satpy.readers.fci_l1c_fdhsi] Row/Cols: 279 / 11136
[DEBUG: 2020-05-04 12:16:07 : satpy.readers.fci_l1c_fdhsi] Calculated area extent: (-5567999.998527619, 4175999.998889502, 5567999.998577303, 3896999.9989620363)
/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
  proj_string = self.to_proj4()
[DEBUG: 2020-05-04 12:16:07 : satpy.readers.fci_l1c_fdhsi] Reading nir_22 from /media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115845_GTT_DEV_20130804115330_20130804115345_N__C_0072_0035.nc
[WARNING: 2020-05-04 12:16:07 : satpy.readers.yaml_reader] Can't load ancillary dataset pixel_quality

[gerritholl]

The effective_radiance data variable within each /data/<channel>/measured group has a variable attribute ancillary_variables with the value pixel_quality. Therefore FileYAMLReader._load_ancillary_variables tries to load this variable, but fails to find it. The complex group structure may complicate this.

[gerritholl]

FileYAMLReader._load_ancillary_variables calls FileYAMLReader.get_dataset_key('pixel_quality'), but this dataset is not found because it's not defined in the yaml file. FCI files have a pixel_quality for each channel, so there would appear to be insufficient information passed to get_dataset_key in any case. I'm not sure how to solve this.

[djhoese]

We have this same problem with ABI L1b and the DQF flag. We currently just don't include it. One solution would be to prefix/suffix the field like DQF_C01 (or the equivalent for FCI).

[gerritholl]

I will try to implement renaming. It's a bit of a hack, but it works because Satpy calls the FileHandler for the variables in the wishlist before it moves on to the ancilarry variables, so I can rewrite the ancillary_variables variable attribute to prepend the channel label to the pixel_quality string. A longer term solution might be built on dynamic dataset IDs (see #1088), which would allow for the pixel_quality to be defined multiple times but with different channels or groups.

[djhoese]

A longer term solution might be built on dynamic dataset IDs (see #1088)

Yeah that's a good point. So the reader would add an extra key for DatasetIDs so it could define DatasetID(name='pixel_quality', parent_channel='C01') (or some other better naming). I like that.

[gerritholl]

On slack @mraspaud suggested that you might want to use the available_datasets mechanism to skip having all the datasets defined in the yaml file. See https://satpy.readthedocs.io/en/latest/api/satpy.readers.html#satpy.readers.file_handlers.BaseFileHandler.available_datasets for more information.