FCI reader can read pixel_quality flags only after reading corresponding channel data

[gerritholl]

Describe the bug

The fci_l1c_fdhsi reader can only read the pixel_quality_* flags (introduced in #1177) after reading the corresponding channel data, or at the same time. An attempt to read the pixel_quality_* fields before reading the corresponding channel data fails with KeyError.

To Reproduce

from satpy import Scene
from glob import glob
from satpy.utils import debug_on
debug_on()
path_to_testdata = "/media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/"
scn = Scene(filenames=glob(path_to_testdata + "/*BODY*0020.nc"),
        reader=['fci_l1c_fdhsi'])
scn.load(["ir_123", "ir_123_pixel_quality"])  # succeeds
scn = Scene(filenames=glob(path_to_testdata + "/*BODY*0020.nc"),
        reader=['fci_l1c_fdhsi'])
scn.load(["ir_123"])
scn.load(["ir_123_pixel_quality"])  # succeeds
scn = Scene(filenames=glob(path_to_testdata + "/*BODY*0020.nc"),
        reader=['fci_l1c_fdhsi'])
scn.load(["ir_123_pixel_quality"])  # fails with KeyError

Expected behavior

I expect that each of the three ways of loading in the example results in the same dataset ir_123_pixel_quality.

Actual results

[DEBUG: 2020-06-04 09:44:29 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/'                                                                                                                                                                  [37/1803]
[DEBUG: 2020-06-04 09:44:29 : satpy.readers] Reading ['/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/etc/readers/fci_l1c_fdhsi.yaml', '/home/gholl/checkouts/satpy/satpy/etc/readers/fci_l1c_fdhsi.yaml']                                                           
[DEBUG: 2020-06-04 09:44:29 : satpy.readers.yaml_reader] Assigning to fci_l1c_fdhsi: ['/media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115500_GTT_DEV_20130804114945_20130804
115000_N__C_0072_0020.nc']
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Reading: /media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115500_GTT_DEV_20130804114945_20130804115000_N__C_0072_00
20.nc
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Start: 2013-08-04 11:49:45
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] End: 2013-08-04 11:50:00
[DEBUG: 2020-06-04 09:44:30 : satpy.composites] Looking for composites config file fci.yaml
[DEBUG: 2020-06-04 09:44:30 : satpy.composites] Looking for composites config file visir.yaml
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Channel ir_123 resolution: 5568
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Row/Cols: 140 / 5568
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Calculated area extent: (-5567999.994206558, 2.984455866879114e-06, 5567999.994200589, -279999.999705529)                                                                                                                     
/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin
ate-reference-systems
  proj_string = self.to_proj4()
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Reading ir_123 from /media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115500_GTT_DEV_20130804114945_20130804115000_N
__C_0072_0020.nc
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Reading ir_123_pixel_quality from /media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115500_GTT_DEV_20130804114945_20
130804115000_N__C_0072_0020.nc
[DEBUG: 2020-06-04 09:44:30 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/'
[DEBUG: 2020-06-04 09:44:30 : satpy.readers] Reading ['/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/etc/readers/fci_l1c_fdhsi.yaml', '/home/gholl/checkouts/satpy/satpy/etc/readers/fci_l1c_fdhsi.yaml']                                                           
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.yaml_reader] Assigning to fci_l1c_fdhsi: ['/media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115500_GTT_DEV_20130804114945_20130804
115000_N__C_0072_0020.nc']
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Reading: /media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115500_GTT_DEV_20130804114945_20130804115000_N__C_0072_00
20.nc
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Start: 2013-08-04 11:49:45
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] End: 2013-08-04 11:50:00
[DEBUG: 2020-06-04 09:44:30 : satpy.composites] Looking for composites config file fci.yaml
[DEBUG: 2020-06-04 09:44:30 : satpy.composites] Looking for composites config file visir.yaml
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Channel ir_123 resolution: 5568
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Row/Cols: 140 / 5568
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Calculated area extent: (-5567999.994206558, 2.984455866879114e-06, 5567999.994200589, -279999.999705529)                                                                                                                     
/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin
ate-reference-systems
  proj_string = self.to_proj4()
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Reading ir_123 from /media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115500_GTT_DEV_20130804114945_20130804115000_N
__C_0072_0020.nc
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Reading ir_123_pixel_quality from /media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115500_GTT_DEV_20130804114945_20
130804115000_N__C_0072_0020.nc
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.fci_l1c_fdhsi] Reading ir_123_pixel_quality from /media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115500_GTT_DEV_20130804114945_20
130804115000_N__C_0072_0020.nc
[DEBUG: 2020-06-04 09:44:30 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/'
[DEBUG: 2020-06-04 09:44:30 : satpy.readers] Reading ['/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/etc/readers/fci_l1c_fdhsi.yaml', '/home/gholl/checkouts/satpy/satpy/etc/readers/fci_l1c_fdhsi.yaml']
[DEBUG: 2020-06-04 09:44:30 : satpy.readers.yaml_reader] Assigning to fci_l1c_fdhsi: ['/media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115500_GTT_DEV_20130804114945_20130804
115000_N__C_0072_0020.nc']
[DEBUG: 2020-06-04 09:44:31 : satpy.readers.fci_l1c_fdhsi] Reading: /media/nas/x21308/2020_04_MTG_unofficial_Testdata/20130804_RC72/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY--L2P-NC4E_C_EUMT_20130804115500_GTT_DEV_20130804114945_20130804115000_N__C_0072_00
20.nc
[DEBUG: 2020-06-04 09:44:31 : satpy.readers.fci_l1c_fdhsi] Start: 2013-08-04 11:49:45
[DEBUG: 2020-06-04 09:44:31 : satpy.readers.fci_l1c_fdhsi] End: 2013-08-04 11:50:00
[DEBUG: 2020-06-04 09:44:31 : satpy.composites] Looking for composites config file fci.yaml
[DEBUG: 2020-06-04 09:44:31 : satpy.composites] Looking for composites config file visir.yaml
Traceback (most recent call last):
  File "mwe60.py", line 15, in <module>
    scn.load(["ir_123_pixel_quality"])  # fails with KeyError
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/scene.py", line 973, in load
    self.read(**kwargs)
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/scene.py", line 880, in read
    return self._read_datasets(nodes, **kwargs)
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/scene.py", line 721, in _read_datasets
    new_datasets = reader_instance.load(ds_ids, **kwargs)
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 872, in load
    ds = self._load_dataset_with_area(dsid, coords, **kwargs)
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 779, in _load_dataset_with_area
    area = self._load_dataset_area(dsid, file_handlers, coords, **kwargs)
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 761, in _load_dataset_area
    return self._load_area_def(dsid, file_handlers, **kwargs)
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 690, in _load_area_def
    return _load_area_def(dsid, file_handlers)
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 884, in _load_area_def
    area_defs = [fh.get_area_def(dsid) for fh in file_handlers]
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 884, in <listcomp>
    area_defs = [fh.get_area_def(dsid) for fh in file_handlers]
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/fci_l1c_fdhsi.py", line 301, in get_area_def
    area_extent, nlines, ncols = self.calc_area_extent(key)
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/fci_l1c_fdhsi.py", line 250, in calc_area_extent
    nlines, ncols = self[measured + "/effective_radiance/shape"]
  File "/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/readers/netcdf_utils.py", line 183, in __getitem__
    val = self.file_content[key]
KeyError: 'data/ir_123_pixel_quality/measured/effective_radiance/shape'

Environment Info:

• OS: openSUSE 15.0
• Satpy Version: 0.21.1.dev290+ga761d23d
• PyResample Version: 1.15.0+192.g1c2b794

Additional context

#1177 used some magic to enable the reading of pixel_quality variables. The problem is that in the source files, they're all just called pixel_quality. The proper way to disambiguate them would be with a parent_dataset field in a dynamic dataset id (#1088), however Satpy doesn't support the latter yet, and as a temporary solution the name in the ancillary_variables is rewritten. That means that loading a channel dataset has a side-effect on the loading of the corresponding pixel quality (later), which probably explains the bug described here.

[gerritholl]

This happens because the FileYAMLReader calls get_area_def using the pixel_quality field, and the FCI reader doesn't know how to determine the area extent of the pixel quality. The magic from #1177 is currently applied to loading datasets only. The KeyError happens before get_dataset is ever called.