Auto filling CDS requests with months and days is incompatible with CDS selection limits

[EddyCMWF]

What happened?

The CDS plugin populates any request that contains a field year with months and days if either don't exist in the request.
This conflicts with the CDS selection limit rules which are based on the number of selections made in the request, for example it is impossible to download data from this catalogue entry:
https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-albedo?tab=form

This is because there is no day field in the form, so the ek-cds plugin populates it with a list of values 1 to 31. which breaks the selection limits for this dataset, which is 20.

The questions are:

1. Why are we doing this, are we really adding value by making it possible to submit a partial request and assume we know the rest of the story?
2. If we do choose to do this, then can we be more methodical about it? e.g. only provide values when we know they are relevant.

What are the steps to reproduce the bug?

ek_alb = ek.data.from_source(
"cds",
'satellite-albedo',
{
'variable': 'albb_bh',
'satellite': 'noaa_7',
'sensor': 'avhrr',
'product_version': 'v2',
'horizontal_resolution': '4km',
'year': '1983',
'month': '01',
'nominal_day': '10',
'format': 'zip',
},
)

Version

latest

Platform (OS and architecture)

MacOS + Linux

Relevant log output

CDS error:
2023-09-18 13:46:36,003 INFO Welcome to the CDS
2023-09-18 13:46:36,005 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/satellite-albedo
2023-09-18 13:46:36,492 INFO Request is queued
2023-09-18 13:46:37,552 INFO Request is failed
2023-09-18 13:46:37,554 ERROR Message: the request you have submitted is not valid
2023-09-18 13:46:37,555 ERROR Reason:  Request too large. Requesting 31 items, limit is 20
2023-09-18 13:46:37,556 ERROR   Traceback (most recent call last):
2023-09-18 13:46:37,557 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 163, in _consume
2023-09-18 13:46:37,558 ERROR       result = handle_locally()
2023-09-18 13:46:37,559 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 252, in <lambda>
2023-09-18 13:46:37,559 ERROR       lambda: self.handle_exception(context, e),
2023-09-18 13:46:37,560 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 383, in handle_exception
2023-09-18 13:46:37,560 ERROR       raise exception
2023-09-18 13:46:37,561 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 163, in _consume
2023-09-18 13:46:37,561 ERROR       result = handle_locally()
2023-09-18 13:46:37,562 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 252, in <lambda>
2023-09-18 13:46:37,562 ERROR       lambda: self.handle_exception(context, e),
2023-09-18 13:46:37,563 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 365, in handle_exception
2023-09-18 13:46:37,563 ERROR       raise exception
2023-09-18 13:46:37,564 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 163, in _consume
2023-09-18 13:46:37,564 ERROR       result = handle_locally()
2023-09-18 13:46:37,564 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 252, in <lambda>
2023-09-18 13:46:37,565 ERROR       lambda: self.handle_exception(context, e),
2023-09-18 13:46:37,565 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 406, in handle_exception
2023-09-18 13:46:37,566 ERROR       raise exception
2023-09-18 13:46:37,566 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 163, in _consume
2023-09-18 13:46:37,567 ERROR       result = handle_locally()
2023-09-18 13:46:37,567 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 252, in <lambda>
2023-09-18 13:46:37,568 ERROR       lambda: self.handle_exception(context, e),
2023-09-18 13:46:37,568 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 132, in handle_exception
2023-09-18 13:46:37,569 ERROR       raise exception
2023-09-18 13:46:37,569 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 163, in _consume
2023-09-18 13:46:37,570 ERROR       result = handle_locally()
2023-09-18 13:46:37,570 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 252, in <lambda>
2023-09-18 13:46:37,571 ERROR       lambda: self.handle_exception(context, e),
2023-09-18 13:46:37,571 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 132, in handle_exception
2023-09-18 13:46:37,572 ERROR       raise exception
2023-09-18 13:46:37,572 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 163, in _consume
2023-09-18 13:46:37,573 ERROR       result = handle_locally()
2023-09-18 13:46:37,573 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 252, in <lambda>
2023-09-18 13:46:37,574 ERROR       lambda: self.handle_exception(context, e),
2023-09-18 13:46:37,574 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 132, in handle_exception
2023-09-18 13:46:37,574 ERROR       raise exception
2023-09-18 13:46:37,575 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 163, in _consume
2023-09-18 13:46:37,576 ERROR       result = handle_locally()
2023-09-18 13:46:37,577 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 252, in <lambda>
2023-09-18 13:46:37,577 ERROR       lambda: self.handle_exception(context, e),
2023-09-18 13:46:37,578 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 132, in handle_exception
2023-09-18 13:46:37,578 ERROR       raise exception
2023-09-18 13:46:37,579 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 163, in _consume
2023-09-18 13:46:37,579 ERROR       result = handle_locally()
2023-09-18 13:46:37,580 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 252, in <lambda>
2023-09-18 13:46:37,580 ERROR       lambda: self.handle_exception(context, e),
2023-09-18 13:46:37,581 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 132, in handle_exception
2023-09-18 13:46:37,581 ERROR       raise exception
2023-09-18 13:46:37,582 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 163, in _consume
2023-09-18 13:46:37,582 ERROR       result = handle_locally()
2023-09-18 13:46:37,583 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 210, in <lambda>
2023-09-18 13:46:37,583 ERROR       lambda: self.handle_request(context, request_data),
2023-09-18 13:46:37,584 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/mappingprocessor.py", line 50, in handle_request
2023-09-18 13:46:37,584 ERROR       return self.mapping_engine.run(cdsinf, request_data)
2023-09-18 13:46:37,585 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/mapping.py", line 337, in run
2023-09-18 13:46:37,585 ERROR       request['specific'] = [apply_mapping(r, mapping) for r in specific]
2023-09-18 13:46:37,586 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/mapping.py", line 337, in <listcomp>
2023-09-18 13:46:37,586 ERROR       request['specific'] = [apply_mapping(r, mapping) for r in specific]
2023-09-18 13:46:37,586 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/mapping.py", line 320, in apply_mapping
2023-09-18 13:46:37,587 ERROR       raise BadRequestException("Request too large. Requesting %s items, limit is %s" % (count, selection_limit), '')
2023-09-18 13:46:37,587 ERROR   cdsinf.exceptions.BadRequestException: Request too large. Requesting 31 items, limit is 20

Accompanying data

No response

Organisation

No response