minor: how to handle bad dataset ids?

[sashakames]

I accidentally had a problem with copy and paste. see example below. In this case, the API simply doesn't retrieve those files. We should decide if a warning is appropriate and how it would be communicated. If not this is a no-op.

# Generated by pcmdi8vm.llnl.gov - 2020/07/09 16:44:38
# Search URL: http://pcmdi8vm.llnl.gov/wget?dataset_id=CMIP6.CMIP.E3SM-Project.E3SM-1-1.piControl.r1i1p1f1.Amon.cl.gr.v20191029%7Caims3.llnl.gov&dataset_id=CMIP6.CMIP.E3SM-Project.E3SM-1-1.piControl.r1i1p1f1.Amon.cli.gr.v20191029%7Caims3.llnl.gov&dataset_id=CMIP6.CMIP.E3SM-Project.E3SM-1-1.piControl.r1i1p1f1.Amon.clivi.gr.v20191029%7Caims3.llnl.govThe%20parameter%20distrib%20is%20used%20to%20enable/disable%20distributed%20search,%20where%20all%20provided%20Solr%20shards%20are
# Request method: GET
version=0.2
CACHE_FILE=.$(basename $0).status
search_url='http://pcmdi8vm.llnl.gov/wget?dataset_id=CMIP6.CMIP.E3SM-Project.E3SM-1-1.piControl.r1i1p1f1.Amon.cl.gr.v20191029%7Caims3.llnl.gov&dataset_id=CMIP6.CMIP.E3SM-Project.E3SM-1-1.piControl.r1i1p1f1.Amon.cli.gr.v20191029%7Caims3.llnl.gov&dataset_id=CMIP6.CMIP.E3SM-Project.E3SM-1-1.piControl.r1i1p1f1.Amon.clivi.gr.v20191029%7Caims3.llnl.govThe%20parameter%20distrib%20is%20used%20to%20enable/disable%20distributed%20search,%20where%20all%20provided%20Solr%20shards%20are'
request_method='GET'
file_limit=1000
distrib='true'
datasets=(
'CMIP6.CMIP.E3SM-Project.E3SM-1-1.piControl.r1i1p1f1.Amon.cl.gr.v20191029|aims3.llnl.gov'
'CMIP6.CMIP.E3SM-Project.E3SM-1-1.piControl.r1i1p1f1.Amon.cli.gr.v20191029|aims3.llnl.gov'
'CMIP6.CMIP.E3SM-Project.E3SM-1-1.piControl.r1i1p1f1.Amon.clivi.gr.v20191029|aims3.llnl.govThe parameter distrib is used to enable/disable distributed search, where all provided Solr shards are'
)

[mauzey1]

@sashakames Do the dataset ids have a pattern that could be matched to see if they are valid?

[sashakames]

@mauzey1 about to open a separate issue and never saw your comment. Yes datasets will always have a pattern. You can even get the pattern in the record: "dataset_id_template_":["%(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s"] this doesn't have the version|data_node part though. Maybe its sufficient to assume a (1) dot-delimited tuple (2) last element has .vNN where NN is a positive integer (3) followed by |<DATA_NODE>