Test failed due to unsorted url input list

[jbusecke]

I just investigated a case of a store failing the tests.

Loading the store and checking the time dimension:

import xarray as xr
import matplotlib.pyplot as plt
path = 'gs://leap-persistent-ro/data-library/cmip6-testing/a618127503-6017922106-1/CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp245.r17i1p1f1.day.pr.gn.v20210901.zarr'
ds = xr.open_dataset(path, engine='zarr', chunks={})
plt.figure()
plt.plot(ds.time.data)

plt.figure()
plt.plot(ds.sortby('time').time.data)

suggest that these chunks of time are just out of order.

And in fact calling pangeo-forge-esgf with the latest commit form the PR branch:

! pip install git+https://github.com/jbusecke/pangeo-forge-esgf.git@15a752839db4f19375f528ce757c613ddf2881ef #commit to reproduce 
from pangeo_forge_esgf import get_urls_from_esgf
url_dict = await get_urls_from_esgf(['CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp245.r17i1p1f1.day.pr.gn.v20210901'])
url_dict['CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp245.r17i1p1f1.day.pr.gn.v20210901']

gives a list that is not properly ordered.

['https://esgf-data1.llnl.gov/thredds/fileServer/css03_data/CMIP6/ScenarioMIP/MPI-M/MPI-ESM1-2-LR/ssp245/r17i1p1f1/day/pr/gn/v20210901/pr_day_MPI-ESM1-2-LR_ssp245_r17i1p1f1_gn_20750101-20941231.nc',
 'https://esgf-data1.llnl.gov/thredds/fileServer/css03_data/CMIP6/ScenarioMIP/MPI-M/MPI-ESM1-2-LR/ssp245/r17i1p1f1/day/pr/gn/v20210901/pr_day_MPI-ESM1-2-LR_ssp245_r17i1p1f1_gn_20350101-20541231.nc',
 'https://esgf-data1.llnl.gov/thredds/fileServer/css03_data/CMIP6/ScenarioMIP/MPI-M/MPI-ESM1-2-LR/ssp245/r17i1p1f1/day/pr/gn/v20210901/pr_day_MPI-ESM1-2-LR_ssp245_r17i1p1f1_gn_20550101-20741231.nc',
 'https://esgf-data1.llnl.gov/thredds/fileServer/css03_data/CMIP6/ScenarioMIP/MPI-M/MPI-ESM1-2-LR/ssp245/r17i1p1f1/day/pr/gn/v20210901/pr_day_MPI-ESM1-2-LR_ssp245_r17i1p1f1_gn_20150101-20341231.nc',
 'https://esgf-data1.llnl.gov/thredds/fileServer/css03_data/CMIP6/ScenarioMIP/MPI-M/MPI-ESM1-2-LR/ssp245/r17i1p1f1/day/pr/gn/v20210901/pr_day_MPI-ESM1-2-LR_ssp245_r17i1p1f1_gn_20950101-21001231.nc']

With this fix the results get returned properly ordered.

['https://esgf-data1.llnl.gov/thredds/fileServer/css03_data/CMIP6/ScenarioMIP/MPI-M/MPI-ESM1-2-LR/ssp245/r17i1p1f1/day/pr/gn/v20210901/pr_day_MPI-ESM1-2-LR_ssp245_r17i1p1f1_gn_20150101-20341231.nc',
 'https://esgf-data1.llnl.gov/thredds/fileServer/css03_data/CMIP6/ScenarioMIP/MPI-M/MPI-ESM1-2-LR/ssp245/r17i1p1f1/day/pr/gn/v20210901/pr_day_MPI-ESM1-2-LR_ssp245_r17i1p1f1_gn_20350101-20541231.nc',
 'https://esgf-data1.llnl.gov/thredds/fileServer/css03_data/CMIP6/ScenarioMIP/MPI-M/MPI-ESM1-2-LR/ssp245/r17i1p1f1/day/pr/gn/v20210901/pr_day_MPI-ESM1-2-LR_ssp245_r17i1p1f1_gn_20550101-20741231.nc',
 'https://esgf-data1.llnl.gov/thredds/fileServer/css03_data/CMIP6/ScenarioMIP/MPI-M/MPI-ESM1-2-LR/ssp245/r17i1p1f1/day/pr/gn/v20210901/pr_day_MPI-ESM1-2-LR_ssp245_r17i1p1f1_gn_20750101-20941231.nc',
 'https://esgf-data1.llnl.gov/thredds/fileServer/css03_data/CMIP6/ScenarioMIP/MPI-M/MPI-ESM1-2-LR/ssp245/r17i1p1f1/day/pr/gn/v20210901/pr_day_MPI-ESM1-2-LR_ssp245_r17i1p1f1_gn_20950101-21001231.nc']

Since I am not able to investigate which stores were erroneously marked as failing due to this. I will erase the nonqc bigquery table and resubmit all recipes.

[jbusecke]

This makes me think @cisaacstern if this is something that might have some application upstream? I guess -recipes strictly requires ordered urls, right?