Easy access to CORDEX datasets

[larsbuntemeyer]

Description

(copied this issue here so i can close euro-cordex/py-cordex#39)

To make research reproducible, easy access to input data under FAIR conditions is critical. For EURO-CORDEX (ca. 75TB), it's not trivial if you want to work with the whole ensemble if you have no direct access to the EURO-CORDEX ensemble dataset. Bringing EURO-CORDEX data to a cloud provider would solve lots problems concerning:

• reproducibility
• reusability
• exchange of hands on tutorials with real input data
• avoid selection bias if you have to decide on datasets at download time
• promote the datasets more easily and democratically (low-threshold)

Possible cloud providers

• Pangeo Forge (under development)
  • Proposed Recipes for Euro-Cordex pangeo-forge/staged-recipes#96
• Amazon Open Data Sponsorhip
  • e.g., NA-CORDEX
• EOSC: European Open Science Cloud
• 2i2c

This is also interesting for the Topic 4 in Pilot Lab Exascale Earth System Modelling.

Individual model restrictions

• https://is-enes-data.github.io/CORDEX_RCMs_ToU.txt
• https://github.com/IS-ENES-Data/cordex/blob/master/CORDEX_register.xlsx

[larsbuntemeyer]

see also:
https://gitlab.dkrz.de/data-infrastructure-services/zarr-in-swift-objectstorage

[larsbuntemeyer]

Development of ESGF extractions is now done here: https://github.com/jbusecke/pangeo-forge-esgf

[larsbuntemeyer]

@gnikulin @jesusff

thoughts on licenses

One more thought on licensing issues: It would be nice if CORDEX datasets (CMIP5) could be made available for unrestricted use without authorization in the future similar to the 2020 update of the CMIP5 terms of use. Since the CORDEX terms of use seemed to be inherited from CMIP5, the update might be natural. This should be at least technically and legally possible for datasets with unrestricted use. The authorization adds an unneccessary overhead of complexity for users that wish to access data automatically, e.g.,

• logon for http request ESGF/esgf-pyclient#84
• cordex data access ESGF/esgf-pyclient#69
• access failure when trying to download opendap from an specific node. ESGF/esgf-pyclient#52
• Accessing opendap datasets ESGF/esgf-pyclient#26
• Import trial of MyProxyClient in pyesgf/logon.py outputs misleading error and incompatibility with cryptography from Anaconda main channel ESGF/esgf-pyclient#83
• Dependency problem (version mismatch between pyesgf and requests library) leading to AttributeError: module 'requests_cache' has no attribute 'core' ESGF/esgf-pyclient#78
• Compatibility of LogonManager with OpenSSL 1.1.1e ESGF/esgf-pyclient#55
• etc...

I also see it this as a conflict between the desire to make CORDEX data easily accessible for users but having authorization that prevent automatic use. Especially if you want users to work with the whole ensemble, redistribution to, e.g., zarr formats could be easier...

[gnikulin]

In CORDEX we still have a number of institutions with the non-commercial ToU in contrast to CMIP5 where all institutions with the non-commercial ToU have agreed to relax their ToU to the unrestricted ones. We cannot simply remove the authorization for all CORDEX simulations. I don't know if it's technically possible to remove the authorization for the datasets with the unrestricted ToU. Even in these case users have to be authorised to download the non-commercial datasets to use full ensembles. In CORDEX-CMIP6 we definitely need to skip the authorisation and the ToU will be shown in the global attribute license similar to CMIP6.

[larsbuntemeyer]

Thanks @gnikulin ! Maybe, @agstephens could help out (any feedback would be appreciated very much!)? I am basically just throwing around ideas from my head, the question would be, if it is technically possible to remove authorization similar to CMIP5 and CMIP6 datasets in ESGF for datasets with an unrestricted CORDEX license. I guess there has to be some internal system, e.g., i can, at least, apply for CORDEX_Research and CORDEX_Commercial logins.... So there should be some list of whatsoever that distinguishes datasets based on their license...

[larsbuntemeyer]

More more thought: it would also really be helpful simply for binder based tutorial notebooks to be able to access, e.g., ESGF CORDEX opendat datasets also for users whithout openid.. This would be nice to create, e.g., Jupyter books without having to store credentials in github secrets, etc..

[larsbuntemeyer]

• The future of pangeo CMIP6 in the cloud pangeo-data/pangeo-cmip6-cloud#31

[larsbuntemeyer]

I think this pretty much sums it up for CMIP6: