EURO-CORDEX to the cloud ☁️

[larsbuntemeyer]

Description

(copied this issue here so i can close euro-cordex/py-cordex#39)

To make research reproducible, easy access to input data under FAIR conditions is critical. For EURO-CORDEX (ca. 75TB), it's not trivial if you want to work with the whole ensemble if you have no direct access to the EURO-CORDEX ensemble dataset. Bringing EURO-CORDEX data to a cloud provider would solve lots problems concerning:

• reproducibility
• reusability
• exchange of hands on tutorials with real input data
• avoid selection bias if you have to decide on datasets at download time
• promote the datasets more easily and democratically (low-threshold)

Possible cloud providers

• Pangeo Forge (under development)
  • Proposed Recipes for Euro-Cordex pangeo-forge/staged-recipes#96
• Amazon Open Data Sponsorhip
  • e.g., NA-CORDEX

This is also interesting for the Topic 4 in Pilot Lab Exascale Earth System Modelling.

Individual model restrictions

• https://is-enes-data.github.io/CORDEX_RCMs_ToU.txt
• https://github.com/IS-ENES-Data/cordex/blob/master/CORDEX_register.xlsx

[larsbuntemeyer]

see also:
https://gitlab.dkrz.de/data-infrastructure-services/zarr-in-swift-objectstorage

[larsbuntemeyer]

Development of ESGF extractions is now done here: https://github.com/jbusecke/pangeo-forge-esgf