Use `intake` for data fetching

[VeckoTheGecko]

Intake is a lightweight package for finding, investigating, loading and disseminating data.

Currently we ingest data using copernicusmarine to ingest data. As we look to expand the abilities of virtualship, we will need to also manage data that is from other sources (which could get complicated).

Migrating our approach to use an intake catalogue will allow us to:

• consolidate the datasets used in virtualship in a declarative manner complete with metadata, that is also freely available for users to use and explore
• ingest datasets from various sources
• use datasets in a lazier manner from remote storage (data doesn't per-se have to be eagerly downloaded in future¹)

Intake is well adopted in the Pangeo community.

My understanding of intake at the moment is somewhat limited - still need to investivate

Additional context
Add any other context or screenshots about the feature request here.

Footnotes

1. Currently parcels v3 expects that the data is on disk, but perhaps in v4 this doesn't have to be the case ↩

[VeckoTheGecko]

Did some investigation here. This is what I thought we would initially do:

1. Use pangeo-forge to provide the copernicusmarine data
   a. paper: Pangeo Forge: Crowdsourcing Analysis-Ready, Cloud Optimized Data Production
   • TLDR; Pangeo Forge is an open source framework for Extracting, Transforming, and Loading (ETL) scientific data, specifically focusing on multidimensional array data like NetCDF, GRIB, and HDF. Inspired by the successful crowdsourcing model of Conda Forge, its primary objective is to simplify the process of taking data from traditional archives and transforming it into analysis-ready, cloud-optimized (ARCO) formats, such as Zarr, which are then stored in cloud object storage. This transformation is defined by community members, referred to as "recipe developers," who write Python-based ETL pipelines called "recipes" using the pangeo-forge-recipes SDK. These recipes can be run on shared cloud infrastructure known as Bakeries, abstracting the complexities of distributed computing and cloud automation from the scientists who will use the data. By crowdsourcing the production of ARCO datasets, Pangeo Forge aims to significantly accelerate scientific research, foster open science practices, and broaden participation in data-intensive environmental science by making large datasets more accessible and easier to work with in a cloud environment

b. Pangeo Forge turns datasets from original providers into ARCO (analysis ready, cloud optimised) datasets. These ARCO formats can be lazily loaded by xarray.
c. If CopernicusMarine data is available via Pangeo Forge, then we can point to this dataset for the data fetching (this would be compatible with intake) while pointing to other datasets
2. everything can be collated into a single intake catalogue

Roadblocks:

• (1b) Although it is true that Pangeo Forge provides the tooling and recipes to do this, it doesn't provide the compute (source) . Initially I thought they just provided a public bucket with the data - their documentation doesn't explicitly mention that they only provide the
• (1c) CopernicusMarine only promotes ingesting their data via their Python package (though this can be done lazily via:

ds = copernicusmarine.open_dataset(
    "cmems_mod_glo_phy-thetao_anfc_0.083deg_PT6H-i",
    username=username,
    password=password,
)
ds.sel(depth=slice(0, 10), time=slice("2022-01-01", "2023-01-02"))

)

• (2) given the roadblocks above, I don't think we can bring everything into a single intake catalogue. Saying that, a combination of intake + copernicus (or just pure intake if we don't use copernicus datasets, instead opting for open data via AWS Open Data )

[j-atkins]

I agree this is a good time to start considering these things, especially considering the plans to expand out to new sources of data in the very near future (e.g. CPR). I'm not too familiar with intake but it sounds promising for what we want to do!