Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(providers): wekeo split to wekeo_main and wekeo_ecmwf #1214

Merged
merged 19 commits into from
Aug 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
cbb037c
fix: add authentication for queryables discovery
anesson-cs Jun 14, 2024
dd0ee73
feat: add logs when queryables discovery with 'StacSearch' plugin is …
anesson-cs Jun 14, 2024
7c118d8
feat: split wekeo provider into two providers
anesson-cs Jun 14, 2024
a8f12b9
test: adapt all necessary tests to the two new providers
anesson-cs Jun 14, 2024
2a5bed2
docs: update some tutorials
anesson-cs Jun 14, 2024
96f8264
fix: remove useless parameter in 'wekeo_ecmwf' configuration
anesson-cs Jun 14, 2024
de9c5a4
fix: add 'MARK' comment on wekeo_cmems provider configuration
anesson-cs Jun 17, 2024
cf77299
fix: make product types fetching and discovery works for grouped prov…
anesson-cs Jun 21, 2024
136307d
test: update and add tests for grouped providers
anesson-cs Jun 21, 2024
ba1308f
fix: change the name of 'wekeo_sentinel' to 'wekeo_main''
anesson-cs Jun 24, 2024
7bb4b4b
docs: complete description of wekeo provider
anesson-cs Jun 24, 2024
4502e9b
test: fix end to end test of wekeo_main provider
anesson-cs Jun 24, 2024
579d98f
test: add tests on the new search plugin created for wekeo_main
anesson-cs Jun 25, 2024
65d78cc
test: add checks about wekeo providers on two tests
anesson-cs Jun 25, 2024
b04b3e9
docs: update a notebook after the rebase
anesson-cs Aug 2, 2024
a0a3c41
refactor(core): product types fetch for groups
sbrunato Aug 8, 2024
9b42aaf
refactor: docs and tests
sbrunato Aug 8, 2024
f2d6d41
docs: fix plugins table
sbrunato Aug 8, 2024
32a4b05
docs: pruned providers notification
sbrunato Aug 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
486 changes: 243 additions & 243 deletions docs/_static/product_types_information.csv

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion docs/getting_started_guide/providers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ Products from the following providers are made avaiable through ``eodag``:
* `cop_dataspace <https://dataspace.copernicus.eu/>`_: Copernicus Data Space
* `planetary_computer <https://planetarycomputer.microsoft.com/>`_: Microsoft Planetary Computer
* `hydroweb_next <https://hydroweb.next.theia-land.fr>`_: hydroweb.next thematic hub for hydrology data access
* `wekeo <https://www.wekeo.eu>`_: WEkEO Copernicus and Sentinel data
* `wekeo_main <https://www.wekeo.eu/>`_: WEkEO Copernicus Sentinel, DEM, and CLMS data
* `wekeo_ecmwf <https://www.wekeo.eu/>`_: WEkEO ECMWF data
* `wekeo_cmems <https://www.wekeo.eu>`_: Copernicus Marine (CMEMS) data from WEkEO
* `dedt_lumi <https://polytope.lumi.apps.dte.destination-earth.eu/openapi>`_: DestinE Digital Twin output on Lumi
* `dedl <https://hda.data.destination-earth.eu/ui>`_: Destination Earth Data Lake (DEDL)
Expand Down
6 changes: 4 additions & 2 deletions docs/getting_started_guide/register.rst
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ to each provider supported by ``eodag``:

* ``earth_search_cog``: no authentication needed.

* ``wekeo``: you need an access token to authenticate and to accept terms and conditions with it:
* ``wekeo_main``: you need an access token to authenticate and to accept terms and conditions with it:

* Create an account on `WEkEO <https://www.wekeo.eu/register>`__

Expand Down Expand Up @@ -150,7 +150,9 @@ to each provider supported by ``eodag``:
* Copernicus_DEM_Instance_COP-DEM-GLO-30-F_Global_30m
* Copernicus_DEM_Instance_COP-DEM-GLO-90-F_Global_90m

* ``wekeo_cmems``: The registration procedure is the same as for ``wekeo``. The licence that has to be accepted to access the Copernicus Marine data is ``Copernicus_Marine_Service_Product_License``.
* ``wekeo_ecmwf``: The registration procedure is the same as for ``wekeo_main``.

* ``wekeo_cmems``: The registration procedure is the same as for ``wekeo_main``. The licence that has to be accepted to access the Copernicus Marine data is ``Copernicus_Marine_Service_Product_License``.

* ``dedt_lumi``: Create an account on `DestinE <https://platform.destine.eu/>`__, then use your `username`, `password` in eodag credentials.

Expand Down
2 changes: 1 addition & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ types (Sentinel 1, Sentinel 2, Sentinel 3, Landsat, etc.) that can be searched a
`cop_dataspace <https://dataspace.copernicus.eu/>`_,
`planetary_computer <https://planetarycomputer.microsoft.com/>`_,
`hydroweb_next <https://hydroweb.next.theia-land.fr>`_,
`wekeo <https://www.wekeo.eu/>`_,
`wekeo_main <https://www.wekeo.eu/>`_, `wekeo_ecmwf <https://www.wekeo.eu/>`_,
`wekeo_cmems <https://www.wekeo.eu>`_,
`dedt_lumi <https://polytope.lumi.apps.dte.destination-earth.eu/openapi>`_,
`dedl <https://hda.data.destination-earth.eu/ui>`_,
Expand Down
205 changes: 129 additions & 76 deletions docs/notebooks/api_user_guide/2_providers_products_available.ipynb

Large diffs are not rendered by default.

31 changes: 16 additions & 15 deletions docs/notebooks/api_user_guide/3_configuration.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -40,24 +40,31 @@
"text/plain": [
"['peps',\n",
" 'astraea_eod',\n",
" 'aws_eos',\n",
" 'cop_ads',\n",
" 'cop_cds',\n",
" 'cop_dataspace',\n",
" 'cop_marine',\n",
" 'creodias',\n",
" 'creodias_s3',\n",
" 'dedt_lumi',\n",
" 'earth_search',\n",
" 'earth_search_cog',\n",
" 'earth_search_gcs',\n",
" 'ecmwf',\n",
" 'eumetsat_ds',\n",
" 'hydroweb_next',\n",
" 'meteoblue',\n",
" 'onda',\n",
" 'planetary_computer',\n",
" 'sara',\n",
" 'theia',\n",
" 'usgs_satapi_aws',\n",
" 'wekeo']"
" 'wekeo_cmems',\n",
" 'wekeo_ecmwf',\n",
" 'wekeo_main',\n",
" 'creodias_s3',\n",
" 'dedl',\n",
" 'usgs']"
]
},
"execution_count": 2,
Expand Down Expand Up @@ -212,7 +219,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -222,30 +229,24 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2024-07-11 12:10:43,346 eodag.config [INFO ] Loading user configuration from: /home/sylvain/.config/eodag/eodag.yml\n",
"2024-07-11 12:10:43,401 eodag.core [INFO ] usgs: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-07-11 12:10:43,402 eodag.core [INFO ] aws_eos: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-07-11 12:10:43,404 eodag.core [INFO ] meteoblue: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-07-11 12:10:43,406 eodag.core [INFO ] hydroweb_next: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-07-11 12:10:43,407 eodag.core [INFO ] wekeo_cmems: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-07-11 12:10:43,408 eodag.core [INFO ] dedl: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-07-11 12:10:43,421 eodag.core [INFO ] Locations configuration loaded from /home/sylvain/.config/eodag/locations.yml\n"
"2024-08-02 15:30:40,234 eodag.config [INFO ] Loading user configuration from: /home/anesson/.config/eodag/eodag.yml\n",
"2024-08-02 15:30:40,277 eodag.core [INFO ] Locations configuration loaded from /home/anesson/.config/eodag/locations.yml\n"
]
},
{
"data": {
"text/plain": [
"<eodag.api.core.EODataAccessGateway at 0x7e69cad185e0>"
"<eodag.api.core.EODataAccessGateway at 0x7f7857c36700>"
]
},
"execution_count": 11,
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
Expand Down Expand Up @@ -278,7 +279,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
"version": "3.8.10"
},
"nbsphinx": {
"execute": "always"
Expand Down
84 changes: 21 additions & 63 deletions docs/notebooks/tutos/tuto_wekeo.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,11 @@
"source": [
"# The wekeo provider in EODAG\n",
"\n",
"This tutorial will show you how use EODAG to download data from [WEkEO](https://www.wekeo.eu/) using `eodag`. WEkEO offers a wide range of Copernicus and Sentinel products."
"This tutorial will show you how use EODAG to download data from [WEkEO](https://www.wekeo.eu/) using `eodag`. WEkEO offers a wide range of Copernicus and Sentinel products decomposed into three providers classified according to their kind of data:\n",
"\n",
"* `wekeo_main`: Sentinel and some Copernicus data\n",
"* `wekeo_ecmwf`: Copernicus data from ECMWF\n",
"* `wekeo_cmems`: Copernicus Marine data"
]
},
{
Expand All @@ -32,8 +36,8 @@
"# instantiate EODAG API\n",
"dag = EODataAccessGateway()\n",
"\n",
"# Authenticate on wekeo\n",
"auth = dag._plugins_manager.get_auth_plugin(\"wekeo\").authenticate()\n",
"# Authenticate on the wekeo adapted to your request, here wekeo_main\n",
"auth = dag._plugins_manager.get_auth_plugin(\"wekeo_main\").authenticate()\n",
"licence_name = \"EUMETSAT_Copernicus_Data_Licence\"\n",
"\n",
"# Use your authentication to accept Terms and Conditions\n",
Expand All @@ -47,16 +51,15 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Depending on which data you want to use, different licences have to be accepted. If you want to use all the product types available in EODAG for the provider `wekeo`, the following licences have to be accepted:\n",
"Depending on which data you want to use, different licences have to be accepted. If you want to use all the product types available in EODAG for the provider `wekeo_main`, the following licences have to be accepted:\n",
" \n",
"* EUMETSAT_Copernicus_Data_Licence\n",
"* Copernicus_Land_Monitoring_Service_Data_Policy\n",
"* Copernicus_Sentinel_License\n",
"* Copernicus_ECMWF_License\n",
"* Copernicus_DEM_Instance_COP-DEM-GLO-30-F_Global_30m\n",
"* Copernicus_DEM_Instance_COP-DEM-GLO-90-F_Global_90m\n",
"\n",
"To access the products of the provider `wekeo_cmems` the licence Copernicus_Marine_Service_Product_License has to be accepted."
"To access the products of the provider `wekeo_ecmwf` or `wekeo_cmems` the licence Copernicus_ECMWF_License or Copernicus_Marine_Service_Product_License have to be accepted respectively."
]
},
{
Expand All @@ -70,7 +73,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Many product types are available for `wekeo` provider:"
"Many product types are available for the three `wekeo` providers, here let's see for `wekeo_ecmwf`:"
]
},
{
Expand All @@ -83,7 +86,7 @@
{
"data": {
"text/plain": [
"81"
"48"
]
},
"execution_count": 2,
Expand All @@ -92,7 +95,7 @@
}
],
"source": [
"wekeo_product_types = dag.list_product_types(\"wekeo\")\n",
"wekeo_product_types = dag.list_product_types(\"wekeo_ecmwf\")\n",
"len(wekeo_product_types)"
]
},
Expand All @@ -106,7 +109,8 @@
{
"data": {
"text/plain": [
"['CAMS_EAC4',\n",
"['AG_ERA5',\n",
" 'CAMS_EAC4',\n",
" 'CAMS_EAC4_MONTHLY',\n",
" 'CAMS_EU_AIR_QUALITY_FORECAST',\n",
" 'CAMS_EU_AIR_QUALITY_RE',\n",
Expand All @@ -118,19 +122,6 @@
" 'CAMS_GRF',\n",
" 'CAMS_GRF_AUX',\n",
" 'CAMS_SOLAR_RADIATION',\n",
" 'CLMS_CORINE',\n",
" 'CLMS_GLO_DMP_333M',\n",
" 'CLMS_GLO_FAPAR_333M',\n",
" 'CLMS_GLO_FCOVER_333M',\n",
" 'CLMS_GLO_GDMP_333M',\n",
" 'CLMS_GLO_LAI_333M',\n",
" 'CLMS_GLO_NDVI_1KM_LTS',\n",
" 'CLMS_GLO_NDVI_333M',\n",
" 'COP_DEM_GLO30_DGED',\n",
" 'COP_DEM_GLO30_DTED',\n",
" 'COP_DEM_GLO90_DGED',\n",
" 'COP_DEM_GLO90_DTED',\n",
" 'EEA_DAILY_VI',\n",
" 'EFAS_FORECAST',\n",
" 'EFAS_HISTORICAL',\n",
" 'EFAS_REFORECAST',\n",
Expand All @@ -150,33 +141,12 @@
" 'GLOFAS_SEASONAL',\n",
" 'GLOFAS_SEASONAL_REFORECAST',\n",
" 'GRIDDED_GLACIERS_MASS_CHANGE',\n",
" 'S1_SAR_GRD',\n",
" 'S1_SAR_OCN',\n",
" 'S1_SAR_RAW',\n",
" 'S1_SAR_SLC',\n",
" 'S2_MSI_L1C',\n",
" 'S2_MSI_L2A',\n",
" 'S2_MSI_L2AP',\n",
" 'S3_EFR',\n",
" 'S3_ERR',\n",
" 'S3_LAN_HY',\n",
" 'S3_LAN_LI',\n",
" 'S3_LAN_SI',\n",
" 'S3_OLCI_L2LFR',\n",
" 'S3_OLCI_L2LRR',\n",
" 'S3_OLCI_L2WFR',\n",
" 'S3_OLCI_L2WRR',\n",
" 'S3_SLSTR_L1RBT',\n",
" 'S3_SLSTR_L2',\n",
" 'S3_SRA',\n",
" 'S3_SRA_A',\n",
" 'S3_SRA_BS',\n",
" 'S3_WAT',\n",
" 'S5P_L1B_IR_ALL',\n",
" 'S5P_L2_IR_ALL',\n",
" 'SATELLITE_CARBON_DIOXIDE',\n",
" 'SATELLITE_FIRE_BURNED_AREA',\n",
" 'SATELLITE_METHANE',\n",
" 'SATELLITE_SEA_ICE_CONCENTRATION',\n",
" 'SATELLITE_SEA_ICE_EDGE_TYPE',\n",
" 'SATELLITE_SEA_ICE_THICKNESS',\n",
" 'SATELLITE_SEA_LEVEL_BLACK_SEA',\n",
" 'SATELLITE_SEA_LEVEL_GLOBAL',\n",
" 'SATELLITE_SEA_LEVEL_MEDITERRANEAN',\n",
Expand Down Expand Up @@ -213,7 +183,7 @@
{
"data": {
"text/plain": [
"SearchResult([EOProduct(id=4bd37b17b45d146f378e349ab3f90464, provider=wekeo)])"
"SearchResult([EOProduct(id=cems-glofas-forecast-9ef890e45d8d9c86746601c50f9f20a7, provider=wekeo_ecmwf)])"
]
},
"execution_count": 4,
Expand All @@ -227,7 +197,7 @@
" start=\"2023-01-01\", \n",
" end=\"2023-01-02\", \n",
" geom=[0.25, 43.2, 2.8, 43.9],\n",
" provider=\"wekeo\"\n",
" provider=\"wekeo_ecmwf\"\n",
")\n",
"products"
]
Expand All @@ -247,7 +217,7 @@
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "9925bcf0ed804fd0bbec2f5efe4ea519",
"model_id": "92740041a22f496ab5d85297a1748a8c",
"version_major": 2,
"version_minor": 0
},
Expand All @@ -258,22 +228,10 @@
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"[Retry #1] Waiting 11.999954s until next download try for ordered product (retry every 0.2' for 20')"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"'/tmp/4bd37b17b45d146f378e349ab3f90464'"
"'/tmp/cems-glofas-forecast-9ef890e45d8d9c86746601c50f9f20a7'"
]
},
"execution_count": 5,
Expand Down
Loading
Loading