feat: hydroweb_next as new provider (#711)

docs/getting_started_guide/providers.rst

@@ -26,6 +26,7 @@ Products from the following providers are made avaiable through ``eodag``:
 * `meteoblue <https://content.meteoblue.com/en/business-solutions/weather-apis/dataset-api>`_: Meteoblue forecast
 * `cop_dataspace <https://dataspace.copernicus.eu/>`_: Copernicus Data Space
 * `planetary_computer <https://planetarycomputer.microsoft.com/>`_: Microsoft Planetary Computer
+* `hydroweb_next <https://hydroweb.next.theia-land.fr>`_: Hydroweb.next hydrology data acces
 
 Providers available through an external plugin:
 

docs/getting_started_guide/register.rst

@@ -8,45 +8,49 @@ the users obtain a set of credentials (e.g. login/password, API key, etc.). Thes
 need to be provided to ``eodag`` (see :ref:`configure`). The list below explains how to register
 to each provider supported by ``eodag``:
 
-* `usgs`: create an account  `here <https://ers.cr.usgs.gov/register/>`__ and then `request an access <https://ers.cr.usgs.gov/profile/access>`_ to the `Machine-to-Machine (M2M) API <https://m2m.cr.usgs.gov/>`_.
+* ``usgs``: create an account  `here <https://ers.cr.usgs.gov/register/>`__ and then `request an access <https://ers.cr.usgs.gov/profile/access>`_ to the `Machine-to-Machine (M2M) API <https://m2m.cr.usgs.gov/>`_.
   Product requests can be performed once access to the M2M API has been granted to you.
 
-* `theia`: create an account `here <https://sso.theia-land.fr/theia/register/register.xhtml>`__
+* ``theia``: create an account `here <https://sso.theia-land.fr/theia/register/register.xhtml>`__
 
-* `peps`: create an account `here <https://peps.cnes.fr/rocket/#/register>`__, then use your email as `username` in eodag credentials.
+* ``peps``: create an account `here <https://peps.cnes.fr/rocket/#/register>`__, then use your email as `username` in eodag credentials.
 
-* `creodias`: create an account `here <https://portal.creodias.eu/register.php>`__
+* ``creodias``: create an account `here <https://portal.creodias.eu/register.php>`__
 
-* `onda`: create an account `here: <https://www.onda-dias.eu/cms/>`__
+* ``onda``: create an account `here: <https://www.onda-dias.eu/cms/>`__
 
-* `mundi`: create an account `here <https://mundiwebservices.com>`__ (click on "login" and then go in the "register" tab).
+* ``mundi``: create an account `here <https://mundiwebservices.com>`__ (click on "login" and then go in the "register" tab).
   Then use as *apikey* the Web Token provided `here <https://mundiwebservices.com/account/profile>`__
 
-* `ecmwf`: create an account `here <https://apps.ecmwf.int/registration/>`__.
+* ``ecmwf``: create an account `here <https://apps.ecmwf.int/registration/>`__.
   Then use *email* as *username* and *key* as *password* from `here <https://api.ecmwf.int/v1/key/>`__ in eodag credentials.
   EODAG can be used to request for public datasets as for operational archive. Please note that for public datasets you
   might need to accept a license (e.g. for `TIGGE <https://apps.ecmwf.int/datasets/data/tigge/licence/>`__)
 
-* `cop_ads`: create an account `here <https://ads.atmosphere.copernicus.eu/user/register>`__.
+* ``cop_ads``: create an account `here <https://ads.atmosphere.copernicus.eu/user/register>`__.
   Then go to your profile and use from the section named "API key" the *UID* as *username* and *API Key* as *password* in eodag credentials.
   EODAG can be used to request for public datasets, you can browse them `here <https://ads.atmosphere.copernicus.eu/cdsapp#!/search?type=dataset>`__.
 
-* `cop_cds`: create an account `here <https://cds.climate.copernicus.eu/user/register>`__.
+* ``cop_cds``: create an account `here <https://cds.climate.copernicus.eu/user/register>`__.
   Then go to your profile and use from the section named "API key" use *UID* as *username* and *API Key* as *password* in eodag credentials.
   EODAG can be used to request for public datasets, you can browse them `here <https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset>`__.
 
-* `sara`: create an account `here <https://copernicus.nci.org.au/sara.client/#/register>`__, then use your email as `username` in eodag credentials.
+* ``sara``: create an account `here <https://copernicus.nci.org.au/sara.client/#/register>`__, then use your email as `username` in eodag credentials.
 
-* `meteoblue`: eodag uses `dataset API <https://content.meteoblue.com/en/business-solutions/weather-apis/dataset-api>`_
+* ``meteoblue``: eodag uses `dataset API <https://content.meteoblue.com/en/business-solutions/weather-apis/dataset-api>`_
   which requires the access level `Access Gold <https://content.meteoblue.com/en/business-solutions/weather-apis/pricing>`_.
   Contact `support@meteoblue.com <mailto:support@meteoblue.com>`_ to apply for a free API key trial.
 
-* `cop_dataspace`: create an account `here <https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/auth?client_id=cdse-public&redirect_uri=https%3A%2F%2Fdataspace.copernicus.eu%2Fbrowser%2F&response_type=code&scope=openid>`__
+* ``cop_dataspace``: create an account `here <https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/auth?client_id=cdse-public&redirect_uri=https%3A%2F%2Fdataspace.copernicus.eu%2Fbrowser%2F&response_type=code&scope=openid>`__
 
-* `planetary_computer`: most datasets are anonymously accessible, but a subscription key may be needed to increase `rate limits and access private datasets <https://planetarycomputer.microsoft.com/docs/concepts/sas/#rate-limits-and-access-restrictions>`_.
+* ``planetary_computer``: most datasets are anonymously accessible, but a subscription key may be needed to increase `rate limits and access private datasets <https://planetarycomputer.microsoft.com/docs/concepts/sas/#rate-limits-and-access-restrictions>`_.
   Create an account `here <https://planetarycomputer.microsoft.com/account/request>`__, then view your keys by signing in with your Microsoft account `here <https://planetarycomputer.developer.azure-api.net/>`__.
 
-* `aws_eos`: you need credentials for both EOS (search) and AWS (download):
+* ``hydroweb_next``: Go to `https://hydroweb.next.theia-land.fr <https://hydroweb.next.theia-land.fr>`_, then login or
+  create an account by clicking on ``Log in`` in the top-right corner. Once logged-in, create an API key in the user
+  settings page, and used it as *apikey* in EODAG provider auth credentials.
+
+* ``aws_eos``: you need credentials for both EOS (search) and AWS (download):
 
   * Create an account on `EOS <https://auth.eos.com>`__
 
@@ -66,7 +70,7 @@ to each provider supported by ``eodag``:
 
     EOS free account is limited to 100 requests.
 
-* `astraea_eod, earth_search, usgs_satapi_aws`: you need AWS credentials for download:
+* ``astraea_eod``, ``earth_search``, ``usgs_satapi_aws``: you need AWS credentials for download:
 
   * Create an account on `AWS <https://aws.amazon.com/>`__
 
@@ -83,7 +87,7 @@ to each provider supported by ``eodag``:
     A credit card number must be provided when creating an AWS account because fees apply
     after a given amount of downloaded data.
 
-* `earth_search_gcs`: you need HMAC keys for Google Cloud Storage:
+* ``earth_search_gcs``: you need HMAC keys for Google Cloud Storage:
 
   * Sign in using a `google account <https://accounts.google.com/signin/v2/identifier>`__.
 

docs/index.rst

@@ -36,7 +36,8 @@ types (Sentinel 1, Sentinel 2, Sentinel 3, Landsat, etc.) that can be searched a
       `sara <https://copernicus.nci.org.au>`_,
       `meteoblue <https://content.meteoblue.com/en/business-solutions/weather-apis/dataset-api>`_,
       `cop_dataspace <https://dataspace.copernicus.eu/>`_,
-      `planetary_computer <https://planetarycomputer.microsoft.com/>`_
+      `planetary_computer <https://planetarycomputer.microsoft.com/>`_,
+      `hydroweb_next <https://hydroweb.next.theia-land.fr>`_
 
 EODAG has the following primary features:
 

docs/notebooks/api_user_guide/2_providers_products_available.ipynb

@@ -9,7 +9,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
+   "execution_count": 1,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -33,7 +33,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": 2,
    "metadata": {},
    "outputs": [
     {
@@ -49,6 +49,7 @@
        " 'earth_search_cog',\n",
        " 'earth_search_gcs',\n",
        " 'ecmwf',\n",
+       " 'hydroweb_next',\n",
        " 'meteoblue',\n",
        " 'mundi',\n",
        " 'onda',\n",
@@ -60,7 +61,7 @@
        " 'usgs_satapi_aws']"
       ]
      },
-     "execution_count": 24,
+     "execution_count": 2,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -72,14 +73,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 25,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "eodag has 19 providers already configured.\n"
+      "eodag has 20 providers already configured.\n"
      ]
     }
    ],
@@ -96,7 +97,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 26,
+   "execution_count": 4,
    "metadata": {},
    "outputs": [
     {
@@ -153,7 +154,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 27,
+   "execution_count": 5,
    "metadata": {},
    "outputs": [
     {
@@ -171,14 +172,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 28,
+   "execution_count": 6,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "EODAG has 126 product types stored in its extended catalog, after having fetched providers.\n"
+      "EODAG has 231 product types stored in its extended catalog, after having fetched providers.\n"
      ]
     }
    ],
@@ -196,14 +197,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 29,
+   "execution_count": 7,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "list_product_types() keeps returning 126 product types.\n"
+      "list_product_types() keeps returning 231 product types.\n"
      ]
     }
    ],
@@ -214,7 +215,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 30,
+   "execution_count": 8,
    "metadata": {},
    "outputs": [
     {
@@ -244,7 +245,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 31,
+   "execution_count": 9,
    "metadata": {},
    "outputs": [
     {
@@ -374,7 +375,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 32,
+   "execution_count": 10,
    "metadata": {},
    "outputs": [
     {
@@ -409,7 +410,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 33,
+   "execution_count": 11,
    "metadata": {},
    "outputs": [
     {
@@ -439,14 +440,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 34,
+   "execution_count": 12,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "The provider with the largest number of product types is 'creodias' with 61.\n"
+      "The provider with the largest number of product types is 'planetary_computer' with 105.\n"
      ]
     }
    ],

docs/notebooks/api_user_guide/3_configuration.ipynb

@@ -48,6 +48,7 @@
        " 'earth_search_cog',\n",
        " 'earth_search_gcs',\n",
        " 'ecmwf',\n",
+       " 'hydroweb_next',\n",
        " 'meteoblue',\n",
        " 'mundi',\n",
        " 'onda',\n",
@@ -139,7 +140,7 @@
     {
      "data": {
       "text/plain": [
-       "('tamn', 6)"
+       "('tamn', 2)"
       ]
      },
      "execution_count": 5,
@@ -197,15 +198,15 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2023-02-23 15:55:07,900 eodag.config                     [INFO    ] (config           ) Loading user configuration from: /home/sylvain/.config/eodag/eodag.yml\n",
-      "2023-02-23 15:55:08,006 eodag.core                       [DEBUG   ] (core             ) Opening product types index in /home/sylvain/.config/eodag/.index\n",
-      "2023-02-23 15:55:08,010 eodag.core                       [INFO    ] (core             ) Locations configuration loaded from /home/sylvain/.config/eodag/locations.yml\n"
+      "2023-04-14 14:49:24,033 eodag.config                     [INFO    ] (config           ) Loading user configuration from: /home/sylvain/.config/eodag/eodag.yml\n",
+      "2023-04-14 14:49:24,237 eodag.core                       [DEBUG   ] (core             ) Opening product types index in /home/sylvain/.config/eodag/.index\n",
+      "2023-04-14 14:49:24,245 eodag.core                       [INFO    ] (core             ) Locations configuration loaded from /home/sylvain/.config/eodag/locations.yml\n"
      ]
     },
     {
      "data": {
       "text/plain": [
-       "<eodag.api.core.EODataAccessGateway at 0x7fd035786bb0>"
+       "<eodag.api.core.EODataAccessGateway at 0x7f37e28b19d0>"
       ]
      },
      "execution_count": 7,
@@ -216,13 +217,6 @@
    "source": [
     "EODataAccessGateway()"
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {

docs/plugins.rst

@@ -68,6 +68,8 @@ The providers are implemented with a triplet of *Search/Authentication/Download*
 +--------------------+-----------------------+--------------------------+----------------+
 | planetary_computer | StacSearch            | SASAuth                  | HTTPDownload   |
 +--------------------+-----------------------+--------------------------+----------------+
+| hydroweb_next      | StacSearch            | HTTPHeaderAuth           | HTTPDownload   |
++--------------------+-----------------------+--------------------------+----------------+
 
 .. _creating_plugins:
 

eodag/resources/providers.yml

@@ -3618,7 +3618,7 @@
   roles:
     - host
   description: Planetary Computer
-  url:  https://planetarycomputer.microsoft.com
+  url: https://planetarycomputer.microsoft.com
   search: !plugin
     type: StacSearch
     api_endpoint: https://planetarycomputer.microsoft.com/api/stac/v1/search
@@ -3651,3 +3651,37 @@
     signed_url_key: href
     headers:
       Ocp-Apim-Subscription-Key: "{apikey}"
+---
+!provider
+  name: hydroweb_next
+  priority: 0
+  roles:
+    - host
+  description: Hydroweb.next
+  url: https://hydroweb.next.theia-land.fr
+  search: !plugin
+    type: StacSearch
+    api_endpoint: https://hydroweb.next.theia-land.fr/api/v1/rs-catalog/stac/search
+    need_auth: true
+    auth_error_code: 401
+    pagination:
+      max_items_per_page: 10_000
+    metadata_mapping:
+      startTimeFromAscendingNode:
+        - '{{"query":{{"end_datetime":{{"gte":"{startTimeFromAscendingNode#to_iso_utc_datetime}"}}}}}}'
+        - '$.properties.start_datetime'
+      completionTimeFromAscendingNode:
+        - '{{"query":{{"start_datetime":{{"lte":"{completionTimeFromAscendingNode#to_iso_utc_datetime}"}}}}}}'
+        - '$.properties.end_datetime'
+  products:
+    GENERIC_PRODUCT_TYPE:
+      productType: '{productType}'
+  download: !plugin
+    type: HTTPDownload
+    base_uri: https://hydroweb.next.theia-land.fr
+    flatten_top_dirs: true
+    auth_error_code: 401
+  auth: !plugin
+    type: HTTPHeaderAuth
+    headers:
+      X-API-Key: "{apikey}"

eodag/resources/user_conf_template.yml

@@ -190,3 +190,11 @@ planetary_computer:
             apikey:
     download:
         outputs_prefix:
+hydroweb_next:
+    priority: # Lower value means lower priority (Default: 0)
+    search:   # Search parameters configuration
+    auth:
+        credentials:
+            apikey:
+    download:
+        outputs_prefix:

tests/context.py

@@ -81,6 +81,7 @@
     flatten_top_directories,
     deepcopy,
     cached_parse,
+    sanitize,
 )
 from eodag.utils.exceptions import (
     AddressNotFound,

tests/resources/wrong_credentials_conf.yml

@@ -62,3 +62,8 @@ meteoblue:
     auth:
         credentials:
             apikey: "wrong_apikey"
+
+hydroweb_next:
+    auth:
+        credentials:
+            apikey: "wrong_apikey"