Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add cop_marine provider #1131

Merged
merged 38 commits into from
Jun 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
26687eb
feat: start 'cop_marine' provider configuration
anesson-cs Apr 9, 2024
4f89ac7
feat: add product types discovery for 'StaticStacSearch' plugin
anesson-cs Apr 15, 2024
6911e67
test: add 'cop_marine' provider in test_core constants
anesson-cs Apr 15, 2024
3778755
docs: add 'cop_marine' marker in providers.yml
anesson-cs Apr 15, 2024
477125d
refactor: use cop marine items as collections in discover product types
jlahovnik May 7, 2024
b08a00d
feat: search for cop marine provider
jlahovnik May 13, 2024
93faab9
feat: search with cop marine - fixes and server mode
jlahovnik May 14, 2024
f69c12f
feat: download with copernicus marine
jlahovnik May 14, 2024
336d786
refactor: search with cop marine provider without usage of library
jlahovnik May 16, 2024
142f7b7
feat: download (library + server mode) for cop_marine provider
jlahovnik May 17, 2024
d8d1e8a
feat: handle pagination for search with cop_marine provider
jlahovnik May 21, 2024
445745d
refactor: use products from cop_marine as collections
jlahovnik May 21, 2024
0dfdf97
refactor: use more product and dataset properties for cop marine items
jlahovnik May 22, 2024
4179d89
refactor: add thumbnail and quicklook to properties for cop_marine items
jlahovnik May 23, 2024
5bedd6a
test: create tests for cop marine search
jlahovnik May 24, 2024
00121f3
test: add cop marine provider
jlahovnik May 24, 2024
0f05167
test: avoid fetching of s3 credentials for cop_marine test
jlahovnik May 24, 2024
381423c
style: fix linting issue
jlahovnik May 24, 2024
179dce4
refactor: avoid use of pytz library
jlahovnik May 24, 2024
d4f0af9
refactor: boto3 stubs
sbrunato Jun 4, 2024
e9339be
refactor: cop_marine search
sbrunato Jun 4, 2024
3f6d0aa
fix: pagination for single file items
sbrunato Jun 5, 2024
ad6e7d3
feature: add cop marine product types
jlahovnik Jun 7, 2024
526f0cc
feat: add remaining cop_marine products
jlahovnik Jun 11, 2024
ddbc516
fix: bugs after rebase
jlahovnik Jun 11, 2024
673bcb2
fix: remove * in product types description
jlahovnik Jun 11, 2024
54011dc
feat: update cop_marine product types
jlahovnik Jun 12, 2024
9cbdd0e
fix: revert server cli update
sbrunato Jun 12, 2024
9811fae
fix: search_all pagination
sbrunato Jun 5, 2024
7c0e7e0
fix: single product type discovery
sbrunato Jun 5, 2024
2d0cf6e
refactor: PrepredSearch usage and dates to utc
sbrunato Jun 12, 2024
54017b7
refactor: linter fix
sbrunato Jun 12, 2024
6d86151
refactor: mocked StaticStacSearch queries
sbrunato Jun 12, 2024
dd27081
refactor: CMEMS product types metadata
sbrunato Jun 12, 2024
c178e93
refactor: provider conf update
sbrunato Jun 12, 2024
ff70fc0
docs: add cop_marine provider
jlahovnik Jun 13, 2024
b3a8fac
docs: update documentation
sbrunato Jun 13, 2024
909dcd3
fix: cop_marine pagination
sbrunato Jun 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
264 changes: 137 additions & 127 deletions docs/_static/params_mapping_extra.csv

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/_static/params_mapping_offline_infos.json

Large diffs are not rendered by default.

94 changes: 47 additions & 47 deletions docs/_static/params_mapping_opensearch.csv

Large diffs are not rendered by default.

433 changes: 240 additions & 193 deletions docs/_static/product_types_information.csv

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/getting_started_guide/providers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Products from the following providers are made avaiable through ``eodag``:
* `ecmwf <https://www.ecmwf.int/>`_: European Centre for Medium-Range Weather Forecasts
* `cop_ads <https://ads.atmosphere.copernicus.eu>`_: Copernicus Atmosphere Data Store
* `cop_cds <https://cds.climate.copernicus.eu>`_: Copernicus Climate Data Store
* `cop_marine <https://marine.copernicus.eu>`_: Copernicus Marine Service
* `sara <https://copernicus.nci.org.au>`_: Sentinel Australasia Regional Access
* `meteoblue <https://content.meteoblue.com/en/business-solutions/weather-apis/dataset-api>`_: Meteoblue forecast
* `cop_dataspace <https://dataspace.copernicus.eu/>`_: Copernicus Data Space
Expand Down
2 changes: 2 additions & 0 deletions docs/getting_started_guide/register.rst
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ to each provider supported by ``eodag``:
Then go to your profile and use from the section named "API key" use *UID* as *username* and *API Key* as *password* in eodag credentials.
EODAG can be used to request for public datasets, you can browse them `here <https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset>`__.

* ``cop_marine``: no account is required

* ``sara``: create an account `here <https://copernicus.nci.org.au/sara.client/#/register>`__, then use your email as `username` in eodag credentials.

* ``meteoblue``: eodag uses `dataset API <https://content.meteoblue.com/en/business-solutions/weather-apis/dataset-api>`_
Expand Down
2 changes: 1 addition & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ types (Sentinel 1, Sentinel 2, Sentinel 3, Landsat, etc.) that can be searched a
`creodias <https://creodias.eu/>`_, `creodias_s3 <https://creodias.eu/>`_,
`onda <https://www.onda-dias.eu/cms/>`_, `ecmwf <https://www.ecmwf.int/>`_,
`cop_ads <https://ads.atmosphere.copernicus.eu>`_, `cop_cds <https://cds.climate.copernicus.eu>`_,
`sara <https://copernicus.nci.org.au>`_,
`cop_marine <https://marine.copernicus.eu>`_, `sara <https://copernicus.nci.org.au>`_,
`meteoblue <https://content.meteoblue.com/en/business-solutions/weather-apis/dataset-api>`_,
`cop_dataspace <https://dataspace.copernicus.eu/>`_,
`planetary_computer <https://planetarycomputer.microsoft.com/>`_,
Expand Down
21 changes: 12 additions & 9 deletions docs/notebooks/api_user_guide/2_providers_products_available.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -39,28 +39,31 @@
{
"data": {
"text/plain": [
"['astraea_eod',\n",
"['dedl',\n",
" 'dedt_lumi',\n",
" 'peps',\n",
" 'astraea_eod',\n",
" 'aws_eos',\n",
" 'cop_ads',\n",
" 'cop_cds',\n",
" 'cop_dataspace',\n",
" 'cop_marine',\n",
" 'creodias',\n",
" 'creodias_s3',\n",
" 'earth_search',\n",
" 'earth_search_cog',\n",
" 'earth_search_gcs',\n",
" 'ecmwf',\n",
" 'eumetsat_ds',\n",
" 'hydroweb_next',\n",
" 'meteoblue',\n",
" 'onda',\n",
" 'peps',\n",
" 'planetary_computer',\n",
" 'sara',\n",
" 'theia',\n",
" 'usgs',\n",
" 'usgs_satapi_aws',\n",
" 'wekeo']"
" 'wekeo',\n",
" 'wekeo_cmems']"
]
},
"execution_count": 2,
Expand Down Expand Up @@ -136,7 +139,7 @@
"\n",
"Note\n",
"\n",
"If a provider is configured to need authentication for search, and has no crendentials set, it will be pruned on EODAG initialization, and will not appear in available providers list.\n",
"If a provider is configured to need authentication for search, and has no credentials set, it will be pruned on EODAG initialization, and will not appear in available providers list.\n",
"\n",
"</div>"
]
Expand All @@ -157,14 +160,14 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"EODAG has 181 product types stored in its internal catalog.\n"
"EODAG has 245 product types stored in its internal catalog.\n"
]
}
],
Expand Down Expand Up @@ -597,7 +600,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
"version": "3.11.4"
},
"nbsphinx": {
"execute": "always"
Expand Down
14 changes: 9 additions & 5 deletions docs/notebooks/api_user_guide/3_configuration.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,17 @@
{
"data": {
"text/plain": [
"['astraea_eod',\n",
"['creodias_s3',\n",
" 'peps',\n",
" 'astraea_eod',\n",
" 'aws_eos',\n",
" 'cop_ads',\n",
" 'cop_cds',\n",
" 'cop_dataspace',\n",
" 'cop_marine',\n",
" 'creodias',\n",
" 'creodias_s3',\n",
" 'dedl',\n",
" 'dedt_lumi',\n",
" 'earth_search',\n",
" 'earth_search_cog',\n",
" 'earth_search_gcs',\n",
Expand All @@ -53,13 +57,13 @@
" 'hydroweb_next',\n",
" 'meteoblue',\n",
" 'onda',\n",
" 'peps',\n",
" 'planetary_computer',\n",
" 'sara',\n",
" 'theia',\n",
" 'usgs',\n",
" 'usgs_satapi_aws',\n",
" 'wekeo']"
" 'wekeo',\n",
" 'wekeo_cmems']"
]
},
"execution_count": 2,
Expand Down Expand Up @@ -245,7 +249,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
"version": "3.10.12"
},
"nbsphinx": {
"execute": "always"
Expand Down
14 changes: 7 additions & 7 deletions docs/notebooks/api_user_guide/4_search.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,13 @@
"output_type": "stream",
"text": [
"2024-03-27 18:19:20,816 eodag.config [INFO ] Loading user configuration from: /home/aubin/.config/eodag/eodag.yml\n",
"2024-03-27 18:19:20,830 eodag.core [INFO ] usgs: provider needing auth for search has been pruned because no crendentials could be found\n",
"2024-03-27 18:19:20,831 eodag.core [INFO ] aws_eos: provider needing auth for search has been pruned because no crendentials could be found\n",
"2024-03-27 18:19:20,832 eodag.core [INFO ] meteoblue: provider needing auth for search has been pruned because no crendentials could be found\n",
"2024-03-27 18:19:20,832 eodag.core [INFO ] hydroweb_next: provider needing auth for search has been pruned because no crendentials could be found\n",
"2024-03-27 18:19:20,833 eodag.core [INFO ] wekeo: provider needing auth for search has been pruned because no crendentials could be found\n",
"2024-03-27 18:19:20,834 eodag.core [INFO ] wekeo_cmems: provider needing auth for search has been pruned because no crendentials could be found\n",
"2024-03-27 18:19:20,835 eodag.core [INFO ] creodias_s3: provider needing auth for search has been pruned because no crendentials could be found\n",
"2024-03-27 18:19:20,830 eodag.core [INFO ] usgs: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-03-27 18:19:20,831 eodag.core [INFO ] aws_eos: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-03-27 18:19:20,832 eodag.core [INFO ] meteoblue: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-03-27 18:19:20,832 eodag.core [INFO ] hydroweb_next: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-03-27 18:19:20,833 eodag.core [INFO ] wekeo: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-03-27 18:19:20,834 eodag.core [INFO ] wekeo_cmems: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-03-27 18:19:20,835 eodag.core [INFO ] creodias_s3: provider needing auth for search has been pruned because no credentials could be found\n",
"2024-03-27 18:19:20,840 eodag.core [INFO ] Locations configuration loaded from /home/aubin/.config/eodag/locations.yml\n"
]
}
Expand Down
2 changes: 2 additions & 0 deletions docs/plugins.rst
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ The providers are implemented with a triplet of *Search/Authentication/Download*
+--------------------+-----------------------+-------------------------------+----------------+
| cop_cds | BuildSearchResult | GenericAuth | HTTPDownload |
+--------------------+-----------------------+-------------------------------+----------------+
| cop_marine | CopMarineSearch | AwsDownload | AwsAuth |
+--------------------+-----------------------+-------------------------------+----------------+
| meteoblue | BuildPostSearchResult | HttpQueryStringAuth | HTTPDownload |
+--------------------+-----------------------+-------------------------------+----------------+
| cop_dataspace | QueryStringSearch | KeycloakOIDCPasswordAuth | HTTPDownload |
Expand Down
9 changes: 6 additions & 3 deletions eodag/api/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -434,7 +434,7 @@ def _prune_providers_list(self) -> None:
)
update_needed = True
logger.info(
"%s: provider needing auth for search has been pruned because no crendentials could be found",
"%s: provider needing auth for search has been pruned because no credentials could be found",
provider,
)
elif hasattr(conf, "search") and getattr(conf.search, "need_auth", False):
Expand Down Expand Up @@ -462,7 +462,7 @@ def _prune_providers_list(self) -> None:
)
update_needed = True
logger.info(
"%s: provider needing auth for search has been pruned because no crendentials could be found",
"%s: provider needing auth for search has been pruned because no credentials could be found",
provider,
)
elif not hasattr(conf, "api") and not hasattr(conf, "search"):
Expand Down Expand Up @@ -1794,7 +1794,10 @@ def _do_search(
max_items_per_page = getattr(search_plugin.config, "pagination", {}).get(
"max_items_per_page", DEFAULT_MAX_ITEMS_PER_PAGE
)
if kwargs.get("items_per_page", DEFAULT_ITEMS_PER_PAGE) > max_items_per_page:
if (
kwargs.get("items_per_page", DEFAULT_ITEMS_PER_PAGE) > max_items_per_page
and max_items_per_page > 0
):
logger.warning(
"EODAG believes that you might have asked for more products/items "
"than the maximum allowed by '%s': %s > %s. Try to lower "
Expand Down
2 changes: 1 addition & 1 deletion eodag/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,7 @@ class OrderStatus(TypedDict):
pagination: PluginConfig.Pagination
sort: PluginConfig.Sort
query_params_key: str
discover_metadata: Dict[str, str]
discover_metadata: Dict[str, Union[str, bool]]
discover_product_types: Dict[str, Any]
discover_queryables: Dict[str, Any]
metadata_mapping: Dict[str, Union[str, List[str]]]
Expand Down
4 changes: 2 additions & 2 deletions eodag/plugins/search/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ class PreparedSearch:
"""An object collecting needed information for search."""

product_type: Optional[str] = None
page: Optional[int] = DEFAULT_PAGE
items_per_page: Optional[int] = DEFAULT_ITEMS_PER_PAGE
page: int = DEFAULT_PAGE
items_per_page: int = DEFAULT_ITEMS_PER_PAGE
auth: Optional[Union[AuthBase, Dict[str, str]]] = None
auth_plugin: Optional[Authentication] = None
count: bool = True
Expand Down
Loading
Loading