From 9b3f63e12d43673506938c84e2bae5c9d84544da Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 20 Jun 2024 17:51:25 +0200 Subject: [PATCH 1/4] Add Remote UDP extension #515 --- CHANGELOG.md | 5 +++ README.md | 1 + extensions/remote-udp/README.md | 65 +++++++++++++++++++++++++++++++++ 3 files changed, 71 insertions(+) create mode 100644 extensions/remote-udp/README.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 7622bc24..baf9432a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## Unreleased / Draft +### Added + +- **New extensions:** + - [Remote UDP Extension](./extensions/remote-udp/README.md) + ### Fixed - `GET /file_formats`: Base paramater on top of normal JSON Schema, not Process JSON Schema diff --git a/README.md b/README.md index aef37924..2a9ac639 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ See also the [changelog](CHANGELOG.md) and the [milestones](https://github.com/O | ---------------------------------------------- | ------- | ------------ | ----------- | | [Commercial Data](extensions/commercial-data/) | 0.1.0 | experimental | Provides an interface for discovering, ordering and using commercial data. | | [Federation](extensions/federation/) | 0.1.0 | experimental | Covers federation aspects, i.e. where multiple back-ends are exposed as a single API. | +| [Remote UDP](extensions/remote-udp/) | 0.1.0 | experimental | Load user-defined processes that are hosted externally through the process namespace into process graphs. | ## Repository diff --git a/extensions/remote-udp/README.md b/extensions/remote-udp/README.md new file mode 100644 index 00000000..3ffa1e1a --- /dev/null +++ b/extensions/remote-udp/README.md @@ -0,0 +1,65 @@ +# Remote UDP Extension + +The openEO API is a specification for interoperable cloud-based processing of large Earth observation datasets. + +This extension enables user to load user-defined processes that are hosted on other hosts through the process namespace into process graphs. + +- Version: **0.1.0** +- Stability: **experimental** +- Conformance class: `https://api.openeo.org/extensions/remote-udp/0.1.0` + +## Justification + +The openEO API defines the `namespace` property in a process node of a process graph as follows: +> The following options are predefined by the openEO API, but additional namespaces may be introduced by back-ends or in a future version of the API. +> * `null` [...] +> * `backend` [...] +> * `user` [...] + +This makes it possible that this extension adds additional allowed values to the `namespace` property. + +## Specification + +This extension extends the `namespace` property of process graph nodes so that it accepts **absolute** URL with the protocols `https` (**recommended**) and `http` (discouraged). The URLs specified MUST either return + +1. a single process (compatible to the endpoint `GET /process_graphs/{process_graph_id}`) +2. a list of processes (compatible to the endpoint `GET /process_graphs`) + +If a URL is provided for the `namespace` property, the `id` property of the process graph node MUST be set to the `id` of the process. + +### Compatibility + +Compatible means in this context that the requests and responses must comply to the openEO API specification with the following exceptions: + +- The `Authorization` header is not required and will usually not be sent. +- Lists of processes MUST NOT paginate and the full process description MUST be provided for each process (i.e., the recommendation to omit large properties such as `process_graph` doesn't apply). + +### Client Considerations + +Clients MUST only offer this functionality to users if the conformance class of this extension is listed in the `conformsTo` property of the `GET /` endpoint. + +The protocol `http` is discouraged for URLs as web-based clients may not be able to retrieve HTTP URLs from a HTTPS context. +For the same reason it is also RECOMMENDED to enable CORS for all URLs. + +### Error Handling + +The following error SHOULD be reported if the namespace can't be resolved: + +- Code: `ProcessNamespaceInvalid` +- Message: `The value passed for namespace '{namespace}' in process '{process}' is invalid: {reason}` +- HTTP Status Code: 400 + +## Example + +An exemplary process graph node: + +```json +{ + "process_id": "echo", + "namespace": "https://hub.openeo.org/processes/echo", + "arguments": { + "message": "Hello World" + }, + "result": true +} +``` From 92a7e560d6a11f48a31a3d3893c9602507df8657 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Fri, 28 Jun 2024 15:50:47 +0200 Subject: [PATCH 2/4] Apply suggestions from code review Co-authored-by: Stefaan Lippens --- extensions/remote-udp/README.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/extensions/remote-udp/README.md b/extensions/remote-udp/README.md index 3ffa1e1a..27e535b8 100644 --- a/extensions/remote-udp/README.md +++ b/extensions/remote-udp/README.md @@ -2,7 +2,7 @@ The openEO API is a specification for interoperable cloud-based processing of large Earth observation datasets. -This extension enables user to load user-defined processes that are hosted on other hosts through the process namespace into process graphs. +This extension enables user to load user-defined processes that are hosted external to the openEO API (e.g., GitHub or cloud storage) through the process namespace into process graphs. - Version: **0.1.0** - Stability: **experimental** @@ -16,22 +16,24 @@ The openEO API defines the `namespace` property in a process node of a process g > * `backend` [...] > * `user` [...] -This makes it possible that this extension adds additional allowed values to the `namespace` property. +This makes it possible for this extension to add additional allowed values to the `namespace` property. ## Specification -This extension extends the `namespace` property of process graph nodes so that it accepts **absolute** URL with the protocols `https` (**recommended**) and `http` (discouraged). The URLs specified MUST either return +This extension extends the `namespace` property of process graph nodes so that it accepts **absolute** URL with the protocols `https` (**recommended**) and `http` (discouraged). The URLs specified MUST return one of the following two options: -1. a single process (compatible to the endpoint `GET /process_graphs/{process_graph_id}`) -2. a list of processes (compatible to the endpoint `GET /process_graphs`) - -If a URL is provided for the `namespace` property, the `id` property of the process graph node MUST be set to the `id` of the process. +1. A single process, compatible to the endpoint `GET /process_graphs/{process_graph_id}`. + In this case, the `id` property of the process graph node MUST be equal to the `id` of the process, + otherwise a `ProcessNamespaceInvalid` error is thrown +2. A list of processes, compatible to the endpoint `GET /process_graphs`. + In this case, the `id` property of the process graph node is used to identify the process from the list. + If not found a `ProcessNamespaceInvalid` error is thrown ### Compatibility Compatible means in this context that the requests and responses must comply to the openEO API specification with the following exceptions: -- The `Authorization` header is not required and will usually not be sent. +- The `Authorization` header MUST NOT not be sent. - Lists of processes MUST NOT paginate and the full process description MUST be provided for each process (i.e., the recommendation to omit large properties such as `process_graph` doesn't apply). ### Client Considerations @@ -59,7 +61,7 @@ An exemplary process graph node: "namespace": "https://hub.openeo.org/processes/echo", "arguments": { "message": "Hello World" - }, + }, "result": true } ``` From b3c7cef59a76066e29d1c81748ab4e61e64bb483 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Fri, 28 Jun 2024 16:12:50 +0200 Subject: [PATCH 3/4] Updates according to code review --- extensions/remote-udp/README.md | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/extensions/remote-udp/README.md b/extensions/remote-udp/README.md index 27e535b8..795955ee 100644 --- a/extensions/remote-udp/README.md +++ b/extensions/remote-udp/README.md @@ -22,23 +22,25 @@ This makes it possible for this extension to add additional allowed values to th This extension extends the `namespace` property of process graph nodes so that it accepts **absolute** URL with the protocols `https` (**recommended**) and `http` (discouraged). The URLs specified MUST return one of the following two options: -1. A single process, compatible to the endpoint `GET /process_graphs/{process_graph_id}`. +1. A single process, compatible\* to the endpoint `GET /process_graphs/{process_graph_id}`. In this case, the `id` property of the process graph node MUST be equal to the `id` of the process, otherwise a `ProcessNamespaceInvalid` error is thrown -2. A list of processes, compatible to the endpoint `GET /process_graphs`. +2. A list of processes, compatible\* to the endpoint `GET /process_graphs`. In this case, the `id` property of the process graph node is used to identify the process from the list. If not found a `ProcessNamespaceInvalid` error is thrown -### Compatibility +\* Compatible means in this context that the requests and responses must comply to the openEO API specification with the following exceptions: -Compatible means in this context that the requests and responses must comply to the openEO API specification with the following exceptions: - -- The `Authorization` header MUST NOT not be sent. -- Lists of processes MUST NOT paginate and the full process description MUST be provided for each process (i.e., the recommendation to omit large properties such as `process_graph` doesn't apply). +- User credentials / tokens that are obtained through the openEO API MUST NOT not be sent to URIs that are external to the openEO API. + The requirement to provide an `Authorization` header for the respective endpoints doesn't apply. +- For a list of processes, the full process description MUST be provided for the process with the given ID within the first request. + This means that the recommendation to omit large roperties such as `process_graph` doesn't apply. + It also requires that the requester doesn't need to paginate through additional pages to find the process with the given ID. + Ideally, the list of processes is not paginated as otherwise the process with the given ID may move to other pages over time. ### Client Considerations -Clients MUST only offer this functionality to users if the conformance class of this extension is listed in the `conformsTo` property of the `GET /` endpoint. +If a client is conncected to a specific backend, the client MUST only offer this functionality to users if the conformance class of this extension is listed in the `conformsTo` property of the `GET /` endpoint. The protocol `http` is discouraged for URLs as web-based clients may not be able to retrieve HTTP URLs from a HTTPS context. For the same reason it is also RECOMMENDED to enable CORS for all URLs. @@ -57,11 +59,11 @@ An exemplary process graph node: ```json { - "process_id": "echo", - "namespace": "https://hub.openeo.org/processes/echo", - "arguments": { - "message": "Hello World" - }, - "result": true + "process_id": "echo", + "namespace": "https://hub.openeo.org/processes/echo", + "arguments": { + "message": "Hello World" + }, + "result": true } ``` From 7e11ff9f318986abf8779b9474fcf1a5731bf45b Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 1 Jul 2024 15:05:47 +0200 Subject: [PATCH 4/4] Update extensions/remote-udp/README.md --- extensions/remote-udp/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions/remote-udp/README.md b/extensions/remote-udp/README.md index 795955ee..812568bc 100644 --- a/extensions/remote-udp/README.md +++ b/extensions/remote-udp/README.md @@ -60,7 +60,7 @@ An exemplary process graph node: ```json { "process_id": "echo", - "namespace": "https://hub.openeo.org/processes/echo", + "namespace": "https://openeo.example/processes/echo", "arguments": { "message": "Hello World" },