How will Coverage Processing be supported?

[ghobona]

It is foreseeable that a future extension of OGC API - Coverages may offer coverage processing capabilities similar to those offered by WCPS.

So what would such an extension look like?

Please coordinate with the OGC API - Processes SWG on this.

[pebau]

@ghobona seems like a low-hanging fruit:
https://github.com/opengeospatial/ogc_api_coverages/blob/master/extensions/processing.adoc

[jerstlouis]

@ghobona @bpross-52n @Schpidi

In the previous issue ( #79 ) which is exactly about this, I propose that this can nicely integrate with OGC API - Processes, e.g. as a process at .../processes/wcps that specifies that one of its input is WCPS code.

Also considering uploading WCPS code as a deployment method (called Processes transactional right now, though I really think the name transaction is miused and it should be deployment) to create a specific process.

This integration to act upon different data source would require substitution capabilities which we are investigating as part of our Modular OGC API Workflows project (generic WCPS code which would work on different collections, and also considering different AoI / scale and formats for MOAW workflows).

[nmtoken]

I'm not sure why you would have .../processes/wcps (or expanding the acronym .../processes/web_coverage_processing_service) it's not like there's .../processes/wps

That doesn't seem like integration of WPS and WCPS into a single form.

[jerstlouis]

@nmtoken Well the idea is that processing is a general concept for generic processing, while a WCPS process (integrated within that generic processing framework) would be able to perform processing on coverages, with code supplied in a specific WCPS language as a parameter. As I mentioned, a service could also potentially be able to create a number of different processes by uploading (generic) WCPS code.

[nmtoken]

@jerstlouis but it could be integrated

POST /processes/{process-id}/jobs

Executes a process, i.e. creates a new job. Inputs and outputs will have to be specified in a JSON document

like:

{
"q": "for $c in (ERA5 - cube), $d in (DEM) return encode($c.temperature > 20 and $d > 100, 'image/png')"
}

and

GET /processes

Could list the WCPS processioning capability (equivalent to <ows:Profile>http://www.opengis.net/spec/WCPS/1.0/conf/wcps-core</ows:Profile> in a WCS GetCapabilities response etc.)

GET /processes/{process-id}

Could return a detailed description of the WCPS functionality i.e the supported coverage processing expressions

[jerstlouis]

@nmtoken The integration is the idea, yes.
But /processes should still be a generic OGC API - Processes end-point that vanilla OGC API - Processes clients can use, but discover a WCPS process, and learn that they need to pass a WCPS query in one of the inputs for that WCPS process.

I see both processes at /processes/{processId} as being defined by specific (but parameterizable) WCPS code (e.g. an NDVI algorithm deployed to a deployment-enabled (currently called Transactional) Processes API as WCPS code), as well as a generic WCPS runner process e.g. at /processes/wcps.

[pebau]

Maybe we are seeing two perspectives here:

• WCPS, being part of the WCS suite, is a data service: derive data from data in a flexible way. Hence - and this fits the RESTful paradigm - it is aka "modifier" to coverage data sets (where "data set" is understood in the most general way, no particular definition assumed)
• WPS capitalizes on the processing aspect (which is abhored by REST for all that I know), making algorithms first-class citizens.

IMHO Jerome has shown a way forward by seeing WCPS as one class of algorithms that can be offered via the WPS protocol, in addition to the pre-existing view. This to me induces a natural view of WCPS as an OAPI-Coverages extension (is planned already, actually) plus an embedding in WPS.

This also reflects past work: WPCS is industrially offered via WCS, and a WPS binding has been done earlier. So we know it works :)

[jerstlouis]

@pebau
Regarding WCPS being a data service, while it is true that it allows to access specific data via sub-setting/slicing/trimming/scaling of the data, most of that should be doable with conformance classes outside of the OGC API - Coverages - Processing extension (e.g. through query parameters). When doing more advanced processing and modifications, WCPS really starts being more of a processing service than a data service, and I think it could really shine as such, especially if it embeds well within OGC API - Processes (and considering untying WCPS queries from specific data layers and data ranges via parameterization, and the whole modular workflows magic).

This to me induces a natural view of WCPS as an OAPI-Coverages extension (is planned already, actually) plus an embedding in [OAPI-Processes].

Yes, exactly, but that OGC API - Coverages extension for processing could potentially have a very light dependency on the very core of OGC API - Processes (i.e. simply a link to /processes from the landing page, and in there a link to that generic WCPS process where one can submit WCPS code). With almost no effort this could make it readily usable in clients implementing OGC API - Processes, and paves the way for the implementers should they wish to implement OGC API - Processes more in depth (e.g. process description etc.).

[pebau]

@jerstlouis The data view we should not give up while exploring how to make it also available via WPS. One way to easily marry both is that, based on the OAPI-Coverages definition, there is a conformance class in OAPI-Processes which says "/processing/wcps?q=X" is equivalent to "/collections?q=X". That would be the most lightweight thing I could think of.

[ghobona]

Due to the need to keep Part 1: Core specifications simple enough for the basic developer to use, it would be advisable to plan for an OGC API - Coverages - Part 2: Processing standard that specifies how to use a Coverage Processing Language (CPL) which would be a renaming of WCPS.

This is just a thought.

[jerstlouis]

@pebau I really don't think /collections?q=X is appropriate, because doing a GET on that resource should always return you a listing of collections|coverage|geospatial data. A GET on that resource should not be used to do processing and return results instead, and passing large piece of WCPS code as a query parameter is really not ideal. POSTing the WCPS code to /processing/wcps could offer you that exact same functionality instead. But I am curious to hear what others think.

@ghobona I agree that WCPS (the language) is a processing language rather than a service. A service supporting queries in the language is a service :)

[pebau]

@ghobona I would strongly object to renaming. WCPS must remain WCPS. (Remember the confusion that had been caused by WMS flipping default coordinate sequence.) As we talk about extensions implementers anyway can freely choose what to support. And renaming would not make it simpler to implement.

[pebau]

@jerstlouis WCPS GET is working flawlessly. If it is to be constrained in a WPS environment, that's a decision of the WPS carrier protocol. But for the OAPI-Coverages, why should we constrain.

And I am absolutely sure that I would immediately loose overview on how to use the API if now functionality depends on GET vs POST or ..., it would explode complexity.

[Schpidi]

20200914: Coverages SWG: Discussion resulted in this proposal:

• /wcps for sending WCPS encoded GET requests and getting the evaluation result.
• /processes/wcps for using the vanilla OAPIProcesses approach

[pebau]

friendly amendment:

• /wcps could also be POST or SOAP etc, not necessarily only GET; critical is: request returns evaluation result
• /processing/wcps should, as I understand discussion, be able to host any number of (possibly parametrized) queries in a WPS manner, how can this be invoked? Options might include: allow other names than wcps, or allow sub-paths.

[jerstlouis]

@pebau regarding /processes/wcps, the idea was this could work with either the core OGC API - Processes, or the Modular OGC API Workflow extension approach (or a server could support both).

In both cases, the user could either supply the WCPS code along with the request, or deploy WCPS code as a specific process at an arbitrary /processes/{processId} end-point.

If supplying the WCPS code along with the request (to a generic /processes/wcps WCPS coverage processor), the parameterizable WCPS code is one of the inputs of a process execution document, and input data can be provided as additional inputs. If the WCPS code is instead deployed as a process, then one would do a PUT on /processes/{processId} with parameterizable WCPS code.

With core Processes, the execution document is submitted as a POST request to /processes/{processId}/jobs. In sync mode the output is returned right away, while in async a job is created and can be polled, or a URL can be provided as a callback.

In the MOAW Workflow extension approach, depending on the path used either the workflow is set up in a deferred manner, in which case a description document is returned (e.g. for /processes/{processId} a collection/coverage description document, similar to what you would find at a regular /collections/{coverageId}, with the various links to domain set, range type, coverage which can be accessed with subset / scaling etc.), or in the case of a synchronous path data is returned right away and parameters for subset/scaling can be included already with the request (e.g. for /processes/{processId}/coverage).

[pebau]

@jerstlouis ok, I see you have a solution in mind already. Let me suggest to have the plain /wcps part in the OAPI-Cov section and the /processing/wcps in the OAPI-Proc section, would that make sense?

[jerstlouis]

@pebau While the general mechanics of Processing is defined in the core OGC API - Processes, and the modular workflow extension would be defined in a Workflow extension to Processes, for the /processes/wcps end-point an extension to Coverages would need to define how WCPS can be parameterized and used to deploy processes or included as an input to a generic WCPS coverage processor for process execution, and those coverages-specific paths which can be added e.g. to /processes/{processId} when using the workflow extensions.

For the /wcps end-point, from what I understand there might not be anything to define at all, because you can simply reference the existing Web Coverage Processing Service standard. We could describe it using OpenAPI though, and a server could integrate that OpenAPI definition for the /wcps end-point within the overall /api of the service. The OGC API - Coverages might serve as a way to discover coverages and their domain set / range type etc. for use with that WCPS though.

For the MOAW project I would be interested in us prototyping the /processes/wcps end-point mechanics (with parameterized WCPS code) as a facade on top of a rasdaman /wcps end-point, thus really demonstrating both approaches.

[ghobona]

2020-09-15 OGC API Cross Cutting Issues session during OGC Member Meeting

The issue was discussed and there was no objection to the proposed approach.

[ghobona]

Given the outcome of the 2020-09-15 OGC API Cross Cutting Issues session, I propose that this issue be closed at the next OGC API - Coverages SWG meeting.

[pebau]

So I understand there will be the (approved) /wcps together with a process-oriented /processing/... - would make sense to me indeed.

[pebau]

@ghobona as you close it, can we summarize the result for the generations to come? I understand this is supporting /wcps and /processing/wcps .

[jerstlouis]

@ghobona @pebau

The /wcps end-point can be defined, following current WCPS standard.

And here is an example OGC API - Processes / Workflows approach to Coverage processing, using generic expressions:

{
    "process": "https://example.com/ogcapi/processes/coverage-processor",
    "inputs": {
        "data" : [
           { "collection": "https://someserver.com/ogcapi/collections/sentinel2" },
           { "collection": "https://someserver.com/ogcapi/collections/landsat8" }
        ],
        "bandsExpressions" : {
            "value" : {
                "ndvi" : "(ds[0].B08 - ds[0].B04 ) / (ds[0].B04 + ds[0].B08)",
                "lsb4" : "ds[1].B04 / 4000"
            }
        }
    },
    "mode": "collection"
}

In Workflows, the area / time range of interest / format is provided separately, e.g. via a subsequent OGC API - Coverages request on the resulting virtual collection.

Different coverage processes can exist supporting different languages, e.g. WCPS, Python, etc.

[pebau]

@jerstlouis thanks for summarizing! I guess the WPS group will want to update their specs, I can take care of the WCPS part (which requires little change, if at all - will check).

[jerstlouis]

@pebau A number of OGC API - Processes issues have recently been resolved bringing the specifications in line with this, but some updates may still be pending.
That example above however is specific to Part 3 - Workflows, which adds the collection types of inputs. and currently makes some things optional like mode and outputs.
However now that we have a single execution end-point, we will probably want to re-introduce mode for Workflows as well with new enumeration values such as:

• collection (get back a document describing a virtual collection, like you would find at /collections/{collectionId})
• landingPage (get back an OGC API landing page, allowing the process to generate more than a single virtual collection, and providing fully OGC API-compliant results)

So I will add mode back in there.
outputs would still be made optional by Workflows, since the follow-up GET request will be specifying the collection of interest.