Need support for asynchronous coverage requests

[pomakis]

Coverages are often many terabytes in size, and therefore it can often take minutes or hours to satisfy a rangeset request (even a subsetted one). This causes HTTP user agents and servers to time out long before the request is satisfied.

CubeWerx resolved this issue in our WCS implementation by adding a vendor-specific "responseHandler" parameter to the GetCoverge operation. This allows a client to request a coverage with "REQUEST=GetCoverage&...&responseHandler=mailto:me@somewhere.com". The client gets an immediate response back indicating that the request has been submitted, and then at some future point in time an e-mail message gets sent to the specified e-mail address that provides either a link to the requested coverage or an indication of failure.

There needs to be a way to do something like this in "OGC API - Coverages".

[cmheazel]

@pomakis In the Pub-Sub Whitepaper I propose development of a Pub-Sub extension to API-Common based on the architecture patterns documentable through AsyncAPI. Such an extension would provide a common Pub-Sub capability which can be leveraged by all OGC Web API standards.

[cmheazel]

@pomakis The call-back pattern is a different issue. OpenAPI provides a pattern for call-backs and, as you note, we have implementations. I suggest that we add development of the following extensions to the Charter:

1. Call-back extension (basic asynchronous)
2. Pub-Sub extension (event driven asynchronous)
3. Discovery (integration with STAC)

[pvretano]

I would not call "responseHandler" a vendor-specific parameter. It has been part of OGC specifications since at least 2012 (see http://docs.opengeospatial.org/is/12-176r7/12-176r7.html#86) and has been tested in many testbeds. The latest incarnation of responseHandler can be found here: https://docs.opengeospatial.org/per/18-045.html#async_extension which would make a very nice, light-weight, building block for OWS common.

[chris-little]

Whichever way the async response handing goes, I would like to confirm that there would be no exclusion of underlying protocols, as an alternative to HTTP(S), such as MQTT, AMQP or other wire-level protocols?

[cmheazel]

@chris-little Absolutely!

[cmheazel]

It seems to me that most of a large CIS coverage can be handled using the paging technique described in Common. Complications arise when we get to the Range Set. Range Sets are often designed for a specific access technique (COG, JPIP, etc.). So the "paging" technique for a Range Set is specific to the Range Type.

One option is to return a striped-down landing page. This landing page would include the conformance classes, an API definition for the protocol, and a link to the access end-point. The server has the option of returning the Range Set or, if it judges that to be too large, a 303 redirect to the landing page.

[pebau]

By way of history, early on I had suggested to establish a WPS profile of WCS for async handling. After quite some OAB discussions it was determined that WPS is not the best way for this, and all went dormant. Later PubSub came out, and talks with its writers revealed that this indeed could be promising. So I am all for doing a WCS-Async extension, it's waiting since long - in the spirit of the existing ones, ie: protocol independent and hence including OAPI-Coverages.

[jerstlouis]

The latest discussions across the OGC API is leading towards the use of the Prefer header.

We would include a recommendation that if a Prefer header is provided, then the server should honor the preference which may indicate a preference for an async response, or to wait up to X seconds for a synchronous response, otherwise fall back to asyncronous. With Processes - Workflows, this also supports the async retrieval of coverage data resulting from processing. This could also be applied to a WCPS end-point. We would need to define how the 201 response works and how the server can poll and/or specify a callback to retrieve the results once they are ready.

[cmheazel]

API-Common has a proposed simple async module that can be used for this purpose. This is an adaptation of the response handler feature that @pvretano referred to. In addition, the OpenAPI Callback element should be able to model this functionality.
Between paging and response handlers I think we have this one covered (at least for HTTP).