Migration/support of Process Description using JSON schema

[christophenoel]

During the Hackaton discussion, we have agreed migrating the JSON Process DEscription to a JSON schema is a great idea.

My understanding of the replacement is the following:

• Some JSON Schema templates/building blocks would define what the "Process Description" JSON Schema should comply to (simple profile).
• In the OGC Processing API, the execute definition request becomes the provided JSON Schema Process Description. How do we define that in the OpenAPI spec ? (generic object type or something smarter ?) The JSON Schema also defines the outputs format, i.e. the jobs/result document

I tried to express the process definition (inputs, outputs and metadata) using a JSON Schema definition instead of “process description” traditional document (see below my quick hackaton example). Some issues encountered when mapping Processing Description to JSON Schema:

1. Title and description can only be mapped to the single description property
2. No metadata can be provided. Can we extend JSON Schema ? This is a critical point ! Inputs may require either constraints (spatial, temporal), indication how to retrieve/find resources for this input (e.g. ows Context can provide this information), or client useful information. How can we provide such metadata in a JSON Schema ? If this cannot be achieved, I’m not in favour of adopting the JSON schema format.
3. For LiteralData types, allowed values cannot be provided. An alternative is the pattern regexp (e.g. pattern: '^\d{3}-\d{2}-\d{4}$')
4. Process Description spec includes for each input and output the supported mimeTypes (and default). OpenAPI only allows defining mime type definition for the whole request/response but not for the individual input/output (binary) files. How can we achieve provide this information in the schema itself ? Can we extend JSON schema ? This is a critical feature, and OGC Testbed 15 is exploring how to map compatible processes (matching input/output between processes).

Do you have workaround for those issues ?

openapi: 3.0.0
info:
  title: Hackaton OpenAPI Process Description Example
  version: "Hackaton-processExamplle"
  description: 'GeomatysNDVIMultiSensor Process (example taken from Testbed 14)'
  contact:
    name: Open Geospatial Consortium
    email: standards@opengeospatial.org
    url: 'http://www.opengeospatial.org'
  license:
    name: CC-BY 4.0 license
    url: 'https://creativecommons.org/licenses/by/4.0/'
components:
  schemas:
    execute:
      type: object
      required:
        - mode
        - response
        # indicates mandatory inputs
        - input1
      properties:
        #We provide all the inputs directly as properties, some examples below
        #Case of an input array of files (binary or documents)
        input1:
          type: array
          description: this is possible to provide the abstract in here
          # cardinality can be expressed as follows (min/maxOccurs)
          minItems: 1
          maxItems: 10
          items:
            # Using the valueType allow to preserve that the prefered mimeType can be provided 
            # and that the file can be provided raw or by reference (prefered)
            $ref: 'valueType.yaml'
        # output 2 and 3 are  a simple params (literal data)      
        input2:
          type: string
        input3: 
          type: boolean
        # output 4 is date with temporal constraints
        # output 5 is integer with constraints
        input5:
          type: integer
          minimum: 1
          maximum: 100
          default: 20
        # output 6 is string with default value and allowed values
        input6: 
          type: string
          default: 'test'
          
        # How do we specify the required output mime types
        outputs:
          type: array
          items:
            $ref: '#/components/schemas/output'
        # If server supports the transmissionMode (sync, async), a conformance req is required
        mode:
          $ref: 'transmissionMode.yaml'
        # If server supports the response format (raw,doc) , a conformance req is required. 
        response:
          $ref: 'responseMode.yaml'
    #to be improved
    result:
      type: object
      properties:
        # defines a single output which is an array of files
        output1:
          type: array
          items:
            # Using the referenceValue type allows to return the mime type information (raw or by ref)
            $ref: 'valueType.yaml'

[christophenoel]

A path forward for describing supported mimeType (topic 4) is by using the format (which can be extended) property.

        input1b:
          type: array
          description: this is possible to provide the abstract in here
          # cardinality can be expressed as follows (min/maxOccurs)
          minItems: 1
          maxItems: 10
          items:
            # Using the valueType allow to preserve that the prefered mimeType can be provided 
            # and that the file can be provided raw or by reference (prefered)
            type:
              oneOf:
                - type: string
                  format: 'image/tiff'
                - type: string
                  format: 'image/png'

The same applies for output of course.

[christophenoel]

For providing metadata (= web client information, any other stuff), in JSON Schema it is not an error to include undefined keywords.

So we may define:

• a generic element (e.g based on addtionalProperties) for providing any kind of key value pair information
• take the opportunity to identify usual blocks of information.

Some regular kind of information:

• OWS context (related to an input) for integration with other OGC service (e.G. how can I retrieve resource for this input from a WMS/WFS/etc.)
• CWL (or equivalent) information for execution invocation (typically met transactional processing: for any command-line based and container applications --> It describe how to pass files/parameters to the command-line when executing the process).
• Spatial/temporal constraints on a particular input

[m-mohr]

During the Hackaton discussion, we have agreed migrating the JSON Process DEscription to a JSON schema is a great idea.

I tried to come up with a simplified JSON Schema: https://github.com/Open-EO/openeo-processes/issues/64

For reference, how we describe processes in openEO with a custom Schema + JSON Schema for parameters (inputs) and return values (outputs): https://open-eo.github.io/openeo-api/v/0.4.1/apireference/#tag/Process-Discovery/paths/~1processes/get

• In the OGC Processing API, the execute definition request becomes the provided JSON Schema Process Description. How do we define that in the OpenAPI spec ? (generic object type or something smarter ?) The JSON Schema also defines the outputs format, i.e. the jobs/result document

I think you are on the wrong track. You'd describe the process description with JSON Schema only. Nothing else would change.

1. Title and description can only be mapped to the single description property

There's both title and description in JSON Schema. I don't see the problem here?

2. No metadata can be provided. Can we extend JSON Schema ? This is a critical point !

Yes, it is possible. But I'm not sure what this would be useful for. It would very much complicate JSON Schema, which was said should even be simplified.

3. For LiteralData types, allowed values cannot be provided. An alternative is the pattern regexp (e.g. pattern: '^\d{3}-\d{2}-\d{4}$')

For strings there's format and pattern, which should fit most cases. For numbers there are other parameters. What does WPS offer in addition to JSON Schema, which is crucial?

4. Process Description spec includes for each input and output the supported mimeTypes (and default). OpenAPI only allows defining mime type definition for the whole request/response but not for the individual input/output (binary) files

Look how we do it for openEO. You are too strictly trying to put everything in either openAPI or JSON Schema. In a combined document. That won't work and was never intended, I think.

[christophenoel]

Not understood the last sentence.

[m-mohr]

The last paragraph?

I don't think the intention was to put the process descriptions into the openAPI files. The intention was to use JSON Schema and/or the JSON Schema derivation from OpenAPI. Therefore you are not limited by what a single OpenAPI document offers and your last point doesn't seem to be a problem.

[christophenoel]

You are probably right but I didn't catch that.

You suggest basically (in brief) that we only replace the complexData/literalData definition with a schema definition as you did for openEO.

What I understood this morning is that we directly provide the process "interface" in the OpenAPI definition of the API (which means we don't need an operation to discover processes). The reason why I was thinking about this is because there was an equivalent way of doing that in XML (but never used it).

Even if you example looks great, I think that my second approach is also interesting to think about. You only need to extend with some custom properties for supporting all usual WPS features.

[bpross-52n]

Moved to the wiki: https://github.com/opengeospatial/wps-rest-binding/wiki/General-API-considerations