OpenSearch Smithy IDL Model - RFC

[pgtgrly]

Is your feature request related to a problem? Please describe.
Currently OpenSearch only provides partial definitions for the available OpenSearch APIs in https://github.com/opensearch-project/OpenSearch/tree/main/rest-api-spec/src/main/resources/rest-api-spec/api. These APIs are defined here as JSON documents and describes only the URL path for the operation and the Query parameters that it accepts. The accepted parameters in the request body for the APIs are not clearly specified anywhere and similarly the expected response structure is not defined. The documentation also does not cover all the possible request parameters / responses.
For users of OpenSearch it would be extremely helpful if all these available APIs were well defined and documented using an interface definition language like Smithy (https://awslabs.github.io/smithy/).

Describe the solution you'd like
We propose to define new API models for the OpenSearch APIs using Smithy interface definition language (IDL). Smithy is an open source IDL which is protocol agnostic, extensible and supports code generation. The API models could be defined in the same OpenSearch Github repository under a new “api model” directory or we could create a separate Github repository for defining the API models.
We have created a POC to demonstrate the use of Smithy models by defining few OpenSearch APIs: https://github.com/pgtgrly/OpenSearch/tree/smithy-specification. The models have been defined in “smithy-specification” directory and it is configured to also generate an OpenAPI model and a typescript client during build.

Describe alternatives you've considered
We could also define the API models using OpenAPI directly, however we felt that Smithy was much more expressive and easier language to define the APIs. Smithy models are much cleaner to read and can be easily converted to OpenAPI during build.

Additional context
Below are some snippets on how the service and APIs can be defined using Smithy:

Defining the service

@restJson1
service OpenSearch {
    version: "2020-09-01",
    operations: [Ping, GetCCRGetAutoFollowPattern, GetCCRGetAutoFollowPatternWithName, PutIndexMapping,
    PostSearchWithIndex, PostSearch, PutCreateIndex, PutIndexMapping]
}

Defining an Operation / API

@http(method: "PUT", uri: "/{index}/_mapping")
@documentation("Updates the index mappings.")
operation PutIndexMapping {
    input: PutIndexMappingInput,
    output: PutIndexMappingOutput
}

Defining the Request body for Put Mapping operation

structure PutIndexMappingInput {
    // https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-put-mapping.html    // RequestBody is different in docs and in typescript    @httpLabel    @required    index: IndexName,

    @httpQuery("allow_no_indices")
    allow_no_indices: Boolean,

    @httpQuery("expand_wildcards")
    expand_wildcards: ExpandWildcards,

    @httpQuery("ignore_unavailable")
    ignore_unavailable: Boolean,

    @httpQuery("include_type_name")
    include_type_name: Boolean,

    @httpQuery("master_timeout")
    master_timeout: Time,

    @httpQuery("timeout")
    timeout: Time,

    @httpQuery("write_index_only")
    write_index_only: Boolean,

    @documentation("Controls whether dynamic date detection is enabled.")
    dynamic: DynamicAttribute,

    properties: PropertyMap

}

Defining Response body for Put Mapping operation

structure PutIndexMappingOutput {
    acknowledged: Boolean
}

Smithy models can easily be configured to translate into OpenAPI format by specifying OpenAPI plugin in the smithy-build.json file

"plugins": {
  "openapi": {
    "service": "OpenSearch#OpenSearch"
  }
}

For more details please refer to the Smithy models defined in POC: https://github.com/pgtgrly/OpenSearch/tree/smithy-specification

Benefits of defining OpenSearch API models using Smithy

• Establishes a clear contract between the Service and Consumer
• Provides ability to version the available operations and ability to indicate deprecated operations / parameters
• Defining APIs using Smithy IDL is intuitive with syntax closer to programming languages and support for adding comments inline
• Smithy provides capability to generate clients for some languages (Go, Rust and Typescript) out of the box, with support for other languages planned
• The Smithy model can be configured to auto generate corresponding OpenAPI model file, which could be further leveraged to generate API documentation / visualizations
• The generated OpenAPI could also be used to generate other clients using OpenAPI client generators

Limitations

• Smithy currently does not support untagged unions, however where such unions are needed for describing the API parameters we can make use of DocumentType
• Some OpenSearch GET method APIs allow request bodies. Smithy and OpenAPI does not support request bodies for GET operations
• Same parameter name cannot be part of query parameters and request body

---

Co-authored by: pgtgrly and sachetalva

[dblock]

Same parameter name cannot be part of query parameters and request body

Do we have instances of such?

[dblock]

I like the proposal and I think you should PR it. It looks like there are no downsides except that we all have to learn to author a different spec?

The change in OpenSearch would imply the same change in plugins as we want to publish a combined API spec for the entire distribution. See opensearch-project/opensearch-build#702. Does this make it easier/harder/different?

[pgtgrly]

Do we have instances of such?

Yup, One such example in the PoC would be the search within Search API, we noticed that the several parameter can be passed as both query parameter and within the body.

Will get back to you regarding the second part

[CEHENKLE]

Hello all;

Thanks for your feedback. We're going to be moving ahead on this project. If you're interested, or you'd like to contribute, we'll be working in a new repo called opensearch-api-specifications. I'll be closing this issue out on February 21st to continue the conversations over there.

See you in the new repo!

[mtdowling]

Very cool to see this!