Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Document MLClient API using spec files #2297

Open
dbwiddis opened this issue Apr 5, 2024 · 2 comments
Open

[FEATURE] Document MLClient API using spec files #2297

dbwiddis opened this issue Apr 5, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@dbwiddis
Copy link
Member

dbwiddis commented Apr 5, 2024

Is your feature request related to a problem?

As a downstream plugin, Flow Framework consumes the ML Commons API via its Client. As such, we are tightly coupled with the API. During the recent 2.13.0 release cycle, a new feature was added to one of the ML Commons APIs. Because we were not closely monitoring the development of the feature, it wasn't noticed until manual testing of a workflow using the new feature discovered its lack of implementation.

There is little visibility into new features without tracking Issues and PRs. Particularly around releases, the documentation is not always written until after the "code freeze" (Release Candidate generation) date, greatly shortening the time window to detect and react to the updates. In addition, this requires human intervention to search for and identify these changes.

We would like to develop an automated way to discover, via regular testing, when the API specification differs from our implementation, as early as possible.

What solution would you like?

In general: some file published in some common location that we can machine-read to identify required and optional parameters for the APIs we implement.

Specifically: OpenSearch recently transisitioned to using the OpenAPI specification for its API. See, for example, the /cat API: https://github.com/opensearch-project/opensearch-api-specification/blob/main/spec/namespaces/cat.yaml

ML Commons could publish a similar API specification for its Client, using OpenAPI. It's easy to do, with web-based authoring tools. I did so for the Extensions SDK Hello World extension at the Swagger website without knowing anything about the spec: https://github.com/opensearch-project/opensearch-sdk-java/blob/main/src/main/java/org/opensearch/sdk/sample/helloworld/spec/openapi.yaml

Then over at Flow Framework we could just read in this spec file, iterate over the APIs we care about and parse the parameters, and make sure we have them all covered.

What alternatives have you considered?

Having fallible humans regularly monitoring PRs, Change Log, and Documentation Website hoping to catch all the changes.

Do you have any additional context?

Other clients (in other languages) may want to consume this spec to auto-generate client classes. See opensearch-project/opensearch-api-specification#189

@dbwiddis dbwiddis added enhancement New feature or request untriaged labels Apr 5, 2024
@navneet1v
Copy link
Contributor

+1 on this.

@owaiskazi19
Copy link
Member

This is how core detects the breaking change opensearch-project/OpenSearch#12974 of the downstream plugins. We should have something similar for ml-commons as Github workflow.

@dhrubo-os dhrubo-os moved this to On-deck in ml-commons projects May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: On-deck
Development

No branches or pull requests

4 participants