Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Add test(s) to compare workflow steps configuration to API specification #651

Closed
dbwiddis opened this issue Apr 6, 2024 · 0 comments · Fixed by #937
Closed
Labels
CI CI related issues enhancement New feature or request

Comments

@dbwiddis
Copy link
Member

dbwiddis commented Apr 6, 2024

Is your feature request related to a problem?

Workflow Steps are tightly coupled to the APIs they implement.

The APIs are hosted in external repositories, and there is not signal to our plugin that an API has added features.

OpenSearch APIs have OpenAPI spec files written for them: https://github.com/opensearch-project/opensearch-api-specification

I have opened an issue on ML Commons requesting they do the same: opensearch-project/ml-commons#2297

We should establish automated testing that runs on a periodic basis (Scheduled GitHub Action runs) to test whether we have implemented all API parameters.

What solution would you like?

  1. Write code to remotely fetch the API specification from wherever it resides. For example, OpenSearch APIs are published here: https://github.com/opensearch-project/opensearch-api-specification/tree/main/spec . It may be acceptable to fetch these file via wget in the GHA script to make it available locally, but if the effort to use a testing client (like we do in integ tests) is low, that would be preferable.
  2. Parse the YAML into an object. This may be complex: for example, creating an index references a request body with a path that has to be looked up elsewhere.
  3. Use the WorkflowSteps enum to compare the schema/content/requestBody definitions with the required and optional inputs. The enum may need to be updated to include other information for comparison.

What alternatives have you considered?

Manually following change logs, documentation, and PRs

Do you have any additional context?

It may not be necessary to go deeply into object parsing; just the first level presence or absence of an object key in the request body is likely sufficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI CI related issues enhancement New feature or request
Projects
None yet
1 participant