Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ingest] OpenAPI spec file #68323

Merged
merged 6 commits into from
Jun 9, 2020
Merged

Conversation

jfsiii
Copy link
Contributor

@jfsiii jfsiii commented Jun 4, 2020

Summary

Adds an OpenAPI spec file with some combination of query/path param, request/response body and examples for each route.

I believe it contains all the routes for Ingest Manager (including Fleet, EPM, etc) but I'll confirm and add any ones we find are missing.

There's probably one thing wrong (a missing header, request or response shape) on many routes, but it's still very useful as-is.

It's not used anywhere right now, but I think there's value in getting the spec file committed so others can reference, import, and improve on it. We can adopt OpenAPI incrementally and in many different ways. Getting the spec file in the repo is the first step.

John Schulz added 3 commits June 4, 2020 17:16
Lots have one or more things wrong but it's a start
Plus some examples for request and responses
It's not an option to change interface name, afaict
lowercase matches the OpenAPI spec/key name
@jfsiii jfsiii changed the title [Ingest] OpenAPI spec and TS types [Ingest] OpenAPI spec file Jun 8, 2020
@jfsiii jfsiii marked this pull request as ready for review June 8, 2020 13:30
@jfsiii jfsiii requested a review from a team June 8, 2020 13:30
@jfsiii jfsiii added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team labels Jun 8, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Team:Ingest Management)

@jfsiii jfsiii linked an issue Jun 8, 2020 that may be closed by this pull request
@ph ph self-assigned this Jun 8, 2020
@jfsiii jfsiii removed a link to an issue Jun 8, 2020
@nchaulet
Copy link
Member

nchaulet commented Jun 8, 2020

@jfsiii Do you generated this, or it's all by hand? Also can we split this into multiple files (I think it could avoid merge conflict later, and I personally found it easier to read/write )?

@jfsiii
Copy link
Contributor Author

jfsiii commented Jun 8, 2020

@nchaulet I mostly used a GUI (https://stoplight.io/studio/) to create the file. I occasionally use an editor to update the file as well.

I totally agree about breaking it up and that's even what I do locally. I exported these into a single file because many tools expect / require it.

Longer term, I was picturing something like a file (or directory) per path. You can start with headers, parameters, etc at a query level, but can lift them up to shared components as needed.

For example many endpoint currently define the perPage param, we can lift that up to a common item and other operations can also use it.

That common parameter is actually defined but it's only used in one place at the moment

Longer term, I see the OpenAPI spec as the source of truth for what's supported and route additions / changes are added to a plain text file, then implemented in code which imports the spec. That way any change to a contract are explicitly marked in text.

That's not a requirement though and I plan on introducing this incrementally and getting feedback in PRs along the way or ideally, watching people add stuff on their own.

This file doesn't change any workflow or code behavior. It's to make it available more real and available for others if they want.

@jfsiii
Copy link
Contributor Author

jfsiii commented Jun 8, 2020

@elasticmachine merge upstream

elasticmachine and others added 2 commits June 8, 2020 11:42
Copy link
Member

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on getting this in and then iterate on it.

On the package-registry side I decided to go with YAML instead: elastic/package-registry#490 The reason is that I think sometimes it is more readable for humans and it allows to add comments outside the spec. But for the second one, I realised all the comments I wanted to add so far went into the description part. JSON and YAML are find for me, just thought worth mentioning.

@jfsiii
Copy link
Contributor Author

jfsiii commented Jun 9, 2020

Merging this so it's available for others. e.g. the discussion in elastic/e2e-testing#126

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jfsiii jfsiii merged commit 974dcab into elastic:master Jun 9, 2020
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jun 9, 2020
* master: (37 commits)
  [DOCS] Adds documentation for drilldowns (elastic#68122)
  [Telemetry] telemetry.sendUsageFrom: "server" by default (elastic#68071)
  [ML] Transforms: Support sub-aggregations (elastic#68306)
  Fixed pre-configured docs link points to the wrong page and functional tests configs (elastic#68606)
  [Ingest Manager] Update queries from `stream.*` to `dataset.*` (elastic#68322)
  Enable Watcher by default to fix bug in which Watcher doesn't render in the side nav (elastic#68602)
  Convert Index Templates API routes to snakecase. (elastic#68463)
  [SECURITY SOLUTION] [Detections] Add / Update e2e tests to ensure initial rule runs are successful (elastic#68441)
  [Ingest] OpenAPI spec file (elastic#68323)
  chore(NA): skip apis Endpoint plugin Endpoint policy api (elastic#68638)
  bumping makelogs version to v6.0.0 (elastic#66163)
  [SIEM] Add create template button (elastic#66613)
  Bump websocket-extensions from 0.1.3 to 0.1.4 (elastic#68414)
  [ML] Sample data modules - use event.dataset instead of index name (elastic#68538)
  [ML] Functional tests - fix job validation API test with maxModelMemoryLimit (elastic#68501)
  [ML] Functional tests - stabilize DFA job creation (elastic#68495)
  [TSVB] Allows the user to change the tooltip mode (elastic#67775)
  chore(NA): skip apis Endpoint plugin Endpoint alert API when data is in elasticsearch (elastic#68613)
  chore(NA): skip endpoint Endpoint Alert Page: when es has data and user has navigated to the page (elastic#68596)
  [SIEM][Detection Engine] Converts from joi to use io-ts and moves the types to common  (elastic#68127)
  ...
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 68323 or prevent reminders by adding the backport:skip label.

@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jun 11, 2020
@jen-huang
Copy link
Contributor

Kicking off a backport for this so that there's no issue when I merge a PR (#68737) that adds a new route to the spec file.

jen-huang added a commit that referenced this pull request Jun 11, 2020
* All(?) routes
Lots have one or more things wrong but it's a start

* Some Agent-related models & routes.

Plus some examples for request and responses

* Ignore rule requring PascalCase interfaces
It's not an option to change interface name, afaict
lowercase matches the OpenAPI spec/key name

* Remove generated specs

* Fix misplaced schema

Response shape was listed as a request body

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: John Schulz <john.schulz@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jun 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.9.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants