Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ingest Manager] Add route for package installation by upload #77044

Merged
merged 4 commits into from
Sep 15, 2020

Conversation

skh
Copy link
Contributor

@skh skh commented Sep 9, 2020

Summary

Partially implements #70582

  • Adds the route POST /api/ingest_manager/epm/packages that takes a buffer as body, which should contain the package archive to be installed.
  • Refactors the existing route handlers for package installation so that "install by upload" and "install from registry" are clearly distinguishable.
  • Adds an integration test for the new endpoint.
  • Archive verification and package installation are not implemented yet. This change intentionally only adds the API endpoint.

How to test this

curl -X POST -u elastic:changeme http://localhost:5601/api/ingest_manager/epm/packages --data @./tmp/packages/foo.tar.gz -H 'kbn-xsrf: xyz' -H 'Content-Type: application/gzip'

should respond with

{"response":"package upload was received ok, but not installed (not implemented yet)"}

No actual package installation takes place yet. The integrity or file type of the archive (foo.tar.gz) isn't verified yet.

Choice of route

With this change,

  • POST /api/ingest_manager/epm/packages/{pkgkey} is used to install a package from the registry, and
  • POST /api/ingest_manager/epm/packages for installing by direct upload.

The initial reason was that the existing endpoint already uses the POST body to optionally send the { force: true|false } parameter, so it can't be used to upload a file, but it would of course be possible to refactor that and move the force parameter to URL params instead. That way, the same route could be used for both ways to install a package.

After thinking a bit longer about it, as both package name and version are set within the uploaded package archive, I think we don't really need to ask users to provide the package key in the POST url, as it would be duplicate information.

Open issues

  • This PR does not contain changes to the OpenAPI spec file. I'd like to add this endpoint to this spec once it is fully implemented, and the response format has been finalized.

@skh skh changed the title Add route to accept a direct package upload. [Ingest Manager] Add route for package installation by upload Sep 9, 2020
@skh skh self-assigned this Sep 9, 2020
@skh skh added Feature:Fleet Fleet team's agent central management project Team:Fleet Team label for Observability Data Collection Fleet team release_note:skip Skip the PR/issue when compiling release notes labels Sep 9, 2020
@skh skh force-pushed the 70582-direct-package-upload-part-1 branch from fab70f2 to bad3b65 Compare September 10, 2020 12:32
@skh skh force-pushed the 70582-direct-package-upload-part-1 branch from bad3b65 to d8f3203 Compare September 14, 2020 10:06
@skh
Copy link
Contributor Author

skh commented Sep 14, 2020

@mtojek @ycombinator This might be a good time to have a look at the API endpoint and comment if this works for you or if you would like to see changes.

@ycombinator
Copy link
Contributor

I like it! All the information is (going to) be parsed from package contents so just asking for the package contents in either zip or gzip format makes sense to me. It should also make it very easy to take the artifact produced by elastic-package build and "pipe" it into this API.

👍 👍 from me.

@skh skh marked this pull request as ready for review September 14, 2020 14:47
@skh skh requested a review from a team September 14, 2020 14:47
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Feature:Fleet)

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

page load bundle size

id value diff baseline
ingestManager 468.5KB +72.0B 468.4KB

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Copy link
Contributor

@jfsiii jfsiii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Thanks for the context/thinking around the routes.

@skh skh merged commit 37b9c81 into elastic:master Sep 15, 2020
skh added a commit to skh/kibana that referenced this pull request Sep 15, 2020
…c#77044)

* Add route to accept a direct package upload.

* Only accept application/zip or application/gzip

* Use better name for response containing only a message.

* Add integration test for direct package upload.
gmmorris added a commit to gmmorris/kibana that referenced this pull request Sep 15, 2020
* master: (25 commits)
  [Security Solution] Add unit tests for Network search strategy (elastic#77416)
  [Alerting] Improves performance of the authorization filter in AlertsClient.find by skipping KQL parsing (elastic#77040)
  [Ingest Manager] Add route for package installation by upload (elastic#77044)
  [APM-UI][E2E] filter PRs from the uptime GH team (elastic#77359)
  [APM] Remove useLocation and some minor route improvements (elastic#76343)
  [Enterprise Search] Update enterpriseSearchRequestHandler to manage range of errors + add handleAPIErrors helper (elastic#77258)
  [SECURITY_SOLUTION] Task/hostname policy response ux updates (elastic#76444)
  Move remaining uses of serviceName away from urlParams (elastic#77248)
  [Lens] Move configuration popover to flyout (elastic#76046)
  [Ingest Manager] Manually build Fleet kuery with Node arguments (elastic#76589)
  skip flaky suite (elastic#59975)
  Neutral-naming in reporting plugin (elastic#77371)
  [Enterprise Search] Add UserIcon styles (elastic#77385)
  [RUM Dashboard] Added loading state to visitor breakdown pie charts (elastic#77201)
  [Ingest Manager] Fix polling for new agent action (elastic#77339)
  Remote cluster - Functional UI test to change the superuser to a test_user with limited role (elastic#77212)
  Stacked headers and navigational search (elastic#72331)
  [ML] DF Analytics creation wizard: Fixing field loading race condition (elastic#77326)
  [Monitoring] Handle no mappings found for sort and collapse fields (elastic#77099)
  Add Lens to Recently Accessed (elastic#77249)
  ...
gmmorris added a commit to gmmorris/kibana that referenced this pull request Sep 15, 2020
* master: (293 commits)
  Fix tsvb filter ration for table (elastic#77272)
  [Security Solution] Add unit tests for Network search strategy (elastic#77416)
  [Alerting] Improves performance of the authorization filter in AlertsClient.find by skipping KQL parsing (elastic#77040)
  [Ingest Manager] Add route for package installation by upload (elastic#77044)
  [APM-UI][E2E] filter PRs from the uptime GH team (elastic#77359)
  [APM] Remove useLocation and some minor route improvements (elastic#76343)
  [Enterprise Search] Update enterpriseSearchRequestHandler to manage range of errors + add handleAPIErrors helper (elastic#77258)
  [SECURITY_SOLUTION] Task/hostname policy response ux updates (elastic#76444)
  Move remaining uses of serviceName away from urlParams (elastic#77248)
  [Lens] Move configuration popover to flyout (elastic#76046)
  [Ingest Manager] Manually build Fleet kuery with Node arguments (elastic#76589)
  skip flaky suite (elastic#59975)
  Neutral-naming in reporting plugin (elastic#77371)
  [Enterprise Search] Add UserIcon styles (elastic#77385)
  [RUM Dashboard] Added loading state to visitor breakdown pie charts (elastic#77201)
  [Ingest Manager] Fix polling for new agent action (elastic#77339)
  Remote cluster - Functional UI test to change the superuser to a test_user with limited role (elastic#77212)
  Stacked headers and navigational search (elastic#72331)
  [ML] DF Analytics creation wizard: Fixing field loading race condition (elastic#77326)
  [Monitoring] Handle no mappings found for sort and collapse fields (elastic#77099)
  ...
skh added a commit that referenced this pull request Sep 15, 2020
#77438)

* Add route to accept a direct package upload.

* Only accept application/zip or application/gzip

* Use better name for response containing only a message.

* Add integration test for direct package upload.
@skh skh deleted the 70582-direct-package-upload-part-1 branch September 15, 2020 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Fleet Fleet team's agent central management project release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.10.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants