[Ingest Manager] Support both zip & tar archives from Registry #76197

jfsiii · 2020-08-27T21:10:01Z

Summary

elastic/package-registry#628 will change the archive format from the from .tar.gz to .zip.

Support both formats for greater resiliency/backwards compatibility

Restored unzip functions from Remove code for handling .zip files #43764
Persist the download value returned by EPR (e.g. /epr/system/system-0.5.3.zip or /epr/system/system-0.5.3.tar.gz) as "archive location" for a package name/version combo.
Based on that value, we decide which decompression to use.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

Testing

Added tests for getBufferExtractor to document tar vs zip behavior

I did a local check reading & installing a few packages while switching the between the master and elastic/package-registry#628.

The local registry logs showed the expected .tar.gz vs .zip archives being requested
The HTTP responses from installing assets were successful / as expected
The appropriate epm-packages saved objects were in place
Browsing to the Installed Integrations and Integration detail page worked as expected

Restored unzip functions from elastic#43764 Persist the `download` value returned by EPR (e.g. `/epr/system/system-0.5.3.zip` or `/epr/system/system-0.5.3.tar.gz`) as "archive key" for a package name/version combo. The same name&version should return the same archive. The value initially given by the registry. Based on that value, we decide which decompression to use.

jfsiii · 2020-08-27T23:46:46Z

x-pack/plugins/ingest_manager/server/services/epm/registry/index.ts

  if (!buffer) {
    buffer = await fetchArchiveBuffer(pkgName, pkgVersion);
-    cacheSet(key, buffer);


moved the caching from this getOrFetch function to inside the fetchArchiveBuffer

jfsiii · 2020-08-27T23:47:06Z

x-pack/plugins/ingest_manager/server/services/epm/registry/index.ts

+  const buffer = await getResponseStream(archiveUrl).then(streamToBuffer);
+
+  setArchiveKey(pkgName, pkgVersion, archivePath);
+  cacheSet(archivePath, buffer);


ensure we always cache a fetched archive buffer

is this cache invalidated if the current package is removed? This is mainly important for package development.

@ruflin No, but that's the same behavior as now. I'm not sure if this means it is an issue or if it's being avoided/worked around in some other way.

@skh @neptunian I thought one of you solve this issue recently or at least debugged it?

There is #68890 -- it's still open because I misunderstood the original description and tested the wrong things, and then it got moved back in the queue.

@skh Thanks for the link. This basically means we can move forward with what we have in this PR and will follow up later in #68890

skh

Changes LGTM 👍

ruflin · 2020-08-31T10:56:02Z

++ on getting this in as it does not seem to change the existing .tar.gz implementation (mostly). Then we can start testing with the zip registry and move over step by step.

jfsiii · 2020-08-31T18:59:40Z

@elasticmachine merge upstream

jfsiii · 2020-08-31T20:56:26Z

@skh @ruflin what types of tests should I add before merging this?

Or what aspects should I test?

ruflin · 2020-09-01T07:27:31Z

@jfsiii I guess for now the most important part is that all things still keep working which seems to be the case. If you could run my registry PR locally to see if it works with your changes, this would be great.

jfsiii · 2020-09-01T10:17:13Z

@ruflin sure thing. I'll double check, but I used your branch while developing initially

I only did a quick check of reading & installing a few packages while switching the between the master and elastic/package-registry#628 but 👍

ruflin · 2020-09-01T11:03:00Z

@jfsiii Great, then lets get it in! 👍 Can you resolve the conflict?

* Add tests for getBufferExtractor * Replace `[aA]rchiveKey*` with `[aA]rchiveLocation*`

elasticmachine · 2020-09-01T17:21:07Z

Pinging @elastic/ingest-management (Team:Ingest Management)

…rt-epr-zip

jfsiii · 2020-09-01T18:31:29Z

@ruflin I fixed the conflict, added some tests and updated the description. I'm planning on merging when CI is 💚

kibanamachine · 2020-09-01T20:52:26Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: e4fd596

Build metrics

async chunks size

id	value	diff	baseline
enterpriseSearch	358.5KB	+24.0B	358.5KB

page load bundle size

id	value	diff	baseline
upgradeAssistant	64.8KB	+24.0B	64.8KB

History

💛 Build #71638 was flaky 547a8e0
💚 Build #71385 succeeded c230acc
💚 Build #71085 succeeded e5c0e92
💚 Build #71054 succeeded a3c303f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@ts-expect-error

* master: (223 commits) skip flaky suite (elastic#75724) [Reporting] Add functional test for Reports in non-default spaces (elastic#76053) [Enterprise Search] Fix various icons in dark mode (elastic#76430) skip flaky suite (elastic#76245) Add `auto` interval to histogram AggConfig (elastic#76001) [Resolver] generator uses setup_node_env (elastic#76422) [Ingest Manager] Support both zip & tar archives from Registry (elastic#76197) [Ingest Manager] Improve agent vs kibana version checks (elastic#76238) Manually building `KueryNode` for Fleet's routes (elastic#75693) remove dupe tinymath section (elastic#76093) Create APM issue template (elastic#76362) Delete unused file. (elastic#76386) [SECURITY_SOLUTION][ENDPOINT] Trusted Apps Create API (elastic#76178) [Detections Engine] Add Alert actions to the Timeline (elastic#73228) [Dashboard First] Library Notification (elastic#76122) [Maps] Add mvt support for ES doc sources (elastic#75698) Add setHeaderActionMenu API to AppMountParameters (elastic#75422) [ML] Remove "Are you sure" from data frame analytics jobs (elastic#76214) [yarn] remove typings-tester, use @ts-expect-error (elastic#76341) [Reporting/CSV] Do not fail the job if scroll ID can not be cleared (elastic#76014) ...

…ic#76197) * Quick pass at restoring support for both zip & tar Restored unzip functions from elastic#43764 Persist the `download` value returned by EPR (e.g. `/epr/system/system-0.5.3.zip` or `/epr/system/system-0.5.3.tar.gz`) as "archive key" for a package name/version combo. The same name&version should return the same archive. The value initially given by the registry. Based on that value, we decide which decompression to use. * Use template literal vs JSON.stringify for keygen * Factor unzip/untar logic out to getBufferExtractor * Add tests for getBufferExtractor * Replace `[aA]rchiveKey*` with `[aA]rchiveLocation*` * Include given name & version in error message Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> # Conflicts: # x-pack/plugins/ingest_manager/server/services/epm/registry/cache.ts

… (#76485) * Quick pass at restoring support for both zip & tar Restored unzip functions from #43764 Persist the `download` value returned by EPR (e.g. `/epr/system/system-0.5.3.zip` or `/epr/system/system-0.5.3.tar.gz`) as "archive key" for a package name/version combo. The same name&version should return the same archive. The value initially given by the registry. Based on that value, we decide which decompression to use. * Use template literal vs JSON.stringify for keygen * Factor unzip/untar logic out to getBufferExtractor * Add tests for getBufferExtractor * Replace `[aA]rchiveKey*` with `[aA]rchiveLocation*` * Include given name & version in error message Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> # Conflicts: # x-pack/plugins/ingest_manager/server/services/epm/registry/cache.ts

jfsiii mentioned this pull request Aug 27, 2020

Discuss: Move to zip instead of tar.gz for the format of the package. elastic/package-registry#474

Closed

jfsiii added the Team:Fleet Team label for Observability Data Collection Fleet team label Aug 27, 2020

Use template literal vs JSON.stringify for keygen

e5c0e92

jfsiii commented Aug 27, 2020

View reviewed changes

jfsiii self-assigned this Aug 29, 2020

skh approved these changes Aug 31, 2020

View reviewed changes

skh mentioned this pull request Aug 31, 2020

[Fleet] Allow to install a package without the registry. #70582

Closed

8 tasks

ruflin mentioned this pull request Aug 31, 2020

Ship packages as zip instead of tar.gz elastic/package-registry#628

Merged

Merge branch 'master' into support-epr-zip

c230acc

John Schulz added 2 commits September 1, 2020 13:01

Factor unzip/untar logic out to getBufferExtractor

9c282d1

* Add tests for getBufferExtractor * Replace `[aA]rchiveKey*` with `[aA]rchiveLocation*`

Merge branch 'master' into support-epr-zip

547a8e0

jfsiii marked this pull request as ready for review September 1, 2020 17:21

jfsiii requested a review from a team September 1, 2020 17:21

Include given name & version in error message

50bc169

jfsiii added release_note:skip Skip the PR/issue when compiling release notes v7.10.0 v8.0.0 labels Sep 1, 2020

Merge branch 'support-epr-zip' of github.com:jfsiii/kibana into suppo…

e4fd596

…rt-epr-zip

jfsiii merged commit 9a7c418 into elastic:master Sep 1, 2020

jfsiii mentioned this pull request Sep 2, 2020

[7.x] [Ingest Manager] Support both zip & tar archives from Registry (#76197) #76485

Merged

ruflin mentioned this pull request Sep 17, 2020

[Ingest Manager] Remove legacy code for path/src and tar.gz packages #77728

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Support both zip & tar archives from Registry #76197

[Ingest Manager] Support both zip & tar archives from Registry #76197

jfsiii commented Aug 27, 2020 •

edited

Loading

jfsiii Aug 27, 2020

jfsiii Aug 27, 2020

ruflin Aug 31, 2020

jfsiii Aug 31, 2020

ruflin Aug 31, 2020

skh Aug 31, 2020

ruflin Aug 31, 2020

skh left a comment

ruflin commented Aug 31, 2020

jfsiii commented Aug 31, 2020

jfsiii commented Aug 31, 2020

ruflin commented Sep 1, 2020

jfsiii commented Sep 1, 2020

ruflin commented Sep 1, 2020

elasticmachine commented Sep 1, 2020

jfsiii commented Sep 1, 2020

kibanamachine commented Sep 1, 2020

[Ingest Manager] Support both zip & tar archives from Registry #76197

[Ingest Manager] Support both zip & tar archives from Registry #76197

Conversation

jfsiii commented Aug 27, 2020 • edited Loading

Summary

Checklist

Testing

jfsiii Aug 27, 2020

Choose a reason for hiding this comment

jfsiii Aug 27, 2020

Choose a reason for hiding this comment

ruflin Aug 31, 2020

Choose a reason for hiding this comment

jfsiii Aug 31, 2020

Choose a reason for hiding this comment

ruflin Aug 31, 2020

Choose a reason for hiding this comment

skh Aug 31, 2020

Choose a reason for hiding this comment

ruflin Aug 31, 2020

Choose a reason for hiding this comment

skh left a comment

Choose a reason for hiding this comment

ruflin commented Aug 31, 2020

jfsiii commented Aug 31, 2020

jfsiii commented Aug 31, 2020

ruflin commented Sep 1, 2020

jfsiii commented Sep 1, 2020

ruflin commented Sep 1, 2020

elasticmachine commented Sep 1, 2020

jfsiii commented Sep 1, 2020

kibanamachine commented Sep 1, 2020

💚 Build Succeeded

Build metrics

async chunks size

page load bundle size

History

jfsiii commented Aug 27, 2020 •

edited

Loading