[Fleet] Allow to install a package without the registry.

[ph]

For development and testing purposes, there should be a way to directly install the package (whatever is generated to be deployed in the package storage, either exploded or as a tarball) without the registry.

We need to investigate how we want to do that:

• Could we call an API?
• Could we drop a zip in the UI?
• Gold+

This feature is initially for developers but could be extended to every user.

---

Full implementation proposal: #70582 (comment)

7.10:

• Refactor EPM package handling & installation code (@skh [Ingest Manager] Install uploaded package #77986)
• Enhance epm-package saved object to include the package source (@skh [Ingest Manager] Install uploaded package #77986)
• Enhance EPM packages endpoint to accept an uploaded package in POST (@skh [Ingest Manager] Add route for package installation by upload #77044)

---

7.11:

• [Fleet] Use epm-packages saved object as main source of truth for installed packages [Fleet] Use epm-packages saved object as main source of truth for installed packages #81995 @neptunian
• [Fleet] Proposal: Store installed packages in cluster [Fleet] Proposal: Store installed packages in cluster #81110 @jfsiii
• [Fleet] Refactor package installation / upgrade / downgrade and error recovery / rollback [Fleet] Refactor package installation / upgrade / downgrade and error recovery / rollback #82007 @skh
• [Fleet] Add error handling: uploaded package missing from in-memory cache [Fleet] Add error handling: uploaded package missing from in-memory cache #82010 @skh
• Add package validator in typescript Add package validator in typescript package-spec#58 @jfsiii

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[exekias]

@mtojek @ycombinator @jsoriano this one should be interesting for the elastic-package tool

I understand that a zip in the UI would end up hitting an API too, so +1 on having an API at least to allow us to do this from the command line

[jsoriano]

This would be great yes, so package development can be done without a registry, and having a independent definition of the package becomes more important.

Also this will allow to have a clearer view of each component:

• elastic-package can be used to develop, maintain and create packages (the final artifact is, or can be, a package).
• elastic-registry can be used to distribute packages.
• kibana uses packages, and can have different installation methods for that.

[ycombinator]

++ for having an API for reasons mentioned by others.

Are there any safety or security considerations here? Do we want to put such an API behind auth? Regardless of auth, we probably need safety checks to prevent overrides (so one user cannot override a package being used by others)? Or, if this is mainly a development concern (which it sounds like), perhaps we only enable this API in Kibana dev mode or behind an opt-in kibana.yml setting - then we don't need to worry about the safety/security issues as much?

[mostlyjason]

We'd like the option to run a local registry to be a Gold+ feature, and this would provide a backdoor. If we enable it in production, we may need to put a license restriction on it. Otherwise, we could limit it to development only mode.

[ph]

@ycombinator I

It should be already behind auth, nothing related to ingest manager should be exposed.

[ycombinator]

It should be already behind auth, nothing related to ingest manager should be exposed.

Yes, of course it is, doh! 🤦

[skh]

Are there any safety or security considerations here?

Anything in Ingest Manager is only available for superuser users, so they can do pretty much everything in the dev console anyway, as @ph says.

However, EPM package installation code never got a real security audit, so if this feature will be enabled in cloud, I would say yes, there are security considerations, not for customer data but for our own hosted cloud environment.

[skh]

Here's my rough proposal what we need to change to allow direct package upload and installation. I've split it into two iterations, the first one would be for 7.10 because the feature is important for integrations developers. The second (and possible third, fourth) would come in later releases as necessary.

---

Direct package upload

7.10: First iteration, development use only, undocumented

File upload

• Provide an API endpoint only, no UI
• Handle uploaded packages in memory only
• Restrict upload file size (so that it's safe to handle the package archive in memory only)
• Restrict to Gold+ just in case
• work with .tar.gz and .zip formats for now

Changes to saved object 'epm-packages'

• add field for package source (registry or direct upload)
• add migration to add that field with default value "registry"

Changes to Package installation (can be done in parallel with Integrations page)

• enhance package-handling code in EPM so that it can handle packages from different sources (registry and upload)
• refactor package installation code so that it can be called from different route handlers
• direct upload has implicit force:true
• (maybe more refactors I'm not thinking about right now)

Changes to Integrations page (can be done in parallel with Package installation)

• Display packages that are not in the registry without errors (doubles as fallback when the registry is not available)
  • get information about the package from saved objects instead
  • just leave out missing information (screenshots, README, icons, ...)
• Indicate on the integrations page that a package has been installed by direct upload
• When an uploaded package also exists in the registry, it needs to be removed manually before the registry package can be installed

All other pages & API endpoints

• ensure these changes are transparent to the rest of Ingest Manager and nothing breaks

7.11ff: Second iteration, more stable, probably still undocumented

File upload

• save raw archives of uploaded packages in a dedicated index
  • unrelated but might be helpful: also cache registry packages in that same index, so intermittently disappearing registries are less disruptive
• add a UI to it? could be hidden (not linked to from anywhere), and the URL only known to us
• add support for .zip
• remove size limit and add support for very large packages (containing data)

Package validation

• add it in the first place
• follow https://github.com/elastic/package-spec
• possibly additional security checks, ensuring a malicious package can't break more than a superuser can on their own (e.g. in cloud)

Package installation

• more refactoring, cleanup, fixing bugs as necessary
• add tests

Integrations page

• Handle packages without errors regardless where we have information about them:
  • registry
  • internal cache
  • saved objects only as fallback
• Improve behaviour when a package installed by direct upload also exists in the registry
  • allow cross-upgrade? or
  • only allow deletion of the package from the integrations page?

[ph]

@skh I like that proposal, thanks for creating clear / steps and goals. for that.

[ruflin]

To potentially make it easier on the UI side, we could have a third tab "local" packages. On tar.gz vs zip we plan to switch to zip for 7.10 so probably best to not do .tar.gz first. Lets also make sure this is an experimental feature at first so we can still make breaking changes. My expectation for 7.10 is internal usage only and "release it" later.

[jsoriano]

save raw archives of uploaded packages in a dedicated index

* unrelated but might be helpful: also cache registry packages in that same index, so intermittently disappearing registries are less disruptive

Are packages needed now after installation?

[skh]

Are packages needed now after installation?

Yes, to display them properly in the UI (screenshots & README, icons in case Kibana doesn't already have an icon for the technology). Right now this causes errors when the registry is unavailable.

[mtojek]

@skh @mostlyjason This draft looks good to me, although I believe there is one area not covered:

1. What would package development look like in the 7.10/7.11?

We're working on enabling the elastic-package tool for package development. It depends on the package-registry now. It would be cool to use the direct package upload feature, but I'm afraid we can't depend on features available for Gold users only. Supporting two options (with/without the package-registry) might not be the simplest option for maintenance.

[ruflin]

@mtojek At the moment most package development happens on clusters with a trial license so I would hope license does not really play a role here?

[mtojek]

@ruflin if we want to improve the user experience, we should consider the direct upload feature. It would eliminate dependency on the package-registry during development and simplify things. Unfortunately the Gold license seems to be a blocker here.

[skh]

Edited: support .tar.gz and .zip from the beginning, since #76197 is already open.

[mostlyjason]

Seems like a substantial investment on something that's not shipped to customers? Can someone help explain the reason for this project and why its valuable? It seems like this is primarily about developer experience, but is there any potential for customer benefit? I always thought running a local registry is as easy as a docker image, but I'm not familiar with the technical reasons.

[mtojek]

Ok, let me play a customer here :) Might be a bit opinionated.

Can someone help explain the reason for this project and why its valuable?

From integration developer perspective, total delivery time for new integration is shorter, the whole environment setup would be simpler, less prone to potential errors, out-of-syncs. No need to lose time to restarts and cleanups. Simply select the integration bundle (.zip) and install it.

but is there any potential for customer benefit?

companies that perform security reviews need to go over every dependency or service and reverify it/validate if it's inline with the security policy. If there is no registry at all or the Ingest Manager is in offline mode (or airgapped), the customer can install only relevant packages and doesn't have to care about policies.

Also, the registry is yet another dependency that can break down. As I said, simpler design means less maintenance issues.

I always thought running a local registry is as easy as a docker image

Docker images will be bigger and bigger every time we update packages, but this is a different issue on our side. At the beginning it won't be a problem for the customer, but they may be confused at some point on how many packages they HAVE to serve in their local setups, but won't use at all.
Running a local registry is a solution, but I bet there can be a better one. Maybe a place or store with limited number of packages? Let's say the approved ones.

[mtojek]

Hey @jen-huang , @joshdover !

Do you it's possible to give this issue a higher priority? It's slipping through iteration by iteration.

cc @akshay-saraswat

[jen-huang]

Hi @mtojek, for 8.0 we are going to tackle this issue of being able to bundle packages with Kibana: #112095

The code changes required for that effort should get us very close to the goal here.

[joshdover]

Note that we have tentatively moved #112095 to a stretch goal for 8.0, it may not land until 8.1

[jsoriano]

@jen-huang @kpollich could this issue be prioritized?

[joshdover]

+1 on this. We're pretty close here. I've been testing the upload endpoint out with my ingest pipeline monitoring POC and it mostly works, but there are a few issues (at least on 8.5.2 where I am testing):

• The package policy editor doesn't use the streams from the uploaded package if another version of the package also exists in the registry
• There's no simple UI for uploading a package zip

[ruflin]

Instead of having a UI to upload packages as a first step, lets add support for it in elastic-package. Having a command like elastic-package upload --kibana=http://localhost:5601 could do the trick. If the command is run inside a package directory, it would directly build and upload the package. Alternative, it could be pointed to a file: elastic-package upload --path=/foo/bar/package.zip --kibana=http://localhost:5601. I will open an issue around this in the elastic-package repo.

[ruflin]

I opened the issue here for the elastic-package implementation: elastic/elastic-package#1084 Using elastic-package for the upload has the benefit that validation of the package can all happen on the elastic-package side for now and does not have to be reimplemented in Fleet. Fleet would just assume it is a valid package.

[jen-huang]

I am closing this long-running issue in favor of #148599 which contains a summary of remaining tasks.

[ruflin]

@jen-huang Good to see we have a more detailed issue around what is still missing. It is a bit unfortunated that this issue got closed as a pretty long list of issues and docs reference this issue. Could we instead keep it open until the linked issue is resolved?

There are 4 checkboxes in this initial issue which are still open and I don't see referenced in your new issue. These are more refactoring tasks but if I remember correctly, pretty foundational to make the new feature happening and stable.