[Fleet] Cannot use apm integration in elastic-package

[ruflin]

I've been playing around with elastic-package and the apm package. Since version 8.5. there is a problem when I want to use APM. In 8.4 it all still works. This is independent of if I use a snapshot or a released build. My assumption is that it has to do with the bundled package change: #137759

First about the error I see and how to reproduce it:

• Spin up elastic-package stack up --version=8.5.3 (or 8.7.0-SNAPSHOT)
• Add the APM integration to one of the elastic agents.

After a long wait, the following error message is shown:

For some reasons, it doesn't seem to take the bundled package but instead tries to get it from the package registry. This is the reason I file this issue here as I assume this is a Fleet issue.

The latest package that exists in the registry is 8.4.2: https://epr.elastic.co/search?package=apm I tried to get out of #137759 on where we landed around still shipping the bundled packages to the registry but it is not clear to me. On my end I still think we should ship a bundled packages to the registry. Especially now with v2 and the prerelease flag, we can continously add the packages. It shouldn't be the solution to this problem here as it would just hide it.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[simitt]

pinging @elastic/apm-server for visibility

[kpollich]

This is a duplicate of #136617. The root cause is that "GET integration" API always tries to use the registry even if the package is bundled.

On my end I still think we should ship a bundled packages to the registry. Especially now with v2 and the prerelease flag, we can continously add the packages. It shouldn't be the solution to this problem here as it would just hide it.

I also agree with this. Shipping many prerelease version of the APM integration to the registry should no longer be a concern in v2. My memory is a little fuzzy on the specifics around the long thread of comments on #137759 (I think there were numerous offline conversations as well), but @axw can probably help clarify some of the motivations for no longer publishing APM to package storage at all.

[ruflin]

Closing this as a duplicate. I'm a bit concerned that #136617 has been open for quite a while.

@axw Can we start to publish apm packages again?

[axw]

I also agree with this. Shipping many prerelease version of the APM integration to the registry should no longer be a concern in v2. My memory is a little fuzzy on the specifics around the long thread of comments on #137759 (I think there were numerous offline conversations as well), but @axw can probably help clarify some of the motivations for no longer publishing APM to package storage at all.

We want the APM integration package:

• to either be automatically installed (e.g. in Cloud), or always available to be installed (including in air-gapped environments)
• to be exactly the same version as Kibana

This means users don't have to think about another version, and removing the possibility of skew means we can reduce the number of possible combinations of APM UI code & integration package version, improving supportability.

@axw Can we start to publish apm packages again?

If there's a benefit to users. Seems to me that #136617 could be fixed by preferring the bundled package.