Document how to automatically install APM integration package

[axw]

The APM integration package must be installed regardless of whether APM Server is run as an Elastic Agent integration or as a standalone process. Automatically installing the APM integration package is a common requirement.

We should ensure this is well documented, probably in the APM Guide Quick start section. To install the package, it is enough to include the following in kibana.yml:

xpack.fleet.packages:
- name: apm

This will even work in air-gapped environments, as we bundle the APM integration package with Kibana.

See also https://discuss.elastic.co/t/how-to-install-apm-server-in-legacy-mode-without-elastic-apm-integration/325724/2?u=axw

[bmorelli25]

This will even work in air-gapped environments, as we bundle the APM integration package with Kibana.

It sounds like this might not be the case. @kpollich says:

In an airgapped environment they'll need to have some accessible package registry available to use Fleet. See https://www.elastic.co/guide/en/fleet/current/air-gapped.html.
They should be setting xpack.fleet.registryProxyUrl or xpack.fleet.registryUrl based on whether they're proxying requests to EPR or hosting their own.

[kpollich]

Airgapped environments still require access to EPR in some capacity. This is why we document the need for either proxying requests to epr.elastic.co through an internally available proxy server or for self-hosting EPR using Elastic's provided docker images here: https://www.elastic.co/guide/en/fleet/current/air-gapped.html

The "inaccessible EPR" case differs from the general airgapped case, and we've got an issue for making the experience around inaccessible EPR environments better here: elastic/kibana#136617

[axw]

@kpollich even for preconfiguration? If so, why? Can that be changed?

[kpollich]

even for preconfiguration? If so, why? Can that be changed?

I should clarify. The bundled integration will still be installed via preconfiguration if there's no reachable package registry. When I say "a reachable instance of EPR is required" I mean required for operation of the Fleet and Integrations apps. Even without a reachable EPR configured we can still install + configure the APM integration and ingest data.

As for the "why" - Fleet will always try to reach out to EPR to determine whether a given package is out of date. The "update required" status for packages is driven entirely by what's reported by EPR, so we're always checking with that service when packages are queried, installed, etc. There's work to do on our end around intelligently disabling those "is package up to date?" style requests, and failing fast when EPR isn't reachable.

Some related issues:
elastic/kibana#128130
elastic/kibana#131505
elastic/kibana#125548

[axw]

@kpollich thanks for clarifying.

So the goal of this docs issue is just to document how to install the package automatically, and anything related to interacting with the Fleet/Integrations apps was meant to be out of scope.

Maybe we can just mention that in airgapped environments, interacting with the app still requires access to a self-hosted EPR, and link to the Fleet air-gapped environment setup docs, which go into self-hosting EPR etc. We should be careful not to leave users with the impression that this is required for installation though.

[lucabelluccini]

Hello, I'm chiming in on this issue as some users are getting affected by this problem in air-gapped environments.
I think we have 2 problems.

1. Users with APM Server Standalone experience Fleet is mandatory, but it is not as the APM Package is bundled with Kibana (at least in recent versions). Imagine a user who doesn't want to use any Fleet feature: they would be unable to install the integration.

2. Regardless of having Fleet correctly setup or not, Kibana Fleet plugin should be aware of the packages available locally and should allow installing them via UI, skipping reaching out to EPR. Bonus points if the UI also warns about the fact EPR is not available from Kibana server.
   It could be seen like a Fleet degraded mode, where only the packages available locally are reachable.
   This would be an issue to create or follow up on the Kibana / Fleet side (not 100% sure).

Mitigation

We could document a valid workaround (seems to work fine without EPR access):

POST kbn:/api/fleet/epm/packages/apm/<stack version>
{ "force": true }

And the one suggested by @axw.

Next steps

Would it be possible to follow up with the APM/Fleet PMs on how to make APM not completely dependant on Fleet being fully functional?

[kpollich]

@lucabelluccini - Thanks for chiming in. You are correct that installation via the UI still requires an EPR connection. The entire integrations UI is not built to support an inaccessible package registry, even with the presence of bundled packages. Installation via the API is still functional, though, and the expected behavior is that in cloud environments where APM is set up as a default package, it will be automatically installed during Kibana's startup even if EPR is not reachable.

elastic/kibana#136617 captures this issue well, I think, and we'd just need to prioritize it on the Fleet side to ship a fix.

cc @mukeshelastic - It'd be good to get PM input here and note the request to prioritize the above. I believe you're responsible for anything related to integrations UI now, so you're hopefully the right person to ping 🙂

[AlbusLumos]

Hi, I add the below setting in kibana.yml

xpack.fleet.packages:
  - name: apm

but I got the error message: [ERROR][plugins.apm] FleetUnauthorizedError: User does not have adequate permissions to access Fleet packages.

How to fix it?

[kpollich]

Hi @AlbusLumos - your user needs the Integrations:All permissions from Kibana's role management UI.

• https://www.elastic.co/guide/en/kibana/current/kibana-role-management.html
• https://www.elastic.co/guide/en/kibana/current/kibana-privileges.html

[AlbusLumos]

Thanks, @kpollich
But it's test environment, so it didn't enable the security.
Does it means that it must enable security?
If security is disabled, why would such a check still be performed?

[juliaElastic]

@AlbusLumos AFAIK security is enabled by default, and Fleet doesn't work with disabled security. The Integrations:All is a required permission to install integration packages.

[GyurkanM]

Option to use it with security disabled will be welcome, no need for security when it is deployed on an isolated VM for demonstration, testing purposes.

[simitt]

The support to disable security depends on Fleet support (part of Kibana). @juliaElastic does an enhancement request exist for this and could you link it here if so (I didn't manage to find it).

[axw]

@simitt I opened this a while back: elastic/kibana#160486