[Fleet] Installing an uninstalled bundled package should not require a reachable EPR in integrations UI

[axw]

Some integration packages, like APM, are bundled with Kibana. These can be installed using the Fleet preconfiguration settings. I would also expect them to be usable when not using preconfiguration, and installing the package via the Integrations UI.

When navigating to the Integrations UI with EPR inaccessible, the UI presents a spinner and the well-defined Elastic integrations at the top:

If you click on Elastic APM, it takes you to the APM tutorial, and you would have to navigate back to the Integrations UI to install the package. The link back to the Integrations UI spins, because EPR is inaccessible:

Eventually it loads, and then you get taken to the APM integration, which fails to load:

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jen-huang]

@axw Thanks for filing! We actually had this tracked in #134326 but I'm going to close that one in favor of yours as yours also includes the problem on the package detail page.

[axw]

@jen-huang not sure if it warrants two separate issues or not, but I think they're a bit different. #134326 is about listing installed integrations without EPR access, whereas this issue is about listing uninstalled but bundled packages without EPR access.

[juliaElastic]

I found a similar issue mentioned here that package assets (icons, readme) are not loading on Integration Overview page if the package is not in EPR e.g. apm-8.6.0

[criamico]

I tried to reproduce locally with following procedure on Kibana 8.12.0 (dev).

• Set up kibana.dev.yml:
  • Configure xpack.fleet.registryUrl to an unreacheable host
  • Make sure that APM and other bundled packages are not present in preconfiguration
• Dowloaded apm and endpoint from EPR and copied them in kibana/x-pack/plugins/fleet/target/bundled_packages. This should ensure that the bundled packages are present.
• Go to an uninstalled bundled package page, in my case endpoint (apm was already installed). The pageapp/integrations/detail/endpoint-8.12.0/overview loads properly with no errors.
• Try to add the integration. The integration policy editor loads properly:
  
• Click Save and Continue. Here's where the process fails, but in a different way than what's described above. The page spins for a while and in the logs I can see Failed to fetch latest version of endpoint from registry: Error connecting to package registry: request to http://localhost:8080/search?package=endpoint&prerelease=false&kibana.version=8.13.0 failed, reason:. Then it seems to install but fails after a while with:

[2024-02-05T16:26:06.561+01:00][DEBUG][plugins.fleet] retrieved installed package endpoint-8.12.0 from ES
[2024-02-05T16:26:06.561+01:00][DEBUG][plugins.fleet] retrieved installed package endpoint-8.12.0 from ES
[2024-02-05T16:26:06.565+01:00][DEBUG][plugins.fleet] Creating new package policy
[2024-02-05T16:26:06.565+01:00][DEBUG][plugins.fleet] Creating new package policy
[2024-02-05T16:26:06.565+01:00][DEBUG][plugins.fleet] Running 6 external callbacks for packagePolicyCreate
[2024-02-05T16:26:06.565+01:00][DEBUG][plugins.fleet] Running 6 external callbacks for packagePolicyCreate
[2024-02-05T16:26:07.090+01:00][ERROR][plugins.fleet] An integration policy with the name New already exists. Please rename it or choose a different name.
[2024-02-05T16:26:07.090+01:00][ERROR][plugins.fleet] An integration policy with the name New already exists. Please rename it or choose a different name.

This is also visible in the page:

So it seems that we get past the previous error described above, which could be explained by the fact that this issue is quite old and there were changes done to this area of the code. This error is now about the package policy name already existing, but I'm not sure where's coming from. I'll dig a little bit further.

@kpollich Are my assumptions about the configuration correct? I'm not sure if my results are different because of different configuration or this code path just changed.

[criamico]

Also, by reading the related issues it seems that a very good enhancement could be done by implementing what's described here. This would allow the requests to be shortened and getting to the error much faster. In my test it took at least a couple of minutes to get to the above error, and the whole navigation was really slow.

[kpollich]

Also, by reading the related issues it seems that a very good enhancement could be done by implementing what's described here. This would allow the requests to be shortened and getting to the error much faster. In my test it took at least a couple of minutes to get to the above error, and the whole navigation was really slow.

Yep this is accurate. We do have the xpack.fleet.isAirgapped flag now that should be usable for the same purpose.

Try to add the integration. The integration policy editor loads properly:

The only thing I think that differs in your steps to reproduce and what was reported initially here is that other parts of the integrations app were causing issues. Namely the integration details page. I think this may have changed recently too, as we use the packageInfo object sourced from the package info as the "source of truth" for much of this UI now.

[criamico]

Summarizing here what I found so far. With the setup described in #136617 (comment), I tried to install apm via API on an existing agent policy:

POST kbn:/api/fleet/package_policies
{
  "name": "apm-new-8",
  "policy_id": "4a38a321-7020-4cc2-ad9a-ac57ac33c499",
  "package": {
    "name": "apm",
    "version": "8.12.0-preview-1701143518"
  }
}

The request fails with 409 conflict:

 An integration policy with the name apm-new-8 already exists. Please rename it or choose a different name.

It's interesting that the package policy appears as installed in the UI, even though the request completes with the error.

Investigation

The reason seems to be that PackagePolicyService.create gets called at least twice with the same package policy name, hence causing the conflict.

I checked the external callbacks visible in the logs:

@kpollich pointed me to the apm callbacks that trigger a second update of the package policy after creation, but it doesn't explain why the whole creation process happens again after the first time.

I'm attaching Kibana logs below since they're too long to be pasted here. The line Creating new package policy (pointing here) gets called in two distinct places and the second time the policy creation fails.

kibana_logs.json

Currently I'm not sure of the root cause of this error; The same process happens when the registry is reachable but no conflict error happens.

[kpollich]

@criamico - Thank for investigating this. I'm trying to get a repro locally but running into some trouble. Even with the bundled packages set up a basic GET request for a package spins for several minutes until I eventually get a 503, e.g.

GET kbn:/api/fleet/epm/packages/apm/8.13.0

{
  "statusCode": 502,
  "error": "Bad Gateway",
  "message": "Error connecting to package registry: request to http://localhost:8080/package/apm/8.13.0/ failed, reason: "
}

The bundled packages seem to come back properly, so I'm not sure what's breaking. I'll continue to troubleshoot, but curious if maybe I've missed a step.

[kpollich]

I think this is worth looking into again with fresh eyes. Moving out of blocked.

[criamico]

@kpollich Took a look at this for ON week, I have a PR where I introduced the xpack.fleet.isAirGapped configuration key to allow installing bundled packages even in case of unreachable registry.