[Fleet] Enrollment API keys in Fleet can go out of sync with security state

[criamico]

Kibana version:
All versions

Steps to reproduce:

• Create a new enrollment API key from Fleet (manually or by creating a new Agent policy)
• From dev tools, verify that the API key is active.

GET kbn:/api/fleet/enrollment_api_keys

{
  "item": {
    "id": "40295421-0da5-4f74-ad1c-bf362ba505db",
    "active": true,
    "api_key_id": "FPjvapEBtiE6A9XKT3yz",
    "api_key": "****",
    "name": "Default (40295421-0da5-4f74-ad1c-bf362ba505db)",
    "policy_id": "5c1f84e6-8f66-4fb3-9b63-c0da0d4fe9a6",
    "created_at": "2024-08-19T13:59:14.553Z"
  }
}

• Check that the api key is valid through the security api as well:

GET /_security/api_key?id=FPjvapEBtiE6A9XKT3yz

• Now invalidate the api key by using the security apis:

DELETE /_security/api_key
{
  "ids":["FPjvapEBtiE6A9XKT3yz"]
}

• Check again the state of the api key in Fleet:

GET /_security/api_key?id=FPjvapEBtiE6A9XKT3yz

The api key is still marked as active, even though it was marked as invalidated from security apis.

Context

When we register a new api key, we create it through the security apis (see

kibana/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts

Line 270 in c2933de

const key = await esClient.security

then we create a new entry in .fleet-enrollment-api-keys.

On the other side, when the GET /api/fleet/enrollment_api_keys gets called, we only query the fleet index. If the api key got invalidated from outside, fleet doesn't know and keeps reporting the api key as active.

We should find a way to sync the state of the api keys in .fleet-enrollment-api-keys with the actual state registered through security to avoid this type of issues.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jillguyonnet]

Linking open question regarding Fleet enrollment token deletion flow: #155550 (comment)

[juliaElastic]

We can probably spend some time in tech definition to figure out the best way to deal with this.
I think users deleting an API key manually is not something we have to fix, as it's not a recommended action.

I've seen a similar issue happening when agent policies became out of sync (due to a bug or error), and agents were still trying to use an old API key reference that was invalidated/deleted in ES.
We could potentially have an API endpoint that would reset/sync the API keys or at least report those that don't exist but are referenced as active.
We should probably also document how enrollment tokens are used in Fleet.