[Fleet] Fleet server migrations SavedObjectsClient#delete race-condition

[kobelb]

The Fleet server migrations call SavedObjectsClient#delete after creating the .fleet-enrollment-api-keys document:

kibana/x-pack/plugins/fleet/server/services/fleet_server_migration.ts

Line 72 in 3b728b7

await soClient.delete(ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE, key.id);

When multiple Kibana instances are running the migrations in parallel, there's a race-condition and the second call to SavedObjectsClient#delete will cause an error to the thrown. At the moment, this error being thrown will crash Kibana until #89251 is resolved.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

Pinging @elastic/fleet (Feature:Fleet)

[ruflin]

Is there a way to write tests for this to catch these things automatically?

[kobelb]

Is there a way to write tests for this to catch these things automatically?

The only option that we have right now is to use unit-tests to mock the error responses. #79743 would have potentially caught these issues in CI, and #79748 would help developers catch them.

[nchaulet]

Good catch there is another race condition here I missed when we get the enrollment key to decrypt it, we should probably just ignore 404 here.

[nchaulet]

Resolved by #89372