[Fleet][Discuss] Integration-specific saved object migration

[jfsiii]

Follow up ticket from #94171

Background

There are two places where fleet runs a saved object migration only for a specific integration. In both cases security_solution updates the package policy saved object.

Questions

How common a need do we think this is?

It's only security_solution at the moment, but might apm need this? Should nginx be able to do this?

Is the current approach reasonable or should we do something else?

Initially the migration code was stored in the plugin, and fleet imported and ran it. That caused a circular reference between the two, so #94171 moved it into fleet.

Adding integration-specific code in fleet feels like something to avoid. Each new integration-specific migration requires commit to fleet code. Ideally, the migration code is defined in the various plugin directories like before but if a) fleet has to run them b) at startup we're back at the circular reference issue.

Must fleet run these migrations? Can we push the definition and execution into the plugins?

Perhaps this really is for a small set of exceptional cases and we can live with it. I just want to get a better sense of what type/rate of tech debt this is.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

I think this could be related to the package upgrade discussion we have currently as there is some "bugs" in the current implementation with saved object migration as the agents do not get the new policy immediately but only when the agent policy will be updated.

[ruflin]

Agree that fleet should not run integration specific migrations. But I see some reasons that solutions with a package want to provide special logic for some of the migrations. What is the reasons this needs to be triggered by fleet? Couldn't the code be in Endpoint for example and then Endpoint only talks to Fleet API's to do a migration?

[paul-tavares]

Following our sync meeting this past week:

I think migration for integration policies fall in two categories:

• migrations needed as part of stack upgrade
• migration needed as part of a package install (upgrade)

We have not had the need for migration at package install time. We have been using the "stack upgrade" approach by "plugin into" the fleet SO migrations and only processing integration policies that are endpoint (I realize now that the updates never make it down to the agent/endpoint until a user changes something again). We need this for a few reasons:

1. new features are introduced with endpoint security that we would like to apply defaults to for existing policies (ex. introduction of malware, etc)
2. compatibility with existing stack changes (ex. move to fleet-server)

From the cases I have seen thus far, I think both of these endpoint cases could be addressed by moving the migration to the package install activity and allowing an extension point to be register a callback that would then perform the migration. I say this because Endpoint integration is set as a required package and I think (🤔 ) it is upgraded auto-magically whenever a new version of it shows up. But that would also imply that we would all need to develop the infrastructure for running package integration migrations (like: accounting for skipped versions, etc..).

The goal ultimately is: to ensure that policies remain compatible with the existing version of the stack before the APIs are available for use and/or the package is upgraded.

/cc: @scunningham , @kevinlog

[paul-tavares]

One last thing:
The immediate need is for v7.13 - endpoint needs to migrate their Integration Policies in order to adjust the relative URL for artifacts, since those artifacts are now downloaded from fleet-server and not kibana. It feels as if this issue here will require a longer time frame to brainstorm and design a migration path for integrations.

@jfsiii , @ruflin , @nchaulet : How do you all feel about us continuing to use the existing SO migration for now until we can come up with a better approach?

[nchaulet]

@paul-tavares I think for the specific case of 7.13 and the move to fleet server it will make sense 👍

[paul-tavares]

Thanks @nchaulet . BTW - I think I need to bring back the feature flag you removed (temporary). our migration should only run if fleet-server enabled until we shut down the kibana agent code

[paul-tavares]

Note: I removed myself from the assignee field. Not sure that I should be assigned to this, but am glad to contribute/collaborate on a possible solution.