Apply changes to index/component templates to affected data streams

[felixbarny]

Today, Fleet is responsible for both updating the index templates and making sure these updates are applied to the corresponding data streams.

The way how Fleet computes the diff between the current state of a data stream and how it determines whether to update or roll over data streams is complex, flawed (see elastic/kibana#158653), brittle (for example elastic/kibana#158654, elastic/kibana#157869), and shouldn't be the responsibility of Fleet as it's a more general data management problem.

Borrowing ideas from Infrastructure as Code, there should be a way to declare the desired end state of new and existing data streams that match a given pattern and ES is responsible to determine what it needs to do to reach that state.

This docs page explains the different steps user need to do to modify a data stream: https://www.elastic.co/guide/en/elasticsearch/reference/current/modify-data-streams.html

The goal is that there's a single way to update a data stream so that users (including Fleet) don't need to determine whether or not they need to trigger a rollover and don't need to manually create a diff to update each data stream to be consistent with its index template.

---

A possible solution for this could be to add a new parameter to the put index template and component template APIs (something like apply: true). This will determine all matching data streams, and apply the changes needed so that the data stream's mappings and settings are consistent with the index template. It will also automatically determine whether the change requires a roll over of the data stream (for example, if a static index setting is changed or when an existing field mapping is changed) or whether updating the data stream is sufficient (for example, when only dynamic index settings are changed or when only adding fields to the mappings), to avoid creating unnecessary backing indices.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nchaulet]

@felixbarny we are currently having some issues when updating mappings in Fleet where we do not trigger a rollover and where we should, I think having something like apply: true will really help us to solve that issue, do you know if there is any progress made/or plan on that issue?

[felixbarny]

Hey Nicolas, I'm glad you like the idea. We haven't discussed this further.

@dakrone what do you think about this approach?

[felixbarny]

An idea that was brought up by @dakrone is that we could prototype this by not exposing the apply parameter in the REST layer. That way, we can add the diff/apply logic in MetadataIndexTemplateService and certain IndexTemplateRegistrys, such as StackTemplateRegistry and APMIndexTemplateRegistry (#97546) can opt-in to a declarative mode for index templates where it auto-applies the changes.

This approach lets us experiment and iterate quickly, without having to worry about backwards compatibility for exposing this to external clients. Once we're happy with the result, we can expose it so that Fleet and other clients can rely on it.

[felixbarny]

There's a potential issue with this approach that may lead to multiple rollovers.

Consider a situation where the user has the following index and component templates:

index_template_1
index_patterns: logs-foo-default
composed_of: [component_template_1, component_template_2]

index_template_2
index_patterns: logs-bar-default
composed_of: [component_template_1]

Now we update these templates that all include changes that require a rollover:

PUT _component_template/component_template_1?apply=true
-> rolls over logs-foo-default
-> rolls over logs-bar-default

PUT _component_template/component_template_2?apply=true
-> rolls over logs-foo-default

PUT _index_template/index_template_1?apply=true
-> rolls over logs-foo-default

In total, we roll over logs-foo-default three times and logs-bar-default one time. This obviously isn't great. We need to find a way to minimize the rollovers.

Maybe there should be a dedicate API for the "apply" step so that you can first update all templates and then rollover where necessary.

Something like this:

Update all templates first, without triggering a rollover

PUT _component_template/component_template_1
PUT _component_template/component_template_2
PUT _index_template/index_template_1

Then, apply changes and trigger a rollover if necessary

POST _index_template/_apply
-> rolls over logs-foo-default
-> rolls over logs-bar-default

alternatively:

POST _index_template/index_template_1,index_template_2/_apply
-> rolls over logs-foo-default
-> rolls over logs-bar-default

or

POST _index_template/index_template_1/_apply
-> rolls over logs-foo-default

or also

POST _component_template/component_template_1/_apply
-> rolls over logs-foo-default
-> rolls over logs-bar-default

[eyalkoren]

@felixbarny I think the current way that registries work is not suitable for this kind of synchronization, as there is no point in time that indicates that "the registry update is complete". ATM, there can only be an indication that assets upgrades were triggered. Even with a change like proposed in #99647 (comment), I don't know if this will be changed as it means changing the way of how component installation synchronization now works based on continuous comparison between desired and current cluster state.

However, if we apply this proposal only to index templates APIs, I think this multi-rollover problem can be resolved, because a data stream is only "affected" by a single index template. Part of this would be to make sure that an index template only gets upgraded after all its dependencies have been upgraded, like we did in #97546 (and need to complete with #99647). This is valid only if limiting the solution for index templates is acceptable.

Related to that, I think the the idea described above implies that if we indeed do such validation in the "client side" (meaning - the side that submits the API requests), then #99647 may be a blocker