[RAC][Rule Registry] Index bootstrapping: make index upgrade logic compatible with adding field aliases and runtime fields to old indices for backwards compatibility

[banderror]

Parent ticket: #101016
Related to: #109293

Summary

(Address before the next release that makes any mapping changes, possibly as soon as 7.15.1)

Work on improving the index upgrade logic because upgrading the mappings in place is potentially incompatible with the plan to add field aliases and runtime fields to old indices for backwards compatibility. If the mapping on a single index can change over time, it's hard to define what aliases/runtime fields would need to be added to make it compatible with the new schema.

Open questions:

• Should we update the mapping in place if we're only adding a new field? Pro: limits oversharding. Con: divides alerts in the index into old/new, harder to apply some types of backwards compatibility changes
• Right now the in-place index upgrade only applies new mappings. Can/should we update index settings as well?

Background

The background for this is our discussions with @kobelb (see #109276 (comment) and above comments) on the "compatibility" of the current index upgrade logic with the ideas for backwards compatibility (#109293).

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[banderror]

Hey everyone, I removed this ticket from the backlog of the Detection Rules area. We (@elastic/security-detections-response-rules) are not the owners anymore (however feel free to still ping us if you have any tech questions about the ticket).

Ownership of this ticket and other tickets related to rule_registry (like #101016) now goes to the Detection Alerts area (Team:Detection Alerts label). Please ping @peluja1012 and @marshallmain if you have any questions.

[marshallmain]

Index upgrade logic shipped with upgrade-in-place strategy and no plans for aliases/runtime field support on old indices.