Updating integrations and field mappings

[philippkahr]

Hi,

certain integrations can be automatically updated in the background and that could lead to certain issues. Let's assume the following:

1. I use the integration fancy-data-source in version 1.0.
2. We made a mistake and called the field user_name instead of user.name.
3. The integration runs for ~5 months, we release version 2.0 with the user.name fix.

However, as an user I already built dashboards, searches, rules and alerts, ... based on the user_name. Maybe I even added runtime fields that need that field. All of the sudden all of the things are broken and I will need to do a deep dive to find out why the field user_name is missing.

As an user I would expect that an integration update:

1. verifies if a field is leveraged in any Kibana feature such as Dashboards, Visualisations, Maps, Canvas, Rules and Alerting, ...,
2. if it is used somewhere
   2.1 it should give me a warning / information that tells me we renamed user_name to user.name.
   2.2. it should give me an option to automatically reconfigure everything that uses the user_name to user.name.
   2.2.1. it should tell me what dashboards, viz, rules, where modified.
3. If it is not used somewhere, just upgrade the integration.

[joshdover]

We only upgrade a few packages without user intervention, all others require the user to click something. The integrations which update automatically are apm, synthetics, endpoint, elastic_agent, fleet_server, and system (but only on versions prior to 8.1, I believe).

The ones that we do upgrade must be upgraded with the Stack or else our 1st party UIs will break, so I don't see a way for us to prevent that.

That said, there shouldn't be breaking changes to field names in these packages unless the upgrade explicitly bumps their package to a new major version. I don't think we have any automation in place to prevent this though. Correct me if I'm wrong, @jsoriano.

We also don't have any UI that guides the user through a major upgrade of a package. This is something that we should improve on, including a changelog or summary of the breaking changes to field names or mappings.

1. verifies if a field is leveraged in any Kibana feature such as Dashboards, Visualisations, Maps, Canvas, Rules and Alerting, ...,

This could work but will be quite a bit of work since these are all owned by different parts of the company. I'd suggest we start with something simpler first and see how much that buys us.

[jsoriano]

That said, there shouldn't be breaking changes to field names in these packages unless the upgrade explicitly bumps their package to a new major version. I don't think we have any automation in place to prevent this though. Correct me if I'm wrong, @jsoriano.

That's right, we don't have anything now preventing breaking changes. We rely on package developers and their review and approval process. We have an open issue about providing automated assistance on this: elastic/elastic-package#579.

We also have some open discussions about supporting deprecations and breaking changes:

• Discuss: guidelines for a package with breaking changes package-spec#229
• Discuss: deprecation path for packages package-spec#192
• Support package deprecation package-spec#227

We also don't have any UI that guides the user through a major upgrade of a package. This is something that we should improve on, including a changelog or summary of the breaking changes to field names or mappings.

Agree, we should work in this use case, this is something that is going to be eventually needed as packages evolve.

[jsoriano]

• We made a mistake and called the field user_name instead of user.name.
• The integration runs for ~5 months, we release version 2.0 with the user.name fix.

Btw, take into account that this is not a new problem, this has been also happening in Beats for years. What we use to do there is to duplicate the field till next major where we can introduce the breaking change and remove the old field. Similar approaches can be followed on integration packages, but we still have to work on the UX to upgrade between majors.

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!