[Ingest Manager] Proposal: Support multiple policy templates per package

[ruflin]

In the Ingest Manager when configuring an integration package, the UI is built based on the policy template in the package (currently called config_templates. In the package, an array of policy_templates is supported but only one is support in Kibana. This issue is to discuss the support for multiple policy templates.

Use case

Having support for multiple policy templates allows creators of packages to offer different default configs to be selected by the user. In the case of the Kafka, the user could select if he wants to monitor partitions, consumers , producers or brokers and based on the selection gets different configurations options offered. In this case, the package would have 4 different policy templates. Like this only 1 Kafka package is needed, but the user can select which part he wants to monitor and only gets the configuration for what he needs as also the Agents running these configs need to be deployed in different places.

An other potential use case is in the system package to provide different defaults on how much data should be collected.

UI

On the UI side, I would expect that if a package contains more then one policy template, the user is offered a selection of the template with potentially a description on what it contains / how it is used.

Behaviour

As soon as a policy is saved for a certain template, the template cannot be adjusted anymore. This is only when creating a new policy.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ruflin]

@exekias @jsoriano We discuss this in the past.

@ycombinator I don't think it will need any changes on the package-spec part but good to be aware of it.

[exekias]

Is this the same as #70608?

[ruflin]

@exekias This feature could be used for data fidelity but the proposal in #70608 is different in that the fidelity is on the dataset level. This policy template selection is also before it is shown what can be collected.

[jen-huang]

Closing in favor of #93315 to support granular integrations - aka multiple policy templates 😉