[REQUEST]: Discourage cloning index templates - offer easier option to customize settings if not using namespaces + Warning/Disclaimer page

[lucabelluccini]

Description

We're aware of the issues cloning index templates of Fleet integration leads to.

We need to attempt to warn users / push users to not use index template cloning or edit the index & component templates.

The page https://www.elastic.co/guide/en/fleet/current/data-streams-ilm-tutorial.html should be reviewed to display 2 options:

• one in case the user wants to edit the ILM policy (this would involve creating a @custom component template to override the ILM policy or edit the default ILM policy coming with the integration) - point to https://www.elastic.co/guide/en/fleet/current/data-streams.html#data-streams-ilm
• one in case the user wants to have a namespace specific ILM policy (this will require index template cloning)
  • for the namespace specific customizations, we need to point to a new "warnings/disclaimer" page

Further enhance https://www.elastic.co/guide/en/fleet/current/integrations-assets-best-practices.html

• Never edit managed assets of Fleet (e.g. the Index template, the @package component templates and/or ingest pipelines bundled with integrations). Any change will be overwritten upon upgrade/install.

• When the user creates a @custom component template for a package integration, while allowed and legit, the user can incur in some risks which might prevent the correct ingestion of data.

  • It can lead to broken indexing, data loss, broken integration packages upgrades.
    • Example: the @package component template of the integration switches from a "normal" data stream to TSDB or LogsDB and some settings/mappings introduced by the user are not compatible with such indexing modes.
    • Example: the user overrides the type of an ECS field from keyword to text - this might prevent the aggregation on such field on built-in dashboards.
    • We warn about this on this page

• When users create a @custom ingest pipeline as we document here, Elastic is not responsible of the correct indexing/behavior of the ingest pipeline. It becomes a custom processing of the data and it's user responsibility to test it.

  • We might add the warning here

• When a user clones the index template of an integration package, the user can incur in the risk of not picking up index template major changes upon upgrades. The "structure" of the index template will be "frozen" at the moment it was cloned.

  • It can lead to broken indexing, data loss, broken integration packages upgrades - because the structure of the Index templates which ship with Fleet can change over minor version and the user might lag behind those changes or pick up only a part of them.
  • Covered in Allow customizing managed data streams at different levels of granularity elasticsearch#97664 (comment) and ingest-dev/3617
  • This is "problematic" as we document to clone the index template at this page...

• Cloning index templates to add/inject additional component templates cannot be tested by Elastic, so we cannot guarantee it can work in future release and it can break at any minor release.

Resources

Referenced above

Collaboration

TBD. The docs and product team will work together to determine the best path forward.

Point of contact.

Main contact: @lucabelluccini

I can help with a draft.

[lucabelluccini]

FYI @jsoriano / @kilfoyle

Some of those warnings are already in the docs - but we need to start to better frame the message and raise more warnings.

[kilfoyle]

@lucabelluccini to get us started, I've opened this PR to add the new warnings you suggest for the Best practices for integration assets page. Please let me know whatever you think we should change.

For the first part, about updating the Customize data retention policies tutorial with two different options, I haven't worked much with component templates, etc., and would really appreciate if you or anyone can suggest exactly what steps should be added/changed.

[lucabelluccini]

For the first part, about updating the Customize data retention policies tutorial with two different options, I haven't worked much with component templates, etc., and would really appreciate if you or anyone can suggest exactly what steps should be added/changed.

Hello @kilfoyle

The key is the current guide we have at https://www.elastic.co/guide/en/fleet/current/data-streams-ilm-tutorial.html is focusing on customizing the ILM policy for a specific namespace.

We need to discourage it as we want the majority of users want to customize the ILM policy for all namespaces. And customizing the ILM policy for all namespaces should not involve cloning the index template. It's a pitfall a lot of users are following generating very painful recovery procedures.

[lucabelluccini]

If you need assistance please book a slot next week so we can go over it together.

[kilfoyle]

Thanks @lucabelluccini!

I updated my pull request so that it adds a prominent warning in that "Customize data retention policies" tutorial. If you can have a look at my comment here, and let me know if we need to rephrase things, I'd appreciate it. Whatever changes you have in mind I'm happy to add in.

[kilfoyle]

If you need assistance please book a slot next week so we can go over it together.

@lucabelluccini I really appreciate the offer but given my backlog and a lot of other competing priorities I honestly have very little time to devote to investigate this, test the updated steps, etc.

@jlind23 This Customize data retention policies tutorial needs some reworking as Luca has written in the description, and also in the docs PR (copied below). Is there a developer familiar with this area who might be able to assist and perhaps add this to a sprint?

---

The page should be split into 2 separate sections:

How to customize ILM policy for a data stream belonging to an integration: simple case, 95% of situations, just create/edit the @Custom component template and add the ILM policy.
How to customize ILM policies for a data stream belonging to an integration for a specific namespace: advanced case, 5% of situations, steps as they're described now

---

[jlind23]

@kilfoyle unfortunately Customize is not owned at all by Ingest.
I hope @mlunadia will be able to assist here.
cc @nimarezainia

[nimarezainia]

@kilfoyle unfortunately Customize is not owned at all by Ingest. I hope @mlunadia will be able to assist here. cc @nimarezainia

@jlind23 this is not about kubernetes :-) it's a page discussing datastreams and creating a customized ILM policy.

[jlind23]

I'm so sorry I misread @kilfoyle's comment 🤦🏼
@elastic/fleet would someone be able to assist David here?

[lucabelluccini]

Happy new year!
I can assist myself if we can book a time slot together with @kilfoyle.
David feel free to pick a slot so we can coordinate on this ❤ .

[kilfoyle]

Sounds good. Thanks @lucabelluccini, and happy new year!

I booked a slot on your calendar for Thursday. If that's not convenient please feel free to pick another time.

I've opened this gdoc for us to capture whatever needs changing in the tutorial:
https://docs.google.com/document/d/18i16TPOaXh-ipFnJnGw7mWC_gpKt1EifLrcW21qk7Ls/edit?tab=t.0

If we can get that gdoc marked up during our meeting I can port the changes over into the docs. Thanks!

[lucabelluccini]

I've done a pass already to the doc and added briefly what we should add. During the meeting I'll expand and answer any questions you might have or things I've missed. Thank you David

[kilfoyle]

Thanks for doing that pass @lucabelluccini!

We'll meet on Thursday, but in advance of that I wanted to let you know I have made a draft mock-up based on your suggestions. This would split the single tutorial into three scenarios, like this:

For now, Scenario 1 and Scenario 2 are exact copies of https://www.elastic.co/guide/en/observability/current/apm-ilm-how-to.html#apm-data-streams-custom-policy, since you've said that the procedure in the scenarios is practically the same. I've copied everything into this google doc, so if you don't mind I'd really appreciate if you can indicate exactly what changes are needed.

Note: For Scenario 1 you mentioned a "only on >= 8.12+" and a "only on >= 8.4+ and < 8.12" version. To avoid making these docs too complex I recommend documenting only the 8.12+ version, especially since I can only update current docs and not earlier versions.

Thanks!

[kilfoyle]

Thanks a lot for the demo and info, Luca!
The PR is all ready here: #1623

[kilfoyle]

Thanks again @lucabelluccini
This is now fixed via #1623