[Ingest pipelines] Adding descriptions for ingest processors

[jloleysens]

Since the introduction of ingest pipelines UI (elastic/kibana#66021) in Kibana, we added the ability to provide descriptions for ingest pipeline processors (#57906).

Recently a user reported (elastic/kibana#94432) that it would be great if ingest pipelines not created by users had more helpful descriptions where appropriate (possibly on all processors). This bit of housekeeping would improve the UX of ingest pipelines in the following ways:

• provide a higher-level description of what each processors is intended to achieve at a glance
• create a more curated experience for users

[jloleysens]

CC @jethr0null

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[jethr0null]

I don't believe this is a core features ask given that the team isn't authoring the integrations pipelines @jakelandis please correct me if you see this differently.

@mukeshelastic @jamiehynds would you mind suggesting who we might work with on your teams to explore the option of adding descriptions to the integrations pipelines that we're building and shipping.

[jakelandis]

I don't believe this is a core features ask given that the team isn't authoring the integrations pipelines @jakelandis please correct me if you see this differently.

+1, I think this needs to be added by the teams that own the pipelines. We have (and have for a while) the ability to set a description at the pipeline level. For example

PUT _ingest/pipeline/foo
{
  "description": "my-pipeline", 
  "processors": [
   ...
  ]
}

[jamiehynds]

Thanks for raising this @jethr0null. I'll discuss with my team.

@P1llus - can discuss on our sync, but something we can consider as we're moving integrations towards ingest node.

[P1llus]

I don't believe this is a core features ask given that the team isn't authoring the integrations pipelines @jakelandis please correct me if you see this differently.

+1, I think this needs to be added by the teams that own the pipelines. We have (and have for a while) the ability to set a description at the pipeline level. For example

PUT _ingest/pipeline/foo
{
  "description": "my-pipeline", 
  "processors": [
   ...
  ]
}

I think this could be quite useful, both for the users and for employees that are new to a specific pipeline.

The description set per pipeline is correct, but newer versions now also allows descriptions per processor, which is what they are asking for here, Il bring it up in our sync today to see what we can do there.

[jamiehynds]

@jethr0null - we were just discussing how to approach descriptions across our security integrations. Do you think automation on the UI side is possible, to auto-generate a description for basic processing. e.g. the lowercase processor would have a default description 'Converting field foo.bar to lowercase equivalent'?

[jloleysens]

Do you think automation on the UI side is possible

I'm not sure how automating/generating descriptions for processors might work. But default descriptions for each processor can be prepared and added to the UI.

I think the most useful purpose a processor-level description can serve is to give more info on why it is there. If we have a lowercase processor: "Lowercase field x.y.z to avoid issues when comparing to a.b.c" would be the kind of thing that is most useful. Given that, I'm not sure how useful a generic description of a processor would be. Not held as a strong opinion.

[Edit]
There is not reason we cannot have both of course :)

[cjcenizal]

@elastic/es-docs WDYT of updating our Ingest Node Pipeline docs to include the processor description field in the code examples? I think this will raise awareness of this feature with pipeline authors and also make the code examples easier to understand.

We'd probably want to update all code examples that showcase processor definitions, throughout the docs. 😬 Might be a lot of work...

[jrodewig]

I recently refactored the ingest docs in #70253 so I'll try to answer for the ES docs team.

We'd probably want to update all code examples that showcase processor definitions, throughout the docs. 😬 Might be a lot of work...

We only have ~80 total _ingest/pipeline snippets in the docs along with a couple UI screenshots. Many of those snippets highlight particular API parameters and wouldn't necessarily need an update. I'd be happy to handle these doc updates.

Thanks for raising this idea! I think these default processor descriptions will be helpful.