[Ingest pipelines] Adding descriptions for ingest processors

[jloleysens]

Since the introduction of ingest pipelines UI (elastic/kibana#66021) in Kibana, we added the ability to provide descriptions for ingest pipeline processors (#57906).

Recently a user reported (elastic/kibana#94432) that it would be great if ingest pipelines not created by users had more helpful descriptions where appropriate (possibly on all processors). This bit of housekeeping would improve the UX of ingest pipelines in the following ways:

• provide a higher-level description of what each processors is intended to achieve at a glance
• create a more curated experience for users

[jloleysens]

CC @jethr0null

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[jethr0null]

I don't believe this is a core features ask given that the team isn't authoring the integrations pipelines @jakelandis please correct me if you see this differently.

@mukeshelastic @jamiehynds would you mind suggesting who we might work with on your teams to explore the option of adding descriptions to the integrations pipelines that we're building and shipping.

[jakelandis]

I don't believe this is a core features ask given that the team isn't authoring the integrations pipelines @jakelandis please correct me if you see this differently.

+1, I think this needs to be added by the teams that own the pipelines. We have (and have for a while) the ability to set a description at the pipeline level. For example

PUT _ingest/pipeline/foo
{
  "description": "my-pipeline", 
  "processors": [
   ...
  ]
}

[jamiehynds]

Thanks for raising this @jethr0null. I'll discuss with my team.

@P1llus - can discuss on our sync, but something we can consider as we're moving integrations towards ingest node.

[P1llus]

I don't believe this is a core features ask given that the team isn't authoring the integrations pipelines @jakelandis please correct me if you see this differently.

+1, I think this needs to be added by the teams that own the pipelines. We have (and have for a while) the ability to set a description at the pipeline level. For example

PUT _ingest/pipeline/foo
{
  "description": "my-pipeline", 
  "processors": [
   ...
  ]
}

I think this could be quite useful, both for the users and for employees that are new to a specific pipeline.

The description set per pipeline is correct, but newer versions now also allows descriptions per processor, which is what they are asking for here, Il bring it up in our sync today to see what we can do there.

[jamiehynds]

@jethr0null - we were just discussing how to approach descriptions across our security integrations. Do you think automation on the UI side is possible, to auto-generate a description for basic processing. e.g. the lowercase processor would have a default description 'Converting field foo.bar to lowercase equivalent'?

[jloleysens]

Do you think automation on the UI side is possible

I'm not sure how automating/generating descriptions for processors might work. But default descriptions for each processor can be prepared and added to the UI.

I think the most useful purpose a processor-level description can serve is to give more info on why it is there. If we have a lowercase processor: "Lowercase field x.y.z to avoid issues when comparing to a.b.c" would be the kind of thing that is most useful. Given that, I'm not sure how useful a generic description of a processor would be. Not held as a strong opinion.

[Edit]
There is not reason we cannot have both of course :)

[cjcenizal]

@elastic/es-docs WDYT of updating our Ingest Node Pipeline docs to include the processor description field in the code examples? I think this will raise awareness of this feature with pipeline authors and also make the code examples easier to understand.

We'd probably want to update all code examples that showcase processor definitions, throughout the docs. 😬 Might be a lot of work...

[jrodewig]

I recently refactored the ingest docs in #70253 so I'll try to answer for the ES docs team.

We'd probably want to update all code examples that showcase processor definitions, throughout the docs. 😬 Might be a lot of work...

We only have ~80 total _ingest/pipeline snippets in the docs along with a couple UI screenshots. Many of those snippets highlight particular API parameters and wouldn't necessarily need an update. I'd be happy to handle these doc updates.

Thanks for raising this idea! I think these default processor descriptions will be helpful.

[P1llus]

@jloleysens After some discussion, I think the easiest approach would be this:
On almost all processors, it would be possible to make a sufficient auto-generated description, with the exception of maybe grok, script and dissect

I think the behavior should be that, if there is not a description field on the processor, and not one of the above processors, the description would be automatically generated based on specific fields in the processor, here is some examples:

rename processor

{
  "rename": {
    "field": "provider",
    "target_field": "cloud.provider"
  }
}

Results in a description like:
Renames the {field} to {target_field}

I think we just have to go through each processor, and create a list of each field that should be mapped where.

If this is a approach you would agree with, then I can help with creating the text and fields used for this generation for each processor.

[jloleysens]

@P1llus this makes sense to me! Happy to work together on the approach.

[P1llus]

@cjcenizal Do you agree with the above statement? It would solve a lot of the usecases right away, and give us more time to create more descriptions.

[cjcenizal]

@P1llus Good idea! I've created elastic/kibana#95486 to track the enhancement of dynamically generating simple descriptions for processors where one hasn't been defined. Feel free to chime in there with anything I might have missed.

I think the behavior should be that, if there is not a description field on the processor, and not one of the above processors, the description would be automatically generated based on specific fields in the processor

I was a bit confused by the bolded bit above though. Why would the preceding processors affect whether any given processor would benefit from a description?

[cjcenizal]

I can help with creating the text and fields used for this generation for each processor

Thank you for volunteering to help, BTW. That would be great! Let's collect this information on the issue I created.

[P1llus]

I guess this one can now be closed @jloleysens ?

[jloleysens]

@P1llus I think so, elastic/kibana#95507 was the first step.

It would probably still be useful for users if teams that create pipelines in ES did an audit to check whether some would benefit from a more detailed description.

But I think this issue is sufficiently addressed for now. Let me know what you think @jethr0null .

Closing for now.