Allow pipeline processor to ignore missing pipelines

[martijnvg]

Add ignore_missing_pipeline option to pipeline processor.
This controls whether the pipeline processor should fail
with an error if no pipeline with a name specified in the
name option exists.

This enhancement is useful to setup a pipeline infrastructure
that lazily adds extension points for overwrites. So that
for specific cluster setups custom pre-processing can be added
at a later point in time.

Relates to #87323

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @martijnvg, I've created a changelog YAML for you.

[elasticmachine]

Pinging @elastic/clients-team (Team:Clients)

[joshdover]

LGTM from a Fleet perspective. I tested this with different combinations of these pipelines dev tools commands and everything worked as expected:

PUT _ingest/pipeline/test-pipeline
{
  "processors": [
    {
      "append": {
        "field": "hello",
        "value": "from test-pipeline"
      }
    },
    {
      "pipeline": {
        "name": "test-custom",
        "ignore_missing_pipeline": true
      }
    }
  ]
}

PUT _ingest/pipeline/test-custom
{
  "processors": [
    {
      "append": {
        "field": "hello",
        "value": "from test-custom"
      }
    },
    {
      "fail": {
        "message": "boom!"
      }
    }
  ]
}

DELETE _ingest/pipeline/test-custom

POST _ingest/pipeline/test-pipeline/_simulate
{
  "docs": [
    { "_index": "x", "_source": { "foo": "bar" } }  
  ]
}

[dakrone]

LGTM, thanks for adding this Martijn!

[andrewkroh]

Great idea.

[andrewkroh]

From a performance aspect the getPipeline() operation is basically a regular Map lookup, right?

This path will get executed a lot if this feature is used in all Fleet data stream pipelines. I just wanted to confirm that lookup failures are not particularly expensive 😄 (like lazy loading on lookup failure).

[martijnvg]

From a performance aspect the getPipeline() operation is basically a regular Map lookup, right?

Yes, this method performs a map lookup. I don't think this map lookup will be noticeable during indexing (many other operations occur during indexing, so I don't think this will be visible).