Add ability to ignore a missing pipeline to the pipeline processor

[joshdover]

Description

I’m trying to create a pipeline processor that will avoid throwing errors if the pipeline it’s referencing doesn’t exist, but will still throw errors if the pipeline does exist but fails. It seems there are two potential approaches:

1. Add a on_failure block that can handle this case

My initial idea was to use an on_failure block with a single fail processor that has a condition to not fail if the failure was due to non-existent pipeline error. However, it seems I can’t figure how to access the _ingest.on_failure_* fields from within the if condition, but they are accessible from within the message option. I tried accessing this via ctx['_ingest'] as well without any luck. It seems we need code changes to expose this to the if painless context. Here’s the pipeline I’m trying to create:

PUT _ingest/pipeline/test-pipeline
{
  "processors": [
    {
      "pipeline": {
        "name": "test-custom",
        "tag": "fleet-custom-caller",
        "on_failure": [
          {
            "fail": {
              "if": "_ingest.on_failure_processor_type != 'pipeline' || _ingest.on_failure_processor_tag != 'fleet-custom-caller' || !_ingest.on_failure_message.contains('non-existent pipeline [test-custom]')",
              "message": "test-custom pipeline failed with {{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    }
  ]
}

2. Add a dedicated ignore_missing option to the pipeline processor

As @felixbarny has discussed in elastic/package-spec#129 (comment), it's probably preferable to have a dedicated processor option for this in order to:

1. Avoid creating an exception at all and the overhead involved
2. Have a less fragile solution that is not dependent on an error message string

PUT _ingest/pipeline/test-pipeline
{
  "processors": [
    {
      "pipeline": {
        "name": "test-custom",
        "ignore_missing": true
      }
    }
  ]
}

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[felixbarny]

Closing as implemented via #87354