Elasticsearch ingest node pipeline mapping

[urso]

Elasticsearch event pipeline can currently only configured via output.elasticsearch.parameters.pipeline. Improve pipeline configuration+selection by:

• dynamically choose ingest pipeline per event
• add support for multiple pipelines
• define optional mapping from event to pipeline name
• support for conditionals to select pipeline on more complicated rules

Introduce new setting output.elasticsearch.pipelines using an array of pipeline mapping rules. The first rule matching defines the pipeline name to use. If no rule matches, no pipeline will be used for said event:

output.elasticsearch.pipelines:
- rule1
- rule2

• Rules support conditionals using the when key-word (like conditionals support in libbeat processors).
• The pipeline name is set using the pipeline-setting. It can be set by events using format strings.
• additional name mappings (based on pipeline-setting) requires an additional name mapping table on via mappings-setting. The default-setting is used, if the pipeline is unknown to the mapping table.
• A pipeline rule only consisting of pipeline-keyword matches only if format-string can be applied.

Sample configurations:

• read pipeline directly from event fields.pipeline using format string (pipelines names can be composed from multiple fields when using format strings):

output.elasticsearch.pipelines:
- pipeline: '%{[fields.pipeline]}'

• use fields.pipeline, but fall back to default-pipeline if field is missing:

output.elasticsearch.pipelines:
- pipeline: '%{[fields.pipeline]}'
- pipeline: 'default-pipeline'

• map event field to pipeline name with default pipeline default-pipeline, if no mapping is defined or field is missing:

output.elasticsearch.pipelines:
- pipeline: '%{[fields.pipeline]}''
  default: ''default-pipeline'
  mappings:
    'abc': 'pipeline-abc'
    'def': 'pipeline-def'

• use conditionals to select pipeline based on HTTP code:

output.elasticsearch.pipelines:
- pipeline: 'ok-pipeline'
  when.range:
    http.code: [200, 299]
- pipeline: 'verybad-pipeline'
  when.range:
    http.code: [500, 999]
- pipeline: 'default-pipeline'

• use filters + mapping for more complicated scenario:

output.elasticsearch.pipelines:
- pipeline: '%{[fields.pipeline]}'
  default: default-pipeline
  mappings:
    abc: pipeline-abc
  when.has_fields: ['event_data.IPAddress']
- pipeline: 'another-pipeline'
  when.regexp:
    source: ...
- pipeline: 'default-pipeline'

[tsg]

Seems quite flexible, I cannot think of a use case not covered :-). I guess a first version could be to just cover the first two sample configs (set pipeline using format strings + a default value in case of errors). That should cover already most use cases, right?

[ruflin]

That means to select a pipeline for a prospector I could use:

filebeat.prospectors:
- input_type: log
  fields:
    pipeline: abc

One additional thing I would like to have is that I can use:

filebeat.prospectors:
- input_type: log
  pipelines: [abc]

This would be possible if we make pipelines (or only the second one) part of meta. The advantage is that this make pipeline configuration "local" in the configuration. The disadvantage is that this is somehow ES specific but the outputs which don't need it, can just ignore it.

If both options are used in a config file, it would "merge" the defined pipelines similar to what it does with fields or tags.

[urso]

original proposal has been implemented and merged

[andrewkroh]

Implemented in #2109