Allowing users to override pipeline ID in fileset input config

[ycombinator]

What does this PR do?

This PR allows users to override the ingest pipeline for a fileset with their custom one. To do this, users must specify the pipeline configuration option (common to all Filebeat inputs) in their fileset configuration's input section. For example:

- module: nginx
  # Access logs
  access:
    enabled: true
    input:
      pipeline: my-custom-pipeline

Why is it important?

Allows users to perform custom processing instead of the default processing provided by the ingest pipelines that come out-of-the-box with module filesets.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

Related issues

• Closes Module should allow to override the pipeline id #9531.

Use cases

See https://discuss.elastic.co/t/filebeat-cannot-change-pipeline-in-nginx-module/219360.

[ycombinator]

jenkins, test this

[sayden]

nice!

[sayden]

LGTM

[kaiyan-sheng]

LGTM!

[kaiyan-sheng]

maybe a changelog?

[ycombinator]

CI failures are unrelated and also seen on master. Merging.

[eedugon]

@ycombinator : do we have this possibility explained and showed in any public doc? I haven't been able to find it.

[ycombinator]

@eedugon We don't have a document specifically showing this option because it's just another input setting that can be overridden. We have this document that shows how to override input settings in general for any module: https://www.elastic.co/guide/en/beats/filebeat/master/advanced-settings.html#advanced-settings. And if you look at the settings under any input, pipeline is one of the settings common to all inputs: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#_pipeline_7.