Add Ingest pipeline loading to setup

[kvch]

1. Update pipeline config option

The yet unreleased flag --update-pipelines is removed. Instead of this a new config option is introduced which does the same thing. Its name is filebeat.overwrite_pipelines. By default it is set to false.

# By default Ingest pipelines are not updated. If this option is enabled Filebeat updates
# pipelines everytime a new Elasticsearch connection is established.
filebeat.update_pipelines: false

2. Load pipelines using setup

Furthermore, it's possible to load Ingest pipelines using the command setup. Please note, only configured pipelines are loaded. To make sure users don't lose events pipeline loading before sending messages is kept.

Closes #6808

[houndci-bot]

comment on exported type UpdatePipelinesCallback should be of the form "UpdatePipelinesCallback ..." (with optional leading article)

[houndci-bot]

comment on exported type PipelineLoaderFactory should be of the form "PipelineLoaderFactory ..." (with optional leading article)

[houndci-bot]

should omit 2nd value from range; this loop is equivalent to for name := range ...

[ruflin]

• Pipelines are not only loaded on the first start but also reconnect it seems. I like that as it's the same behaviour as templates for example. Also by default we don't seem to overwrite it. So if a pipeline with the same id already exists, nothing happens.
• Should we call it overwrite instead of update or forceUpdate because that is what it does?

[tsg]

Maybe clarify to something like: "By default Ingest pipelines are not updated if a pipeline with the same ID already exists."

[kvch]

@ruflin You got me. Overwrite does feel a bit more descriptive than update does. I am renaming it.

[kvch]

Renamed and edited the PR comment.

[kvch]

The failing tests are unrelated.

[urso]

We already have a setup namespace in our configs for templates and dashboards. How about moving the pipeline update setting into this namespace as well?

[ruflin]

@urso I was thinking the same at first again but then was not sure as so far it is only a filebeat feature. I expect us to have it as a global feature soon as other beats will also potentially have pipelines. I'm ok with merging as is as the config option will not potentially conflict with other option, should be fairly easy to migrate to something else and all the code is currently in Filebeat and not libbeat.

[kvch]

@urso As I understand the role of Ingest pipelines in Beats, they are tied to Filebeat. From architectural point of view I don't think it's a good idea to move the config option to instance.Beat, because it is a Filebeat specific config option.
Also, setup namespace could imply that it has something to do with the command setup. But it's not true. When users call setup this option is doesn't change anything, pipelines are overwritten. This flag only changes the behaviour of Filebeat when run is called.

On the other hand I see that from users' point of view it might be more clear to put it under setup namespace.

[urso]

@ruflin changing a config options name is never an easy change, as it's a BC breaking. As you noticed, I wonder if we will ever have ingest node pipelines for use by other beats. Besides Ingest Node pipelines, I wonder if we will ever have Logstash filter configs in beats to be configured by any beat.

But I'm fine with merging the PR as is.