[Filebeat][Apache module] Allow pipeline to be overridden in module config

[nkakouros]

Describe the enhancement:

In the apache module config file, it says:

  ....
    # Input configuration (advanced). Any input configuration option
    # can be added under this section.
    #input:
....

This means that I can add log input options (since the apache module uses that input) to further configure the module.

Indeed, if I configure the apache module as:

- module: apache
  error:
    enabled: false
  access:
    enabled: true
    var:
      paths:
        - /var/log/apache2/cuiteur.access.log
    input:
      fields:
        QQQ: 111
      # pipeline: my-pipe

I can see the new field in Kibana.

However, I cannot change the pipeline by uncommenting the last line.

Describe a specific use case for the enhancement or feature:
The Apache module comes with support for specific access log formats. If sth changes slightly, the module fails. For instance, changing the timestamp to unix millisecond time breaks the module:

LogFormat "%v %h %l %u %{msec}t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" my

It would be great to be able to override the shipped pipeline. This way, one could copy the pipeline, fix the patterns and be done. The fields and anything else would be the same.

As it stands now, one has to either:

• manually copy/insert the fields spec, create a new input, load kibana dashboards, etc, or
• enable the module, use filebeat setup to do all the above, disable the module, create a new input

With support for pipeline as in the snippet above, one would just specify a different pipeline, insert it into elasticsearch and ... done.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[kvch]

Hi, have you uploaded the pipeline you would like to use manually before setting the option pipeline?

[ycombinator]

Hi @nkakouros, I believe the fix you are looking for is this one: #16561. It will be release in v7.7.0.