Move all modules to using filestream

[rdner]

Describe the enhancement:

We currently have a lot of modules that are still using the log input type (deprecated since 7.16.0) with unstable file identification that, due to file rotation, leads to parsing errors (e.g. multiline grok or JSON parser), data duplication and even possible data loss.

• OSS Filebeat – 25 files
• X-Pack Filebeat – 97 files

Describe a specific use case for the enhancement or feature:

We should migrate all of the above mentioned modules to using type: filestream + fingerprint file identity.

For example, this Elasticsearch GC module would change from:

type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
exclude_lines: ["^(OpenJDK|Java HotSpot).* Server VM ", "^CommandLine flags: ", "^Memory: ", "^{"] # exclude JVM8 banner and JSON
multiline:
  pattern: '^(\[?[0-9]{4}-[0-9]{2}-[0-9]{2}|{)'
  negate: true
  match: after
processors:
  - add_fields:
      target: ''
      fields:
        ecs.version: 1.12.0

to

type: filestream
id: <some-globally-unique-id>
take_over: true
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
exclude_lines: ["^(OpenJDK|Java HotSpot).* Server VM ", "^CommandLine flags: ", "^Memory: ", "^{"] # exclude JVM8 banner and JSON
multiline:
  pattern: '^(\[?[0-9]{4}-[0-9]{2}-[0-9]{2}|{)'
  negate: true
  match: after
processors:
  - add_fields:
      target: ''
      fields:
        ecs.version: 1.12.0

1c1,8
< type: log
---
> type: filestream
> id: <some-globally-unique-id>
> take_over: true

The main challenge here is to introduce the ability to set a unique ID to each module. We can use a default value unique to each module but if we allow users to run multiple modules of the same type, their filestream inputs must have unique identifiers.

Useful links

• take_over https://www.elastic.co/guide/en/beats/filebeat/8.16/filebeat-input-filestream.html#filebeat-input-filestream-take-over
• prospector.scanner.fingerprint https://www.elastic.co/guide/en/beats/filebeat/8.16/filebeat-input-filestream.html#filebeat-input-filestream-scan-fingerprint
• file_identity https://www.elastic.co/guide/en/beats/filebeat/8.16/filebeat-input-filestream.html#_file_identity_2

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[rdner]

After #40197 is done, we don't need to put the fingerprint configuration in this task anymore, it's going to be a new default.

[jlind23]

Assigning this to you @flexitrev as per my conversation with @rdner you'll be chasing down the modules owner and ask them to migrate.

[flexitrev]

We should have something to check against options used, to see if they require renaming -
https://www.elastic.co/guide/en/beats/filebeat/current/_step_3_use_new_option_names.html

[nimarezainia]

@flexitrev i wanted to link this older issue here: elastic/integrations#2518
The instructions on how to migrate from logfile to filestream are there i that issue. It was also the basis of what is in the docs.

I think the ask (if I' not mistaken) is to have all the integration owners to modify/migrate their packages to use filestream.

[flexitrev]

@rdner what will be the impact if the modules are not migrated?

[rdner]

@flexitrev

The main impact is that we cannot delete the code of the log input that we deprecated in 7.16.

Additionally, filestream is way more reliable when it comes to log rotations and file identification.