[Filebeat] Duplicating events in log rotation when output is down

[gemaliano]

• Version: 7.5.0

• Description:
  When filebeat output is not running for long period of time and logs are rotating during this time we can see duplicates.
  All the log files (my-server.log.1,my-server.log.2...) are read again.

• Configuration:
  Filebeat input: reading from rotation log

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/my-server/my-server.log*

Filebeat output for example logstash.

• Filebeat process:

1. collect current file stats from disk
2. (State update phase) for each file found:
  2.1 check if file is new and start prepare harvester:
    2.1.1 create harvester
    2.1.2 add state to internal table
    2.1.3 forward new state to the registry
  2.2 if file is known and has been renamed:
    2.2.1 update internal table
    2.2.2 forward updates state to the registry
3. (Cleanup phase) for each file in internal state table:
  3.1 checkf if current file path still matches inode pair
    3.1.1 if file is still open, do nothing
    3.1.2 if they do not match anymore remove it:
      3.1.2.1 remove from internal state table
      3.1.2.2 forward 'removal' to the registry

Step (2.1.3, 2.2.2, and 2.3.2) will block until the output is up again.
When output is down and the logs are rotated, the input is blocked in the "State update phase" (e.g. 2.1.3 or 2.2.2). Once output is up again, the input continues with the 'Cleanup phase', which detects that the current on disk state does not match the internal state anymore. States get removed.

Normally the cleanup phase is expected to run right after the state collection phase. But due to the input being blocked, the input did continue the state cleanup with some very old outdated state.

• Workaround:
  If we have to stop output for long period of time and we want to avoid duplicates:
  https://www.elastic.co/blog/logstash-lessons-handling-duplicates
  https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch

• Related github:
  Filebeat input v2 API #15324

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[andresrc]

This needs changes #15324 to be solved.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.