[Filebeat] aws-s3 input - incorporate Last-Modified time into _id

[andrewkroh]

Describe the enhancement:

The AWS S3 input computes a document _id based on the bucket, object key, and data offset. However, if an object is mutated and subsequently reread, those _id values may not be unique. Incorporating the “Last-Modified” time (https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#SysMetadata) into the value should ensure uniqueness. Since the “Last-Modified” time is not user-controllable, it is assumed to be unique each time an object is mutated.

Describe a specific use case for the enhancement or feature:

Better support use-cases where S3 objects are treated as mutable.

Workaround option

Add a processor the clears the _id field.

processors:
  # Clear generated document IDs because S3 objects are updated, and the IDs might be
  # the same for updated objects.
  - drop_fields:
      fields:
        - '@metadata._id'
      ignore_missing: true

[botelastic]

This issue doesn't have a Team:<team> label.