[Filebeat] Support multiline options in aws s3 input

[andrewkroh]

Describe the enhancement:

When reading log files from S3 users should be able to specify the same multiline options that are available with the log input.

Describe a specific use case for the enhancement or feature:

Reading XML based Windows event logs from S3 that are newline delimited, but the XML itself contains strings with newlines. So in order to get one full XML object we need the multiline reader options.

Ideally config like this would work:

- type: aws-s3
  queue_url: https://sqs.us-east-1.amazonaws.com/foo/queue
  credential_profile_name: beats
  multiline.pattern: ^\<Event\>
  multiline.negate: true
  multiline.match: after

[exekias]

@kvch @urso I saw #24763 🚀 , I wonder if this should be the way to go for all inputs wanting to support parsing? Your guidance here would be good

[urso]

@kvch @urso I saw #24763 🚀 , I wonder if this should be the way to go for all inputs wanting to support parsing? Your guidance here would be good

Yes, we are still improving this on the filestream input. The filestream input is supposed to supersede the current logs input, as it takes on many issues of the logs input (integrations will need to be updated), especially in conjunction with k8s autodiscovery.

The parsing in the logs input was not as 'clean', so we still have to jump through some hoops to make it work properly in a generic fashion. Unfortunately we can't use these parsers via processors, for differrent reasons (e.g. processors can't be stateful), but we're thinking to move the syslog parsing from the syslog input into a parser as well, and provide the parser settings to multiple inputs. This will allow users to mix different levels of multiline, json, syslog at will. You have multiline logs embedded in syslog, embedded in a json log file... no problem :P

[exekias]

That's great to hear. For this case, this input is not based on a file so we would need to update the code to use these parsers. Does that make sense?

[kvch]

I have updated this issue with the current state: #16137

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[leehinman]

Need to re-work to make use of parsers See https://github.com/elastic/beats/tree/master/filebeat/input/filestream for example. This is just for the non-JSON part of awss3 input.

[dnz-wseabrook]

Just noting this is something we're looking for too

[andrewkroh]

This feature has been implemented for 7.14.0. There's a refactoring PR open that could change the config format.

• PR: Add multiline support to awss3 input #25710
• Refactoring to use parsers config format: [Filebeat] change multiline configuration in awss3 input to parsers #25873
• Tentative Docs for 7.14: https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-input-aws-s3.html#input-aws-s3-multiline