Filebeat suggestion - allowing for something like multiline.flush_pattern?

[TheoAndersen]

I'm using multiline for merging my log-lines together to form application-centric events, which can span multiple events.

I feel like it could be a useful addition, to be able to specify a specific pattern, for when a multiline should stop and flush the event. Right now it seems to be only when either; a new multiline starts, the max-lines count is reached or the timeout is reached.

This would make it possible to flush an event right when it ends - whereas right now the last event i receive, will first be flushed when either one the timeout/max-length is reached, or when a new one begins.

multiline
- pattern: "start-line-regex"
- negate: true
- match: after
- flush-pattern: "end-line-regex"

What do you think?

Now i having looked at the codebase (not that familiar with go, but i might give it a try if you suggest me to), but I'm thinking that, being that theres already the other kind of flush-guards in place, it would seem conceptually like a small task to add this?

[ruflin]

We discussed in the past about a "start/end" pattern but went in the first implementation only for one pattern as it was much simpler to do. I definitively think there is value in having something similar as you described above. Here you can also find the initial discussion about multiline and what options we discussed: https://github.com/elastic/filebeat/issues/301

It reminds me also of #3852 Not sure if the two feature requests could be combined into one, means it is not necessarly only mutiline specific.

[TheoAndersen]

The first link you posted is from a private issue, so i can't access it.

Regarding the #3852 issue about one-line logs, I'm not sure that this flush-pattern option would solve this - dosen't the initial line-matching logic happen outside the multiline module?

I've managed to add the flush-pattern code pretty simply into the current multiline code, and will create a PR soon - Just need to test it a bit more, only TDDd it so far.