handle multiline from nested JSON strings

[edsiper]

there is a specific use case where an application running under Docker and generating multiline log messages ends up with logs as follows:

{"log":"2017-07-26 07:54:42.130  WARN parse organization id exception\n","stream":"stdout","time":"2017-07-25T07:54:42.131621351Z"}
{"log":"\n","stream":"stdout","time":"2017-07-25T07:54:42.13171348Z"}
{"log":"java.lang.NumberFormatException: For input string: \"PodQueryBykind\"\n","stream":"stdout","time":"2017-07-25T07:54:42.131723859Z"}

there are 3 JSON log entries, but the contained messages are multiline. We likely need to implement a specific feature in our parsers to reduce the pain.

[djotanov]

Any news on when could this be implemented?

[monotek]

You could also try to use the detect-exceptions plugin mentioned in #476

https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions

Edit: ahh... damn... the fluentd plugins don't seem compatible :-(

[gavrie]

It seems this issue is not being addressed. This is a showstopper for me, so I guess it's back to plain old fluentd...

[abhishek-buragadda]

@gavrie Can we handle the same issue in fluentd?

[gavrie]

fluentd already has an existing Loggly plugin that handles this correctly.

…

On Mon, 25 Jun 2018 at 8:39 abhishek ***@***.***> wrote: @gavrie <https://github.com/gavrie> Can we handle the same issue in fluentd? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#337 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARnjPQlVN21qTk_Sie8FWFIPv5gnBelks5uAHejgaJpZM4OjVIY> .

[abhishek-buragadda]

@gavrie This is a paid plugin rgt? are there any open source plugins that does the same?

[gavrie]

@abhishek the plugin is Open Source (Apache license): https://github.com/patant/fluent-plugin-loggly

…

On Wed, 27 Jun 2018 at 17:11 abhishek ***@***.***> wrote: @gavrie <https://github.com/gavrie> This <https://www.loggly.com/docs/fluentd-logs/> is a paid plugin rgt? are there any open source plugins that does the same? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#337 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARnjK3E_g9QTUyhdjDYYc5znLJN_F-uks5uA5KPgaJpZM4OjVIY> .

[abhishek-buragadda]

@gavrie thanks .

[flypenguin]

@edsiper just a small request for information - is this something which is being developed now (or soon)? if yes I could wait, if not I would have to look for alternatives. I think this use case is rather straightforward, so my (maybe naive ;) hope is that it is at least on the map.

[Markbnj]

This is an issue for us as well. I spun up a fluent-bit daemonset and was happy with the performance and footprint, but I have not been able to figure out a workaround for the issue with multiline logs. It's not difficult to create a parser for the additional lines which drops the docker cruft and captures the message content. The problem is that the named field gets added to the record for each line, which creates a json record with duplicate keys. One idea comes to mind which would solve this problem neatly: an option to allow the input plugin to append to an existing field when it encounters a new field of the same name. For now we're going back to fluentd because the detect_exceptions plugin helps with some of these cases.

[nikolay]

@Markbnj I think, for the time being, Fluent Bit is just an experiment, a POC, as it lacks rudimentary features and cannot be used in the real world.

[breeze7086]

Same issue for me.
The cause of this is default docker logging-driver of json-file.