Formatting JSON logs with logstash

[ranarula]

Using LogBook's JsonHttpLogFormatter in conjunction with composite JSON layout of logstash-logback-encoder double escapes the log, like so

{"@timestamp":"2018-10-04T20:39:49.925-07:00","@version":"1","message":"{\"origin\":\"remote\",\"type\":\"request\", \"correlation\":\"85eff6cd9880ca28\",\"protocol\":\"HTTP/1.1\",\"remote\":\"0:0:0:0:0:0:0:1\",\"method\":\"POST\",\"uri\":\"http://localhost:8080/dummy/log\",\"headers\":{\"Accept\":[\"*/*\"], \"accept-encoding\":[\"gzip, deflate\"]],\"content-length\":[\"18\"],\"Content-Type\":[\"application/json\"],\"Host\":[\"localhost:8080\"]},\"body\":{\"message\" :\"hey\"}}","logger_name":"com.sample.logging", "thread_name":"XNIO-3 task-1","level":"DEBUG","level_value":10000,"x-request-id":"b2441153-86c2-4f62-904e-8debf99a2c66"}

I am looking for suggestions to address this

One possible way I am experimenting is the use of #tryJson{...} in my appender pattern (as documented here),
like so

{"@timestamp":"2018-10-04T14:33:43.791-07:00","@version":"1","logger_name":"com.sample.logging","jsonMessage":{"origin":"remote","type":"request","correlation":"e6f56834dd25766d","protocol":"HTTP/1.1","remote":"0:0:0:0:0:0:0:1","method":"POST","uri":"http://localhost:8080/dummy/log","headers":{"Accept":["*/*"],"accept-encoding":["gzip, deflate"],"Connection":["keep-alive"],"content-length":["18"],"Content-Type":["application/json"],"Host":["localhost:8080"]},"body":{"message":"hey"}}}

Wondering if there is any other way to accomplish this
say by implementing a CustomHttpLogFormatter (which I initially thought but couldn't make progress) or any other way

[whiskeysierra]

You can't solve this within logbook since your appender is doing the post processing. There is nothing you can do within a custom formatter that would help. Your suggestion to use tryJson sounds like the best approach.

[skjolber]

@whiskeysierra @ranarula we're using a custom HttpLogWriter which logs using a Marker. The Marker extends RawJsonAppendingMarker (and autodetects pretty-printing). Essentially we're logging the request-response under a root json property 'http'.

{
.. builtin stuff
http: { logbook }
}

It also formattes a message for request outline, i.e. "GET /a/b/c" which is visiable by default in our log accumulation tool (stackdriver).

@whiskeysierra do you think a Logback-Logstash-module would be possible?

[whiskeysierra]

@skjolber I'm open for contributions. I'm not familiar with logstash. Can you outline what you would like to put into that module?

[ranarula]

@skjolber - I was able to overcome the issue by annotating the logbook content with @JsonRawValue in our CustomJsonFormatter

[skjolber]

Basically the module will be quite small, just wrapping the request/response output as a Marker (or StructuredArgument) in a HttpLogWriter so it comes out correctly with the default library settings.

[whiskeysierra]

Would you care to open a PR introducing such a module?

[skjolber]

Sure, I'll try to make it this week.

[AWinterman]

@skjolber - I was able to overcome the issue by annotating the logbook content with @JsonRawValue in our CustomJsonFormatter

Can you provide an example of this?

[skjolber]

@AWinterman there is JSON support now, so that approach is outdated.

[AWinterman]

Ah gotcha. I was looking for the log stash confit to make the JSON not double serialized

[arjunchhabra-invitae]

@AWinterman - did you happen to get this resolved? We are seeing the same issue.

[simasch]

We have the same issue.
Is there a solution?

@AWinterman @arjunchhabra-invitae

[arjunchhabra-invitae]

@simasch

Here is what we did:

    <!--  Define a custom JSON logger  -->
    <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <jsonFactoryDecorator class="net.logstash.logback.decorate.CharacterEscapesJsonFactoryDecorator">
                <includeStandardAsciiEscapesForJSON>false</includeStandardAsciiEscapesForJSON>
            </jsonFactoryDecorator>
        </encoder>
    </appender>

[simasch]

@arjunchhabra-invitae Thanks!