Compatibility with / Benchmarking with Asynchronous logging

[yan-hic]

How does the appender integrate with Asynchronous logging as described at https://logging.apache.org/log4j/2.x/manual/async.html#AllAsync

According to log4j doc, there is a huge performance gain but not sure if it is leveraged by log4j-s3-search.

If not,

• is there any benefit for integrating with each other ? I assume combining both as-is would lead to buffering twice.
• is there any plan using the core Async mechanism ?

[bluedenim]

is there any benefit for integrating with each other? I assume combining both as-is would lead to buffering twice.

There may be some benefit in lower latency when logging. The current impl definitely did not optimize for that. There will definitely be multiple buffering involved. Currently there is already a buffer that is flushed either by size or by time. I suspect the async logging layer probably has its own buffering layer. I am pretty confident that the async layer WILL have a lower latency than what I have now.

is there any plan using the core Async mechanism?

Maybe. I'll need to learn more about that new async layer and how flexible and configurable it is. If there is a need to improve on the latency of various logging (e.g. info(), warning(), error(), etc.) calls, then that's a good motivation for it.

[bluedenim]

I just got some time to look into this over the holidays.

• Async Appender -- this framework duplicates parts of log4j-s3-appender (or perhaps more correctly put: log4j-s3-appender duplicates part of Log4j) as far as buffer and async publishing. However, the buffering from Log4j is strictly size-based (bufferSize) while log4j-s3-appender supports size- (stagingBufferSize) and time-based (stagingBufferAge) flush policies. I'm not certain how many users actually use the time-based policy. Regardless, it will be some non-trivial work to retool log4j-s3-appender to fit the Async appender.
• Async Logger -- This happens at the logger level and will, I believe, cache/collect events until a certain point (queue size?) when it flushes a bulk batch of events to the appender to publish. In this case, I don't think the appender should do any buffering at all and should be a thin wrapper around a "manager" that publishes to S3, etc. (to use their lingo). Therefore, if we are to integrate with this, we'd probably need to create a new set of light-weight appenders (as opposed to modifying the existing ones) that have no buffering and focuses mostly on publishing the batch to the external storage when called. There are reusable code that can be refactored out, to be sure.

In summary:

• There is not a lot of benefit from integrating with the Async appender (with the exception of perhaps to delegate a lot of buffering and flushing logic to Log4j instead of writing our own). And we lose the ability to flush/publish based on time/age.
• It is possible to support async loggers by adding a set of new appenders specifically designed to work with them and refactoring a bit as necessary so that the existing appender (i.e. Log4j2Appender) and these new appenders can share the publishing code. I believe that is mostly done since the publishing logic is in LoggingEventCache and BufferPublisher and their delegates. The benefit here is, again, delegating the buffering to Log4j instead of our own. Again, though, we will be losing the time-based flush policy.

It'll be a good exercise and perhaps make this library more compliant w/ the Log4j framework. The biggest downside is the loss of time-based publish policy.