Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Low async throughput under heavy load when appender is configured to *not* drop events #619

Closed
brenuart opened this issue Aug 23, 2021 · 1 comment · Fixed by #620
Closed
Milestone

Comments

@brenuart
Copy link
Collaborator

The Async appenders show very bad performance when configured to not drop events when the ring buffer is full (appendTimeout=-1).
The async appender first try publish the event in the ring buffer then sleeps for appendRetryFrequency milliseconds before retrying. This relatively long sleep affects performance dramatically.

An alternative is to use an exponential backoff strategy capped by the appendRetryFrequency. This should give better performance while still limiting the CPU consumption.

@brenuart brenuart changed the title Bad Async performance under heavy load when appender configured to *not* drop events Bad Async performance under heavy load when appender is configured to *not* drop events Aug 23, 2021
@brenuart
Copy link
Collaborator Author

Another problem arise in CPU constraint environments when more threads than the available cores are retrying at the same time. The first retries happen at high rate (a few nanoseconds pauses) then slow down up to the configured appendRetryTimeout. This behaviour increases throughput at the expense of additional CPU cycles. We should limit the retries to a single thread at once to avoid overwhelming the system when only a few cores are available.

@brenuart brenuart changed the title Bad Async performance under heavy load when appender is configured to *not* drop events Low async throughput under heavy load when appender is configured to *not* drop events Aug 24, 2021
@philsttr philsttr added this to the 7.0 milestone Nov 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants