Fix performance issue caused by invalid logging configuration #8908
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Configure the root log level in bin/pulsar startup script to use "info" level by default. This can be done by setting the "pulsar.log.root.level" system property. - since the root log level was debug, all code blocks within log.isDebugEnabled() got executed. This caused a lot of unnecessary memory allocations and wasted CPU cycles.
|
Here's an allocation flamegraph for Pulsar 2.7.0, a lot of logging calls are present:
After applying this fix, the logging calls are gone:
|
|
/pulsarbot run-failure-checks |
merlimat
approved these changes
Dec 11, 2020
sijie
approved these changes
Dec 11, 2020
|
/pulsarbot run-failure-checks |
|
This fix would be needed on branch-2.6 . There's a reported issue #8875 about excessive GC |
codelipenghui
pushed a commit
that referenced
this pull request
Dec 21, 2020
### Motivation Profiling a 3 node Pulsar cluster running with 2.7.0 version showed allocation hotspots in `log.debug` methods. This indicated that the root logger level was set to debug. ### Modifications - Configure the root log level in `bin/pulsar` startup script to use `info` level by default. This can be done by setting the `pulsar.log.root.level` system property. - since the root log level was debug, all code blocks within `log.isDebugEnabled()` got executed. This caused a lot of unnecessary memory allocations and wasted CPU cycles. (cherry picked from commit 18f539e)
This was referenced Nov 22, 2021
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Profiling a 3 node Pulsar cluster running with 2.7.0 version showed allocation hotspots in
log.debugmethods. This indicated that the root logger level was set to debug.Modifications
Configure the root log level in
bin/pulsarstartup script touse
infolevel by default. This can be done by settingthe
pulsar.log.root.levelsystem property.since the root log level was debug, all code blocks within
log.isDebugEnabled()got executed.This caused a lot of unnecessary memory allocations and wasted CPU cycles.