-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
filebeat multiple large files high cpu 300% #2786
Comments
@githubnovee On startup the high CPU usage is kind of expected as filebeat tries to send the events as fast as possible. Is the load going down again after filebeat sent all the data and only has to send new events? |
The logs indicate it's processing around 9.5k to 10k per second. That's not too bad, but for 300% CPU I'd have expected more, considering that there's no multiline and no filtering involved. @githubnovee Are you looking to reduce the CPU usage (accepting slower speed)? You could cap it to 100% by setting |
@githubnovee As a first basic test, it would be interesting to make a test on the same data with the file output instead of the kafka one, to see if the bottleneck is in Filebeat reading the files or in sending them to Kafka. Thanks. |
@tsg I adjusted CPUs by seetting max_procs :1 , CPU load looks normal, but the amount of the log is small. logs as following: |
So you get 3-4K per second, which confirms in a way that Filebeat is CPU bound. Can you try with the file output instead of the Kafka output, please? It will help us get a first feeling of where the bottleneck is. If you can, we'd also appreciate some profiling information that you can get like this:
This will create a profiling.png dot image, which you can attach to this ticket. Let me know if you hit any issues with the steps above. There are also other options for profiling. |
Closing this as no feedback was received in the last 12 month and lots of improvements have been made since then on filebeat. @githubnovee Feel free to reopen if this is still an issue. |
What's happen about filebeat version 5.0.0-rc1 (amd64), libbeat 5.0.0-rc1? what's problem in my config。 log ,read files as following:
filebeat.yml:
log:
read files:
The text was updated successfully, but these errors were encountered: