Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change config defaults #568

Merged
merged 2 commits into from
Dec 22, 2015
Merged

Change config defaults #568

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5e74851
Change config defaults
dedemorton Dec 18, 2015
bb3bb89
Fix incorrect change to default for bulk_max_size
dedemorton Dec 22, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions libbeat/docs/outputconfig.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ operation doesn't succeed after
===== bulk_max_size

The maximum number of events to bulk in a single Elasticsearch bulk API index request.
The default is 50.
The default is 200.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, there's a confusion here. We've changed bulk_max_size' default for the Logstash output, where this setting is not documented at all (no idea why). Might be worth for me or @urso (hint) to come up with a longer explanation since this is critical for performance.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops. Monica mentioned that you changed the default, and since the setting only appears in one place, I assumed I was changing the correct setting. :-) I'll revert this value to 50 and wait for you to submit a description.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we've got multiple output plugins:

  • Elasticsearch
  • logstash
  • console
  • file

The default bulk_max_size is used for all output plugins but elasticsearch, which sets the default to 50.

This options sets maximum number of events that can be combined internally into batches and will be publishable by the output plugins =>

  • if beat tries to send single events, the events are collected into batches

  • if beat tries to publish large batch of event (bigger bulk_max_size), the batch will be split.

    Bigger batch sizes can improve performance due to ammortizing per event sending overhead. On the other hand to big batch sizes can increase processing time such that queues in logstash/elasticsearch can not be processed -> APIs return errors, connections get killed or publish requests time out. This increases latency and lowers throughput for indexing events.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@urso I've created a new doc issue #586 to track the doc changes for bulk_max_size so these comments don't get lost under the outdated diff....


===== timeout

Expand Down Expand Up @@ -506,7 +506,7 @@ generated by default for {beatname_uc} would be "{beatname_lc}", "{beatname_lc}.
===== rotate_every_kb

The maximum size in kilobytes of each file. When this size is reached, the files are
rotated. The default value is 1000 KB.
rotated. The default value is 10240 KB.

===== number_of_files

Expand Down