Update broker/pipeline setup #4650

urso · 2017-07-11T04:33:21Z

remove settings: filebeat.spooler, filebeat.publish_async, filebeat.idle_timeout, queue_size, bulk_queue_size
introduce new setting: queue.<queue_type>
expose memory queue (type = 'mem') settings: events, flush.min_events, flush.timeout

Note: docs will be updated in a separate PR.

ruflin · 2017-07-11T06:50:21Z

metricbeat/metricbeat.reference.yml

-# Do not modify this value.
-#bulk_queue_size: 0
+# Internal queue configuration for buffering events to be published.
+#queue:


It's a bit strange that we call it here queue and in the code broker. This will need some docs and changelog entry. But can happen in a follow up PR. Does this have to be in this PR?

These are the user-visible settings. I decided to rename the broker package to queue, but first I want to get the changes to the settings in and documented, so we won't introduce BC-breaking changes in case package updates PR gets merged late.

ruflin · 2017-07-11T19:32:33Z

auditbeat/auditbeat.reference.yml

+    # Hints the minimum number of events stored in the queue,
+    # before providing a batch of events to the outputs.
+    # A value of 0 (the default) ensures events are immediately available
+    # to be send to the outputs.


s/send/sent/

"to be sent" is correct

ruflin · 2017-07-11T19:33:28Z

auditbeat/auditbeat.reference.yml

+  # Queue type by name (default 'mem')
+  #mem:
+    # Max number of events to store in the queue.
+    #events: 4096


What happens when this is reached? Flush happens? Thinking if there should be flush.min, flush.max and flush.timeout or similar?

Semantics is a little different from old spooler here. The queue/broker does not flush its buffers. By default it basically stream the events right through to the outputs (if output takes somewhat longer it will be presented with all events available - up to bulk_max_size).

By configuring min_events, the queue will block the output until:

at least min events is reached

timeout is trigger

The min events is more of a hint, as the output might still receive a smaller buffer.

The events setting is the maximum number of events the queue can hold. It's independent of min_events and flush timer. Only min_events must be < events.

Naming is a little weird here... as it's more like 'block-processing' settings...

@urso what happens when events is reached? I assume it will block from adding more events? In case of metricbeat assuming sending events is slower then generating events but still all events are acked. What will happen?

once events is reached, the queue will block (no more space). Upon ACK of N events, another N events can be stored in the queue (N producers will be unblocked).

ruflin · 2017-07-11T19:35:08Z

libbeat/publisher/bc/publisher/pipeline.go

 	}

-	p, err := pipeline.New(brokerFactory, out, settings)
+	brokerFactory := broker.FindFactory(brokerType)


As quickly discussed, later we should probably rename broker package to queue to make the relation to the config more obvious.

ruflin · 2017-07-11T19:39:21Z

libbeat/publisher/broker/membroker/broker.go

-func NewBroker(eventer broker.Eventer, sz int, waitOnClose bool) *Broker {
+func NewBroker(
+	eventer broker.Eventer,
+	sz int,


Seems like the 3 params could be a config struct for borkers.

andrewkroh · 2017-07-11T20:09:47Z

winlogbeat/winlogbeat.reference.yml

-# Do not modify this value.
-#bulk_queue_size: 0
+# Internal queue configuration for buffering events to be published.
+#queue:


I'll open a PR to delete this block of code from Winlogbeat so that queue isn't rejected.

hehe, thanks.

this PR is adjusting winlogbeat.

ruflin · 2017-07-12T05:49:16Z

CHANGELOG.asciidoc

@@ -19,6 +19,7 @@ https://github.com/elastic/beats/compare/v6.0.0-alpha2...master[Check the HEAD d
 - The `scripts/import_dashboards` is removed from packages. Use the `setup` command instead. {pull}4586[4586]
 - Change format of the saved kibana dashboards to have a single JSON file for each dashboard {pull}4413[4413]
 - Rename `configtest` command to `test config`. {pull}4590[4590]
+- Remove setting `queue_size` and `bulk_queue_size`. {pull}4650[4650]


Should we add a message to the 5.6 release that these values will change?

ruflin · 2017-07-12T05:49:48Z

CHANGELOG.asciidoc

@@ -81,6 +83,7 @@ https://github.com/elastic/beats/compare/v6.0.0-alpha2...master[Check the HEAD d
 - Add udp prospector type. {pull}4452[4452]
 - Enabled Cgo which means libc is dynamically compiled. {pull}4546[4546]
 - Add Beta module config reloading mechanism {pull}4566[4566]
+- Remove spooler and publisher components and settings. {pull}4644[4644]


Same here, we should probably have a message in 5.6 or even better, not let the beat start in 6.0 if these values are set?

ruflin · 2017-07-12T05:51:00Z

auditbeat/auditbeat.reference.yml

  # new batches.
-  #pipelining: 0
+  #pipelining: 5


Is there a reason we changed the default to 5? Should probably be mentioned in the changelog?

pipeline can operate complete async -> take advantage of this fact

with default settings we have a high chance to publish very small batches (no flush timeout -> reduced latency) if producers don't have a high enough rate. In Logstash case we can easily compensate for this by pushing more batches on the link

Updated changelog.

ruflin · 2017-07-12T05:51:40Z

filebeat/config/config.go

@@ -20,9 +20,6 @@ const (

 type Config struct {
 	Prospectors      []*common.Config `config:"prospectors"`
-	SpoolSize        uint64           `config:"spool_size" validate:"min=1"`
-	PublishAsync     bool             `config:"publish_async"`


Glad to see this one removed ;-)

ruflin · 2017-07-12T05:53:26Z

libbeat/publisher/broker/membroker/config.go

 type config struct {
-	Events int `config:"events" validate:"min=32"`
+	Events         int           `config:"events" validate:"min=32"`


Kind of surprised to see a minimum here.

well, we need to buffer some data for pushing to the outputs. Without queue the pipeline won't be able to operate -> enforce minimum so users don't use 1 or 0.

- remove setting queue_size - remove unused setting bulk_queue_size - introduce settings namespace `queue` with default `mem.events: 4096` - add settings queue.mem.flush.min_events and queue.mem.flush.timeout - change default output.logstash.pipelining to 5 - remove spooler settings from filebeat config object - Update winlogbeat config validate check

urso · 2017-07-12T13:46:41Z

Translating old configurations:

rename queue_size -> queue.mem.events * 2
remove bulk_queue_size
rename or remove filebeat.spool_size -> queue.mem.flush.min_events
rename or remove filebeat.idle_timeout -> queue.mem.flush.timeout

dedemorton · 2018-01-22T18:45:49Z

I'm removing the needs_docs label here because I believe we've covered the doc requirements with the topic https://www.elastic.co/guide/en/beats/filebeat/current/configuring-internal-queue.html and the removed settings that are documented in the breaking changes for the release.

urso added the in progress Pull request is currently in progress. label Jul 11, 2017

ruflin reviewed Jul 11, 2017

View reviewed changes

urso force-pushed the pipeline/config branch 2 times, most recently from fe27bb3 to cc174ea Compare July 11, 2017 12:09

ruflin approved these changes Jul 11, 2017

View reviewed changes

andrewkroh approved these changes Jul 11, 2017

View reviewed changes

urso force-pushed the pipeline/config branch 3 times, most recently from 57756c6 to b3f1b82 Compare July 12, 2017 02:19

urso changed the title ~~[WIP] update broker/pipeline setup~~ Update broker/pipeline setup Jul 12, 2017

urso force-pushed the pipeline/config branch from b3f1b82 to 61f5b7e Compare July 12, 2017 03:00

urso mentioned this pull request Jul 12, 2017

Publisher Pipeline #4598

Closed

22 tasks

urso added needs_docs review and removed in progress Pull request is currently in progress. labels Jul 12, 2017

ruflin approved these changes Jul 12, 2017

View reviewed changes

urso force-pushed the pipeline/config branch from 61f5b7e to cd45c0a Compare July 12, 2017 13:29

ruflin merged commit 7a6a5dd into elastic:master Jul 12, 2017

monicasarbu added the libbeat label Jul 17, 2017

urso deleted the pipeline/config branch July 19, 2017 13:18

tsg mentioned this pull request Jul 24, 2017

Document breaking changes in 6.0 #4737

Closed

28 tasks

dedemorton mentioned this pull request Jul 25, 2017

Beats doc updates in 6.0 #4540

Closed

42 tasks

dedemorton removed the needs_docs label Jan 22, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update broker/pipeline setup #4650

Update broker/pipeline setup #4650

urso commented Jul 11, 2017 •

edited

Loading

ruflin Jul 11, 2017

urso Jul 11, 2017

ruflin Jul 11, 2017

dedemorton Jul 11, 2017

urso Jul 11, 2017

ruflin Jul 11, 2017

urso Jul 11, 2017

urso Jul 11, 2017 •

edited

Loading

urso Jul 11, 2017

ruflin Jul 11, 2017

urso Jul 11, 2017

ruflin Jul 11, 2017

ruflin Jul 11, 2017

andrewkroh Jul 11, 2017

urso Jul 11, 2017

urso Jul 12, 2017

ruflin Jul 12, 2017

ruflin Jul 12, 2017

ruflin Jul 12, 2017

urso Jul 12, 2017 •

edited

Loading

urso Jul 12, 2017

ruflin Jul 12, 2017

ruflin Jul 12, 2017

urso Jul 12, 2017

urso commented Jul 12, 2017

dedemorton commented Jan 22, 2018

Update broker/pipeline setup #4650

Update broker/pipeline setup #4650

Conversation

urso commented Jul 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

urso Jul 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

urso Jul 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

urso commented Jul 12, 2017

dedemorton commented Jan 22, 2018

urso commented Jul 11, 2017 •

edited

Loading

urso Jul 11, 2017 •

edited

Loading

urso Jul 12, 2017 •

edited

Loading