Add file spool to queue docs #6902

urso · 2018-04-19T19:26:11Z

No description provided.

dedemorton

Added some feedback plus a few edits. I'd like to do a more thorough edit of this content later (rather than cluttering up the review with a bunch of small edits).

dedemorton · 2018-04-19T23:58:50Z

libbeat/docs/queueconfig.asciidoc

+[[configuration-internal-queue-spool]]
+=== Configure the file spool queue
+
+The file spooling queue stores all events in an on disk ring buffer. The spool


We should use file spooling queue or file spool queue consistently.

Is a ring buffer something that end users are likely to know about? If not, it's worth adding a sentence to explain what this is.

I'd say yes, as ring buffers are quite common.

dedemorton · 2018-04-20T00:04:42Z

libbeat/docs/queueconfig.asciidoc

@@ -76,3 +77,147 @@ will be immediately available for consumption.

 The default values is 0s.

+[float]
+[[configuration-internal-queue-spool]]
+=== Configure the file spool queue


Now that we have a couple of queue types, we should explain why users might choose to use a specific type of queue.

Yeah. The interaction of a configured queue and the actual beat differs per beat. This would call for some beat specific documentation. As this will require some more work, I'd prefer to postpone the per-beats documentation for now.

dedemorton · 2018-04-20T00:07:44Z

libbeat/docs/queueconfig.asciidoc

+after a signal from the output has been received.
+
+On disk, the spool operates in pages. The `file.page_size` setting configures
+the files page size on creation time. The optimal page size depends on the


"files page size" isn't grammatically correct. Do you mean:

file's page size?
file page size?
page size for files?

search for other instances of "files page size" and replace them with whichever is correct.

file's page size. A file is divided into pages of page_size.

Note: as followup we want to add a FAQ entry on determining the optimal page size. These can differ by OS and filesystem. Tools and nomenclature is different on different OSes as well.

dedemorton · 2018-04-20T00:13:32Z

libbeat/docs/queueconfig.asciidoc

+------------------------------------------------------------------------------
+queue.spool: ~
+------------------------------------------------------------------------------
+


Might be worth pointing the user to the Configuration options section for the defaults.

Added a link to text before sample

dedemorton · 2018-04-20T00:14:48Z

libbeat/docs/queueconfig.asciidoc

+
+This sample configuration creates a spool of 512MiB, with 16KiB pages. The
+write buffer is flushed if 10MiB of contents, or 1024 events have been
+written. If oldest available event is already waiting for 5s in the write


Change to say, "If the oldest available event has been waiting for 5s..."

dedemorton · 2018-04-20T00:22:16Z

libbeat/docs/queueconfig.asciidoc

+
+===== `file.page_size`
+
+The files page size.


see comment above

dedemorton · 2018-04-20T00:34:00Z

libbeat/docs/queueconfig.asciidoc

+
+The default value is 4096 (4KiB).
+
+Note: This setting should match the file systems minimum block size. If the


same comment as above (NOTE instead of Note). I won't flag all of these.

Also it should be "file system's" (possessive) not "file systems".

dedemorton · 2018-04-20T00:37:36Z

libbeat/docs/queueconfig.asciidoc

+The default value is 4096 (4KiB).
+
+Note: This setting should match the file systems minimum block size. If the
+`page_size` is not a multiple of the file systems block size, the file system


ditto: "file system's" (possessive) not "file systems"

dedemorton · 2018-04-20T00:38:54Z

libbeat/docs/queueconfig.asciidoc

+`page_size` is not a multiple of the file systems block size, the file system
+might create additional read operations on writes.
+
+Note: The page size is only set on file creation time. It can not be changed


"only set at file creation time" sounds better to me. (comment applies to other instances)

dedemorton · 2018-04-20T00:44:33Z

libbeat/docs/queueconfig.asciidoc

+===== `read.flush.timeout`
+
+If configured, the `read.flush.timeout` ensures the spool will wait for more
+events to be flushed to the spool file, If the number of available events is


A couple of typos. Should be:

...if the number of available events is less than the output's bulk_max_size...

I have a hard time writing docs on this setting. Did rewrite complete section. Hope it's somewhat better now.

ruflin · 2018-04-20T06:26:18Z

I added the needs_backport label.

urso · 2018-04-25T12:07:52Z

@dedemorton I updated the docs. Thanks for the input.

I think we might need some more follow up docs. Maybe per beat, as the impact of the queue type kind of depends on the actual beat and the data source.

As FAQ entry I can think of determining the optimal 'page size' using OS tools.

dedemorton

@urso Just one small comment. Otherwise the changes LGTM. Sorry this one fell off my radar. I've created an issue for adding more detail about when/why to configure queue options: #7217

dedemorton · 2018-05-31T02:04:33Z

libbeat/docs/queueconfig.asciidoc

@@ -11,7 +11,7 @@ options in the `queue` section of the +{beatname_lc}.yml+ config file. Only one
 queue type can be configured.


-Example configuration:
+This sample configurations configures the memory queue, buffering up to 4096 events:


The phrase "configurations configure" sounds a bit odd (and should be singular not plural). How about:

This sample configuration sets the memory queue to buffer up to 4096 events.

dedemorton · 2018-05-31T16:26:32Z

libbeat/docs/queueconfig.asciidoc

+
+The default value is "${path.data}/spool.dat".
+
+===== `file.permissions`


@urso The content looks good, but just noticed that some of the sections are missing [float] tags starting with file.permissions.

dedemorton

LGTM

* Add spool docs * review * review (phrasing) * Add missing [float] (cherry picked from commit b424793)

* Add spool docs * Add missing [float] (cherry picked from commit b424793)

* Add spool docs * Add missing [float] (cherry picked from commit 96d170a)

Add spool docs

8680748

urso added the docs label Apr 19, 2018

urso mentioned this pull request Apr 19, 2018

Spooling to disk GA #6859

Closed

39 tasks

dedemorton requested changes Apr 20, 2018

View reviewed changes

ruflin added the needs_backport PR is waiting to be backported to other branches. label Apr 20, 2018

review

2f8c693

This was referenced May 29, 2018

Beats doc updates in 6.x #5632

Closed

Add more guidance about configuring the internal queue #7217

Closed

dedemorton reviewed May 31, 2018

View reviewed changes

review (phrasing)

3dd841e

dedemorton reviewed May 31, 2018

View reviewed changes

Add missing [float]

fbb2c08

dedemorton approved these changes May 31, 2018

View reviewed changes

dedemorton merged commit b424793 into elastic:master May 31, 2018

urso pushed a commit to urso/beats that referenced this pull request Jun 5, 2018

Add file spool to queue docs (elastic#6902)

2002a02

* Add spool docs * review * review (phrasing) * Add missing [float] (cherry picked from commit b424793)

urso mentioned this pull request Jun 5, 2018

Cherry-pick #6902 to 6.3: Add file spool to queue docs #7266

Merged

urso added v6.3.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jun 5, 2018

ph pushed a commit that referenced this pull request Jun 5, 2018

Add file spool to queue docs (#6902) (#7266)

de20601

* Add spool docs * Add missing [float] (cherry picked from commit b424793)

urso deleted the doc-spool branch February 19, 2019 18:56

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Add file spool to queue docs (elastic#6902) (elastic#7266)

f0f6e13

* Add spool docs * Add missing [float] (cherry picked from commit 96d170a)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add file spool to queue docs #6902

Add file spool to queue docs #6902

urso commented Apr 19, 2018

dedemorton left a comment

dedemorton Apr 19, 2018

dedemorton Apr 20, 2018

urso Apr 25, 2018

dedemorton Apr 20, 2018

urso Apr 25, 2018

dedemorton Apr 20, 2018

urso Apr 25, 2018

dedemorton Apr 20, 2018

urso Apr 25, 2018 •

edited

Loading

dedemorton Apr 20, 2018

dedemorton Apr 20, 2018

dedemorton Apr 20, 2018

dedemorton Apr 20, 2018

dedemorton Apr 20, 2018

dedemorton Apr 20, 2018

urso Apr 25, 2018

ruflin commented Apr 20, 2018

urso commented Apr 25, 2018

dedemorton left a comment

dedemorton May 31, 2018 •

edited

Loading

urso May 31, 2018

dedemorton May 31, 2018

dedemorton left a comment


		The default value is 4096 (4KiB).

		Note: This setting should match the file systems minimum block size. If the


		The default value is "${path.data}/spool.dat".

		===== `file.permissions`

Add file spool to queue docs #6902

Add file spool to queue docs #6902

Conversation

urso commented Apr 19, 2018

dedemorton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

urso Apr 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Apr 20, 2018

urso commented Apr 25, 2018

dedemorton left a comment

Choose a reason for hiding this comment

dedemorton May 31, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dedemorton left a comment

Choose a reason for hiding this comment

urso Apr 25, 2018 •

edited

Loading

dedemorton May 31, 2018 •

edited

Loading