From da917a00db9fd3d519cc6dfd8c2a0cb3f929a8e2 Mon Sep 17 00:00:00 2001 From: Steffen Siering Date: Wed, 16 Aug 2017 20:11:36 +0200 Subject: [PATCH] Update docs (#4753) (#4878) - remove removed settings in libbeat/filebeat docs - add tags/fields/fields_under_root/processor settings to every beat (cherry picked from commit 2c07801978ace2943a2068550e6bed874015f706) --- .../docs/filebeat-general-options.asciidoc | 39 +-------------- filebeat/docs/filebeat-options.asciidoc | 9 +++- filebeat/docs/load-balancing.asciidoc | 35 -------------- filebeat/docs/migration.asciidoc | 6 +-- filebeat/docs/overview.asciidoc | 2 +- heartbeat/docs/heartbeat-options.asciidoc | 38 +++++++++++++++ libbeat/docs/generalconfig.asciidoc | 11 ++--- metricbeat/docs/metricbeat-options.asciidoc | 9 ++-- packetbeat/docs/packetbeat-options.asciidoc | 48 +++++++++++++++++++ winlogbeat/docs/winlogbeat-options.asciidoc | 17 +++++-- 10 files changed, 121 insertions(+), 93 deletions(-) diff --git a/filebeat/docs/filebeat-general-options.asciidoc b/filebeat/docs/filebeat-general-options.asciidoc index 704c85e589c8..80076d8124cc 100644 --- a/filebeat/docs/filebeat-general-options.asciidoc +++ b/filebeat/docs/filebeat-general-options.asciidoc @@ -16,43 +16,6 @@ Beats. These options are in the `filebeat` namespace. -[float] -==== `spool_size` - -The event count spool threshold. This setting forces a network flush if the number of events in the spooler exceeds -the specified value. - -[source,yaml] -------------------------------------------------------------------------------------- -filebeat.spool_size: 2048 -------------------------------------------------------------------------------------- - -See <> for more information about how this setting affects load balancing. - -[float] -==== `publish_async` - -experimental[] -deprecated[5.3.0] - -If enabled, the publisher pipeline in Filebeat operates in async mode preparing -a new batch of lines while waiting for ACK. This option can improve load-balancing -throughput at the cost of increased memory usage. The default value is false. - -See <> for more information about how this setting affects load balancing. - -[float] -==== `idle_timeout` - -A duration string that specifies how often the spooler is flushed. After the -`idle_timeout` is reached, the spooler is flushed even if the `spool_size` has not been reached. - -[source,yaml] -------------------------------------------------------------------------------------- -filebeat.idle_timeout: 5s -------------------------------------------------------------------------------------- - - [float] ==== `registry_file` @@ -76,7 +39,7 @@ That means in case there are some states where the TTL expired, these are only r The full path to the directory that contains additional prospector configuration files. Each configuration file must end with `.yml`. Each config file must also specify the full Filebeat config hierarchy even though only the prospector part of the file is processed. All global -options, such as `spool_size`, are ignored. +options, such as `registry_file`, are ignored. The `config_dir` option MUST point to a directory other than the directory where the main Filebeat config file resides. diff --git a/filebeat/docs/filebeat-options.asciidoc b/filebeat/docs/filebeat-options.asciidoc index 7e72e9e8caf7..8458096de10f 100644 --- a/filebeat/docs/filebeat-options.asciidoc +++ b/filebeat/docs/filebeat-options.asciidoc @@ -191,6 +191,14 @@ top-level fields in the output document instead of being grouped under a `fields` sub-dictionary. If the custom field names conflict with other field names added by Filebeat, then the custom fields overwrite the other fields. +[float] +==== `processors` + +A list of processors to apply to the data generated by the prospector. + +See <> for information about specifying +processors in your config. + [float] [[ignore-older]] ==== `ignore_older` @@ -484,4 +492,3 @@ by assigning a higher limit of harvesters. The `enabled` option can be used with each prospector to define if a prospector is enabled or not. By default, enabled is set to true. - diff --git a/filebeat/docs/load-balancing.asciidoc b/filebeat/docs/load-balancing.asciidoc index 7b091a12d84b..330db6d82954 100644 --- a/filebeat/docs/load-balancing.asciidoc +++ b/filebeat/docs/load-balancing.asciidoc @@ -17,16 +17,6 @@ output.logstash: The `loadbalance` option is available for Redis, Logstash, and Elasticsearch outputs. The Kafka output handles load balancing internally. -Filebeat can send events in a few different modes: - -* **Send events to one host after another:** -+ -By default, when you configure Filebeat to send events to multiple hosts -(`loadbalance: true`), Filebeat will send the events to one host after -another. This mode requires the least memory and CPU usage. This is not -true load balancing in the sense that Filebeat doesn't account for the load -being processed by each host. -+ The load balancer also supports multiple workers per host. The default is `worker: 1`. If you increase the number of workers, additional network connections will be used. The total number of workers participating @@ -47,28 +37,3 @@ output.logstash: ------------------------------------------------------------------------------- + In this example, there are 4 workers participating in load balancing. - -* **Send events to `N` hosts in lock-step:** -+ -You can configure Filebeat to send events to `N` hosts in lock-step by setting -`spool_size = N * bulk_max_size`. In lock-step mode, the batch collected by the -spooler is split up into smaller batches of size `bulk_max_size`. These smaller -batches are load balanced between available connections. Filebeat waits for all -sub-batches to be published before it retrieves another batch from the spooler. -+ -This mode requires more memory and CPU usage than the previous mode. -+ -Example: -+ -[source,yaml] -------------------------------------------------------------------------------- -filebeat.prospectors: -- type: log - paths: - - /var/log/*.log -filebeat.spool_size: 4096 -output.logstash: - hosts: ["localhost:5044", "localhost:5045"] - loadbalance: true - bulk_max_size: 2048 -------------------------------------------------------------------------------- diff --git a/filebeat/docs/migration.asciidoc b/filebeat/docs/migration.asciidoc index 3952df4d2097..aa268542b8df 100644 --- a/filebeat/docs/migration.asciidoc +++ b/filebeat/docs/migration.asciidoc @@ -321,11 +321,11 @@ replaced by options specified in libbeat. The `config_dir` option specifies the path to the directory that contains additional configuration files. This option MUST point to a directory other than the directory where the main Filebeat config file resides. |`-idle-timeout` -|`idle_timeout` -|`idle_timeout` was moved to the config file and removed as a flag. +| +|`idle_timeout` was removed. Libbeat is used for publishing logs. |`-spool-size` -|`spool_size` +|`queue.mem.events` |`spool_size` was moved to the config file and removed as a flag. |`-harvester-buff-size` diff --git a/filebeat/docs/overview.asciidoc b/filebeat/docs/overview.asciidoc index 8db37a4cc87c..37e98845f6bb 100644 --- a/filebeat/docs/overview.asciidoc +++ b/filebeat/docs/overview.asciidoc @@ -8,7 +8,7 @@ Filebeat is a log data shipper for local files. Installed as an agent on your servers, Filebeat monitors the log directories or specific log files, tails the files, and forwards them either to https://www.elastic.co/products/elasticsearch[Elasticsearch] or https://www.elastic.co/products/logstash[Logstash] for indexing. -Here's how Filebeat works: When you start Filebeat, it starts one or more prospectors that look in the local paths you've specified for log files. For each log file that the prospector locates, Filebeat starts a harvester. Each harvester reads a single log file for new content and sends the new log data to the spooler, which aggregates the events and sends the aggregated data to the output that you've configured for Filebeat. +Here's how Filebeat works: When you start Filebeat, it starts one or more prospectors that look in the local paths you've specified for log files. For each log file that the prospector locates, Filebeat starts a harvester. Each harvester reads a single log file for new content and sends the new log data to libbeat, which aggregates the events and sends the aggregated data to the output that you've configured for Filebeat. image:./images/filebeat.png[Beats design] diff --git a/heartbeat/docs/heartbeat-options.asciidoc b/heartbeat/docs/heartbeat-options.asciidoc index 0cddf6f6fba6..afd22133914a 100644 --- a/heartbeat/docs/heartbeat-options.asciidoc +++ b/heartbeat/docs/heartbeat-options.asciidoc @@ -122,6 +122,44 @@ If the timeout is exceeded, Heartbeat publishes a `service-down` event. If the value specified for `timeout` is greater than `schedule`, intermediate checks will not be executed by the scheduler. +[float] +[[monitor-fields]] +==== `fields` + +Optional fields that you can specify to add additional information to the +output. For example, you might add fields that you can use for filtering log +data. Fields can be scalar values, arrays, dictionaries, or any nested +combination of these. By default, the fields that you specify here will be +grouped under a `fields` sub-dictionary in the output document. To store the +custom fields as top-level fields, set the `fields_under_root` option to true. +If a duplicate field is declared in the general configuration, then its value +will be overwritten by the value declared here. + +[float] +[[monitor-fields-under-root]] +==== `fields_under_root` + +If this option is set to true, the custom <> +are stored as top-level fields in the output document instead of being grouped +under a `fields` sub-dictionary. If the custom field names conflict with other +field names added by Heartbeat, then the custom fields overwrite the other +fields. + +[float] +[[monitor-tags]] +==== `tags` + +A list of tags that will be sent with the monitor event. This setting is optional. + +[float] +[[monitor-processors]] +==== `processors` + +A list of processors to apply to the data generated by the monitor. + +See <> for information about specifying +processors in your config. + [float] [[monitor-watch-poll-file]] ==== `watch.poll_file` diff --git a/libbeat/docs/generalconfig.asciidoc b/libbeat/docs/generalconfig.asciidoc index db07e820cf19..a3f7426160aa 100644 --- a/libbeat/docs/generalconfig.asciidoc +++ b/libbeat/docs/generalconfig.asciidoc @@ -91,15 +91,12 @@ fields: ------------------------------------------------------------------------------ [float] -==== `queue_size` +==== `processors` -The internal queue size for single events in the processing pipeline. The default -value is 1000. +A list of processors to apply to the data generated by the beat. -[float] -==== `bulk_queue_size` - -(DO NOT TOUCH) The internal queue size for bulk events in the processing pipeline. The default value is 0. +See <> for information about specifying +processors in your config. [float] ==== `max_procs` diff --git a/metricbeat/docs/metricbeat-options.asciidoc b/metricbeat/docs/metricbeat-options.asciidoc index 97ba20c2acdf..3e4bc4272d0c 100644 --- a/metricbeat/docs/metricbeat-options.asciidoc +++ b/metricbeat/docs/metricbeat-options.asciidoc @@ -19,7 +19,7 @@ metricbeat: fields: dc: west tags: ["tag"] - filters: + processors: .... ---- @@ -131,9 +131,10 @@ A dictionary of fields that will be sent with the metricset event. This setting A list of tags that will be sent with the metricset event. This setting is optional. [float] -==== `filters` +==== `processors` -deprecated[5.1,This option will be renamed and changed in a future release] +A list of processors to apply to the data generated by the metricset. -A list of filters to apply to the data generated by the module. +See <> for information about specifying +processors in your config. diff --git a/packetbeat/docs/packetbeat-options.asciidoc b/packetbeat/docs/packetbeat-options.asciidoc index bedbffbb87cb..c5565e7ee9da 100644 --- a/packetbeat/docs/packetbeat-options.asciidoc +++ b/packetbeat/docs/packetbeat-options.asciidoc @@ -353,6 +353,41 @@ in time. Periodical reporting can be disabled by setting the value to -1. If disabled, flows are still reported once being timed out. The default value is 10s. +[float] +[[packetbeat-configuration-flows-fields]] +==== `fields` + +Optional fields that you can specify to add additional information to the +output. For example, you might add fields that you can use for filtering log +data. Fields can be scalar values, arrays, dictionaries, or any nested +combination of these. By default, the fields that you specify here will be +grouped under a `fields` sub-dictionary in the output document. To store the +custom fields as top-level fields, set the `fields_under_root` option to true. +If a duplicate field is declared in the general configuration, then its value +will be overwritten by the value declared here. + +[float] +==== `fields_under_root` + +If this option is set to true, the custom <> +are stored as top-level fields in the output document instead of being grouped +under a `fields` sub-dictionary. If the custom field names conflict with other +field names added by Packetbeat, then the custom fields overwrite the other +fields. + +[float] +==== `tags` + +A list of tags that will be sent with the protocol event. This setting is optional. + +[float] +==== `processors` + +A list of processors to apply to the data generated by the protocol. + +See <> for information about specifying +processors in your config. + [[configuration-protocols]] == Specify which transaction protocols to monitor @@ -485,6 +520,19 @@ under a `fields` sub-dictionary. If the custom field names conflict with other field names added by Packetbeat, then the custom fields overwrite the other fields. +[float] +==== `tags` + +A list of tags that will be sent with the transaction event. This setting is optional. + +[float] +==== `processors` + +A list of processors to apply to the data generated by the protocol. + +See <> for information about specifying +processors in your config. + [[packetbeat-icmp-options]] === Capture ICMP traffic diff --git a/winlogbeat/docs/winlogbeat-options.asciidoc b/winlogbeat/docs/winlogbeat-options.asciidoc index 2bcebc575e9e..406d5adca541 100644 --- a/winlogbeat/docs/winlogbeat-options.asciidoc +++ b/winlogbeat/docs/winlogbeat-options.asciidoc @@ -191,10 +191,11 @@ event IDs. [source,yaml] -------------------------------------------------------------------------------- -processors: -- drop_event.when.and: - - equals.log_name: Security - - not.or: +winlogbeat.event_logs: + - name: Security + event_id: ... + processors: + - drop_event.when.not.or: - equals.event_id: 903 - equals.event_id: 1024 - equals.event_id: 4624 @@ -340,3 +341,11 @@ under a `fields` sub-dictionary. If the custom field names conflict with other field names added by Winlogbeat, then the custom fields overwrite the other fields. +[float] +==== `event_logs.processors` + +A list of processors to apply to the data generated by the event log. + +See <> for information about specifying +processors in your config. +