Update docs (elastic#4753) (elastic#4878)

- remove removed settings in libbeat/filebeat docs
- add tags/fields/fields_under_root/processor settings to every beat

(cherry picked from commit 2c07801)

filebeat/docs/filebeat-general-options.asciidoc

@@ -16,43 +16,6 @@ Beats.
 
 These options are in the `filebeat` namespace.
 
-[float]
-==== `spool_size`
-
-The event count spool threshold. This setting forces a network flush if the number of events in the spooler exceeds
-the specified value.
-
-[source,yaml]
--------------------------------------------------------------------------------------
-filebeat.spool_size: 2048
--------------------------------------------------------------------------------------
-
-See <<load-balancing>> for more information about how this setting affects load balancing.
-
-[float]
-==== `publish_async`
-
-experimental[]
-deprecated[5.3.0]
-
-If enabled, the publisher pipeline in Filebeat operates in async mode preparing
-a new batch of lines while waiting for ACK. This option can improve load-balancing
-throughput at the cost of increased memory usage. The default value is false.
-
-See <<load-balancing>> for more information about how this setting affects load balancing.
-
-[float]
-==== `idle_timeout`
-
-A duration string that specifies how often the spooler is flushed. After the
-`idle_timeout` is reached, the spooler is flushed even if the `spool_size` has not been reached.
-
-[source,yaml]
--------------------------------------------------------------------------------------
-filebeat.idle_timeout: 5s
--------------------------------------------------------------------------------------
-
-
 [float]
 ==== `registry_file`
 
@@ -76,7 +39,7 @@ That means in case there are some states where the TTL expired, these are only r
 The full path to the directory that contains additional prospector configuration files.
 Each configuration file must end with `.yml`. Each config file must also specify the full Filebeat
 config hierarchy even though only the prospector part of the file is processed. All global
-options, such as `spool_size`, are ignored.
+options, such as `registry_file`, are ignored.
 
 The `config_dir` option MUST point to a directory other than the directory where the main Filebeat config file resides.
 

filebeat/docs/filebeat-options.asciidoc

@@ -191,6 +191,14 @@ top-level fields in the output document instead of being grouped under a
 `fields` sub-dictionary. If the custom field names conflict with other field
 names added by Filebeat, then the custom fields overwrite the other fields.
 
+[float]
+==== `processors`
+
+A list of processors to apply to the data generated by the prospector.
+
+See <<filtering-and-enhancing-data>> for information about specifying
+processors in your config.
+
 [float]
 [[ignore-older]]
 ==== `ignore_older`
@@ -484,4 +492,3 @@ by assigning a higher limit of harvesters.
 
 The `enabled` option can be used with each prospector to define if a prospector is enabled or not. By default, enabled is set to true.
 
-

filebeat/docs/load-balancing.asciidoc

@@ -17,16 +17,6 @@ output.logstash:
 The `loadbalance` option is available for Redis, Logstash, and Elasticsearch
 outputs. The Kafka output handles load balancing internally.
 
-Filebeat can send events in a few different modes:
-
-* **Send events to one host after another:**
-+
-By default, when you configure Filebeat to send events to multiple hosts
-(`loadbalance: true`), Filebeat will send the events to one host after
-another. This mode requires the least memory and CPU usage. This is not
-true load balancing in the sense that Filebeat doesn't account for the load
-being processed by each host.
-+
 The load balancer also supports multiple workers per host. The default is
 `worker: 1`. If you increase the number of workers, additional network
 connections will be used.  The total number of workers participating
@@ -47,28 +37,3 @@ output.logstash:
 -------------------------------------------------------------------------------
 +
 In this example, there are 4 workers participating in load balancing.
-
-* **Send events to `N` hosts in lock-step:**
-+
-You can configure Filebeat to send events to `N` hosts in lock-step by setting
-`spool_size = N * bulk_max_size`. In lock-step mode, the batch collected by the
-spooler is split up into smaller batches of size `bulk_max_size`. These smaller
-batches are load balanced between available connections. Filebeat waits for all
-sub-batches to be published before it retrieves another batch from the spooler.
-+
-This mode requires more memory and CPU usage than the previous mode.
-+
-Example:
-+
-[source,yaml]
--------------------------------------------------------------------------------
-filebeat.prospectors:
-- type: log
-  paths:
-    - /var/log/*.log
-filebeat.spool_size: 4096
-output.logstash:
-  hosts: ["localhost:5044", "localhost:5045"]
-  loadbalance: true
-  bulk_max_size: 2048
--------------------------------------------------------------------------------

filebeat/docs/migration.asciidoc

@@ -321,11 +321,11 @@ replaced by options specified in libbeat.
 The `config_dir` option specifies the path to the directory that contains additional configuration files. This option MUST point to a directory other than the directory where the main Filebeat config file resides.
 
 |`-idle-timeout`
-|`idle_timeout`
-|`idle_timeout` was moved to the config file and removed as a flag.
+|
+|`idle_timeout` was removed. Libbeat is used for publishing logs.
 
 |`-spool-size`
-|`spool_size`
+|`queue.mem.events`
 |`spool_size` was moved to the config file and removed as a flag.
 
 |`-harvester-buff-size`

filebeat/docs/overview.asciidoc

@@ -8,7 +8,7 @@
 Filebeat is a log data shipper for local files. Installed as an agent on your servers, Filebeat monitors the log directories or specific log files, tails the files,
 and forwards them either to https://www.elastic.co/products/elasticsearch[Elasticsearch] or https://www.elastic.co/products/logstash[Logstash] for indexing.
 
-Here's how Filebeat works: When you start Filebeat, it starts one or more prospectors that look in the local paths you've specified for log files. For each log file that the prospector locates, Filebeat starts a harvester. Each harvester reads a single log file for new content and sends the new log data to the spooler, which aggregates the events and sends the aggregated data to the output that you've configured for Filebeat.
+Here's how Filebeat works: When you start Filebeat, it starts one or more prospectors that look in the local paths you've specified for log files. For each log file that the prospector locates, Filebeat starts a harvester. Each harvester reads a single log file for new content and sends the new log data to libbeat, which aggregates the events and sends the aggregated data to the output that you've configured for Filebeat.
 
 image:./images/filebeat.png[Beats design]
 

heartbeat/docs/heartbeat-options.asciidoc

@@ -122,6 +122,44 @@ If the timeout is exceeded, Heartbeat publishes a `service-down` event. If the
 value specified for `timeout` is greater than `schedule`, intermediate checks
 will not be executed by the scheduler.
 
+[float]
+[[monitor-fields]]
+==== `fields`
+
+Optional fields that you can specify to add additional information to the
+output. For example, you might add fields that you can use for filtering log
+data. Fields can be scalar values, arrays, dictionaries, or any nested
+combination of these. By default, the fields that you specify here will be
+grouped under a `fields` sub-dictionary in the output document. To store the
+custom fields as top-level fields, set the `fields_under_root` option to true.
+If a duplicate field is declared in the general configuration, then its value
+will be overwritten by the value declared here.
+
+[float]
+[[monitor-fields-under-root]]
+==== `fields_under_root`
+
+If this option is set to true, the custom <<monitor-fields>>
+are stored as top-level fields in the output document instead of being grouped
+under a `fields` sub-dictionary. If the custom field names conflict with other
+field names added by Heartbeat, then the custom fields overwrite the other
+fields.
+
+[float]
+[[monitor-tags]]
+==== `tags`
+
+A list of tags that will be sent with the monitor event. This setting is optional.
+
+[float]
+[[monitor-processors]]
+==== `processors`
+
+A list of processors to apply to the data generated by the monitor.
+
+See <<filtering-and-enhancing-data>> for information about specifying
+processors in your config.
+
 [float]
 [[monitor-watch-poll-file]]
 ==== `watch.poll_file`

libbeat/docs/generalconfig.asciidoc

@@ -91,15 +91,12 @@ fields:
 ------------------------------------------------------------------------------
 
 [float]
-==== `queue_size`
+==== `processors`
 
-The internal queue size for single events in the processing pipeline. The default
-value is 1000.
+A list of processors to apply to the data generated by the beat.
 
-[float]
-==== `bulk_queue_size`
-
-(DO NOT TOUCH) The internal queue size for bulk events in the processing pipeline. The default value is 0.
+See <<filtering-and-enhancing-data>> for information about specifying
+processors in your config.
 
 [float]
 ==== `max_procs`

metricbeat/docs/metricbeat-options.asciidoc

@@ -19,7 +19,7 @@ metricbeat:
       fields:
         dc: west
       tags: ["tag"]
-      filters:
+      processors:
         ....
 ----
 
@@ -131,9 +131,10 @@ A dictionary of fields that will be sent with the metricset event. This setting
 A list of tags that will be sent with the metricset event. This setting is optional.
 
 [float]
-==== `filters`
+==== `processors`
 
-deprecated[5.1,This option will be renamed and changed in a future release]
+A list of processors to apply to the data generated by the metricset.
 
-A list of filters to apply to the data generated by the module.
+See <<filtering-and-enhancing-data>> for information about specifying
+processors in your config.
 

packetbeat/docs/packetbeat-options.asciidoc

@@ -353,6 +353,41 @@ in time. Periodical reporting can be disabled by setting the value to -1. If
 disabled, flows are still reported once being timed out. The default value is
 10s.
 
+[float]
+[[packetbeat-configuration-flows-fields]]
+==== `fields`
+
+Optional fields that you can specify to add additional information to the
+output. For example, you might add fields that you can use for filtering log
+data. Fields can be scalar values, arrays, dictionaries, or any nested
+combination of these. By default, the fields that you specify here will be
+grouped under a `fields` sub-dictionary in the output document. To store the
+custom fields as top-level fields, set the `fields_under_root` option to true.
+If a duplicate field is declared in the general configuration, then its value
+will be overwritten by the value declared here.
+
+[float]
+==== `fields_under_root`
+
+If this option is set to true, the custom <<packetbeat-configuration-flows-fields>>
+are stored as top-level fields in the output document instead of being grouped
+under a `fields` sub-dictionary. If the custom field names conflict with other
+field names added by Packetbeat, then the custom fields overwrite the other
+fields.
+
+[float]
+==== `tags`
+
+A list of tags that will be sent with the protocol event. This setting is optional.
+
+[float]
+==== `processors`
+
+A list of processors to apply to the data generated by the protocol.
+
+See <<filtering-and-enhancing-data>> for information about specifying
+processors in your config.
+
 
 [[configuration-protocols]]
 == Specify which transaction protocols to monitor
@@ -485,6 +520,19 @@ under a `fields` sub-dictionary. If the custom field names conflict with other
 field names added by Packetbeat, then the custom fields overwrite the other
 fields.
 
+[float]
+==== `tags`
+
+A list of tags that will be sent with the transaction event. This setting is optional.
+
+[float]
+==== `processors`
+
+A list of processors to apply to the data generated by the protocol.
+
+See <<filtering-and-enhancing-data>> for information about specifying
+processors in your config.
+
 [[packetbeat-icmp-options]]
 === Capture ICMP traffic
 

winlogbeat/docs/winlogbeat-options.asciidoc

@@ -191,10 +191,11 @@ event IDs.
 
 [source,yaml]
 --------------------------------------------------------------------------------
-processors:
-- drop_event.when.and:
-    - equals.log_name: Security
-    - not.or:
+winlogbeat.event_logs:
+  - name: Security
+    event_id: ...
+    processors:
+      - drop_event.when.not.or:
         - equals.event_id: 903
         - equals.event_id: 1024
         - equals.event_id: 4624
@@ -340,3 +341,11 @@ under a `fields` sub-dictionary. If the custom field names conflict with other
 field names added by Winlogbeat, then the custom fields overwrite the other
 fields.
 
+[float]
+==== `event_logs.processors`
+
+A list of processors to apply to the data generated by the event log.
+
+See <<filtering-and-enhancing-data>> for information about specifying
+processors in your config.
+