From 5062eddba7208ab78183ed6a686a9e48526438ee Mon Sep 17 00:00:00 2001 From: Tudor Golubenco Date: Wed, 26 Jul 2017 18:40:55 +0300 Subject: [PATCH] Added breaking changes doc for 6.0 Part of #4737, but there are some TODOs left. --- libbeat/docs/breaking.asciidoc | 135 ++++++++++++++++++++++++++++++++- 1 file changed, 134 insertions(+), 1 deletion(-) diff --git a/libbeat/docs/breaking.asciidoc b/libbeat/docs/breaking.asciidoc index 78156627471..fcf83d31021 100644 --- a/libbeat/docs/breaking.asciidoc +++ b/libbeat/docs/breaking.asciidoc @@ -2,15 +2,148 @@ == Breaking changes As a general rule, we strive to keep backwards compatibility between minor -versions (e.g. 5.x to 5.y) so you can upgrade without any configuration file +versions (e.g. 6.x to 6.y) so you can upgrade without any configuration file changes, but there are breaking changes between major versions (e.g. 5.x to 6.y). +* <> * <> We also list the breaking changes at the top of the <> for each version. +[[breaking-changes-6.0]] +=== Breaking changes in 6.0 + +This section discusses the main changes that you should be aware of if you +upgrade the Beats from version 5.x to 6.x. Please also review the relevant +Breaking Changes sections of the <>. + +// TODO: better link to the consolidated release notes for 6.0.0. + + +[[breaking-changes-spooler-removed]] +==== Filebeat spooler removed + +Version 6.0 comes with a new architecture for the internal pipeline of all +Beats. This architecture refactoring is mostly internal, but one of the more +visible effects is that the Spooler component of Filebeat is removed. The +functionality of the Spooler was similar to the one of the publisher queue from +libbeat (the code shared by all Beats), and the presence of multiple queues +made the performance tuning of Filebeat more complex than it needed to be. + +As a result, the following options are removed: + +- `spool_size` +- `publish_async` +- `idle_timeout` +- `queue_size` +- `bulk_queue_size` + +The first three are specific to Filebeat, while `queue_size` and +`bulk_queue_size` exist in all Beats. If any of these options is set, Filebeat +6.0 will refuse to start. + +Instead of the settings above, the `queue.mem` settings are introduced. If you +had to tune the `spool_size` or the `queue_size` before, you might want to tune +the `queue.mem.events` when upgrading. However, it is best to leave the rest of +the `queue.mem` settings to their default values, as they are appropriate for +all loads. + +The `publish_async` option (which was deprecated since 5.3) is removed because +the new pipeline already works asynchronously by default. + +// TODO: for the above new settings, link to their configuration settings. + + +[[breaking-changes-types]] +==== Filebeat prospector type and document type changes + +The `document_type` setting, from the prospector configuration, was removed +because the `_type` concept is being removed from Elasticsearch. Instead of the +`document_type` setting, you can use a custom field. + +This has led also to the rename of the `input_type` configuration setting to +`type`. This change is backwards compatible because the old setting still +works. However, the `input_type` output field was renamed to `prospector_type`. + +[[breaking-changes-default-config]] +==== Filebeat default prospector disabled in the configuration file + +The default startup behaviour (based on the included sample configuration) of +Filebeat was to read all the files matching the `/var/log/*.log` pattern. +Starting with version 6.0, Filebeat doesn't read any files in its default +configuration. However, you can easily enable the `system` module, for example +with a CLI flag: + +["source","sh",subs="attributes"] +---- +filebeat --modules=system +---- + +[[breaking-changes-import-dashboards]] +==== Changes for importing the Kibana dashboards + +The `import_dashboards` program, used to load the Kibana dashboards in previous +versions of Beats, is replaced by the `setup` command. For example, the +following command: + +["source","sh",subs="attributes"] +---- +./scripts/import_dashboards -user elastic -pass changeme +---- + +Can be replaced with: + +["source","sh",subs="attributes"] +---- +./filebeat setup -E "output.elasticsearch.username=elastic" -E "output.elasticsearch.password=changeme" +---- + +Note that the `-E` flags are only required if the Elasticsearch output is not +already configured in the configuration file. + +Besides the change in the commands, it's important to note that loading the +Kibana dashboards works differently in the 6.0 version of the stack. Prior to +6.0, the dashboards were inserted directly in the `.kibana` Elasticsearch +index. Starting with 6.0, the Beats use a Kibana server API. This means that +the Beat that loads the dashboards needs direct access to Kibana and that the +Kibana URL needs to be set. The option to set the Kibana URL is +`setup.kibana.host`, which you can set in the configuration file or via the +`-E` CLI flag: + + +["source","sh",subs="attributes"] +---- +./filebeat setup -E "setup.kibana.host=http://kibana-host:5601" +---- + +The default value for the Kibana host is `localhost:5601`. + +[[breaking-changes-filters]] +==== Metricbeat filters renamed to processors + +The "local" processors, which are configured at the module level, used to be +called `filters` in Metricbeat, but were offering similar functionality with +the global `processors`. A notable difference between the two was that the +filters accessed fields relatively to the metricset (for example, +`mount_point`), while the processors referred to fields by their fully +qualified name (for example, `system.filesystem.mount_point`). + +Starting with version 6.0, the `filters` are renamed to `processors` and they +can access the fields only by using the fully qualified names. + +[[breaking-changes-cgo]] +==== Binaries are dynamically compiled against libc + +Prior to 6.0, Metricbeat and Packetbeat were compiled using +https://golang.org/cmd/cgo/[Cgo], while Filebeat, Winlogbeat, and Heartbeat +were compiled using the pure Go compiler. One of the side-effects of compiling +with Cgo is that libc is dynamically compiled. Starting with 6.0, all the Beats +are compiled using Cgo and therefore dynamically compiled against libc. This +can reduce the portability of the binaries, but none of the supported platforms +is affected. + [[breaking-changes-5.0]] === Breaking changes in 5.0