-
Notifications
You must be signed in to change notification settings - Fork 4.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Combined 5.0.0 release notes (#2813)
* Add 5.0.0 combined release notes * Remove experimental from Kafka
- Loading branch information
1 parent
ed9e4b9
commit 8a69e53
Showing
3 changed files
with
244 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,240 @@ | ||
| [[release-notes-5.0.0]] | ||
| === Beats version 5.0.0 | ||
|
|
||
| The list below covers changes between 1.x to 5.0.0 releases. | ||
|
|
||
|
|
||
| ==== Breaking changes | ||
|
|
||
| *Affecting all Beats* | ||
|
|
||
| - Rename the `filters` section to `processors`. {pull}1944[1944] | ||
| - Require braces for environment variable expansion in config files {pull}1304[1304] | ||
| - On DEB/RPM installations, the binary files are now found under `/usr/share/{{beat_name}}/bin`, not in `/usr/bin`. {pull}1385[1385] | ||
| - The logs are written by default to self rotating files, instead of syslog. {pull}1371[1371] | ||
| - Remove deprecated `host` option from elasticsearch, logstash and redis outputs. {pull}1474[1474] | ||
| - All configuration settings under `shipper:` are moved to be top level configuration settings. I.e. | ||
| `shipper.name:` becomes `name:` in the configuration file. {pull}1570[1570] | ||
| - The `topology_expire` option of the Elasticserach output was removed. {pull}1907[1907] | ||
| - The Elasticsearch template is now loaded by default. {pull}1993[1993] | ||
| - The Redis output `index` setting is renamed to `key`. `index` still works but it's deprecated. {pull}2077[2077] | ||
| - The undocumented file output `index` setting was removed. Use `filename` instead. {pull}2077[2077] | ||
| - Change Elasticsearch output index configuration to be based on format strings. If index has been configured, no date will be appended anymore to the index name. {pull}2119[2119] | ||
| - If the path specified by the `-c` flag is not absolute and `-path.config` is not specified, it | ||
| is considered relative to the current working directory. {pull}2245[2245] | ||
| - Rename `tls` configurations section to `ssl`. {pull}2330[2330] | ||
| - Rename `certificate_key` configuration to `key`. {pull}2330[2330] | ||
| - Replace `tls.insecure` with `ssl.verification_mode` setting. {pull}2330[2330] | ||
| - Replace `tls.min/max_version` with `ssl.supported_protocols` setting requiring full protocol name. {pull}2330[2330] | ||
| - A dynamic mapping rule is added to the default Elasticsearch template to treat strings as keywords by default. {pull}2688[2688] | ||
|
|
||
|
|
||
| *Filebeat* | ||
|
|
||
| - Scalar values in used in the `fields` configuration setting are no longer automatically converted to strings. {pull}1092[1092] | ||
| - Count field was removed from event as not used in filebeat {issue}778[778] | ||
| - Default location for the registry file was changed to be `data/registry` from the binary directory, | ||
| rather than `.filebeat` in the current working directory. This affects installations for zip/tar.gz/source, | ||
| the location for DEB and RPM packages stays the same. {pull}1373[1373] | ||
| - The state for files which fall under ignore_older is not stored anymore. This has the consequence, that if a file which fell under ignore_older is updated, the whole file will be crawled. | ||
| - Ignore symlinks by default, but they can be enabled with `symlinks` config {pull}1686[1686] {pull}2478[2478] | ||
| - Set `close_inactive` default to 5 minutes (was 1 hour before) | ||
| - Set `clean_removed` and `close_removed` to true by default | ||
|
|
||
|
|
||
| *Packetbeat* | ||
|
|
||
| - Rename output fields in the dns package. Former flag `recursion_allowed` becomes `recursion_available`. {pull}803[803] | ||
| Former SOA field `ttl` becomes `minimum`. {pull}803[803] | ||
| - The fully qualified domain names which are part of output fields values of the dns package now terminate with a dot. {pull}803[803] | ||
| - Remove the `count` field from the exported event {pull}1210[1210] | ||
| - Configuration of redis topology support changed. {pull}1353[1353] | ||
| - Move all Packetbeat configuration options under the packetbeat namespace {issue}1417[1417] | ||
| - Set `enabled` ` in `packetbeat.protocols.icmp` configuration to `true` by default. {pull}1988[1988] | ||
| - Group HTTP fields under `http.request` and `http.response` {pull}2167[2167] | ||
| - Export `http.request.body` and `http.response.body` when configured under `include_body_for` {pull}2167[2167] | ||
| - Move `ignore_outgoing` config to `packetbeat.ignore_outgoing` {pull}2393[2393] | ||
|
|
||
| *Winlogbeat* | ||
|
|
||
| - The `message_inserts` field was replaced with the `event_data` field {issue}1053[1053] | ||
| - The `category` field was renamed to `task` to better align with the Windows Event Log API naming {issue}1053[1053] | ||
| - Remove the `count` field from the exported event {pull}1218[1218] | ||
|
|
||
|
|
||
| ==== Bugfixes | ||
|
|
||
| *Affecting all Beats* | ||
|
|
||
| - Logstash output will not retry events that are not JSON-encodable {pull}927[927] | ||
| - Drain response buffers when pipelining is used by Redis output. {pull}1353[1353] | ||
| - Unterminated environment variable expressions in config files will now cause an error {pull}1389[1389] | ||
| - Fix issue with the automatic template loading when Elasticsearch is not available on Beat start. {issue}1321[1321] | ||
| - Fix bug affecting `-cpuprofile`, `-memprofile`, and `-httpprof` CLI flags {pull}1415[1415] | ||
| - Fix race when multiple outputs access the same event with logstash output manipulating event {issue}1410[1410] {pull}1428[1428] | ||
| - Seed random number generator using crypto.rand package. {pull}1503{1503] | ||
| - Fix beats hanging in `-configtest` {issue}1213[1213] | ||
| - Reset backoff factor on partial ACK. {issue}1803[1803] | ||
| - Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. {issue}1829[1829] | ||
| - Fix logstash output with pipelining mode enabled not reconnecting. {issue}1876[1876] | ||
| - Empty configuration sections become merge-able with variables containing full path. {pull}1900[1900] | ||
| - Fix error message about required fields missing not printing the missing field name. {pull}1900[1900] | ||
| - Fix sync publisher `PublishEvents` return value if client is closed concurrently. {pull}2046[2046] | ||
| - Fix Logstash output handles error twice when asynchronous sending fails. {pull}2441[2441] | ||
| - Fix Elasticsearch structured error response parsing error. {issue}2229[2229] | ||
| - Fixed the run script to allow the overriding of the configuration file. {issue}2171[2171] | ||
| - Fix logstash output crash if no hosts are configured. {issue}2325[2325] | ||
| - Fix beats failing to start due to invalid duplicate key error in configuration file. {pull}2521[2521] | ||
| - Fix panic on non writable logging directory. {pull}2571[2571] | ||
| - Fix ignoring all fields from drop_fields in case the first field is unknown. {pull}2685[2685] | ||
| - Fix dynamic configuration int/uint` to float type conversion. {pull}2698[2698] | ||
| - Fix primitive types conversion if values are read from environment variables. {pull}2698[2698] | ||
|
|
||
|
|
||
| *Filebeat* | ||
|
|
||
| - Stop filebeat if started without any prospectors defined or empty prospectors {pull}644[644] {pull}647[647] | ||
| - Improve shutdown of crawler and prospector to wait for clean completion {pull}720[720] | ||
| - Omit `fields` from Filebeat events when null {issue}899[899] | ||
| - Improvements in registrar dealing with file rotation. {pull}1281[1281] | ||
| - Multiline reader normalizing newline to use `\n`. {pull}1552[1552] | ||
| - Fix potential data loss between Filebeat restarts, reporting unpublished lines as published. {issue}2041[2041] | ||
| - Fix open file handler issue. {issue}2028[2028] {pull}2020[2020] | ||
| - Fix async publisher sending empty events {pull}2455[2455] | ||
| - Fix potential issue with multiple harvester per file on large file numbers or slow output {pull}2541[2541] | ||
| - Fix input buffer on encoding problem. {pull}2416[2416] | ||
| - Fix issue when `clean_removed` and `clean_inactive` were used together that states were not directly removed from the registry. | ||
| - Fix issue where upgrading a 1.x registry file resulted in duplicate state entries. {pull}2792[2792] | ||
|
|
||
|
|
||
| *Packetbeat* | ||
|
|
||
| - Create a proper BPF filter when ICMP is the only enabled protocol {issue}757[757] | ||
| - Check column length in pgsql parser. {issue}565[565] | ||
| - Harden pgsql parser. {issue}565[565] | ||
| - Add missing nil-check to memcached GapInStream handler. {issue}1162[1162] | ||
| - Fix NFSv4 Operation returning the first found first-class operation available in compound requests. {pull}1821[1821] | ||
| - Fix TCP overlapping segments not being handled correctly. {pull}1898[1898] | ||
| - Fix mapping for some Packetbeat flow metrics that were not marked as being longs. {issue}2177[2177] | ||
| - Fix handling of messages larger than the maximum message size (10MB). {pull}2470[2470] | ||
| - Fix the `bpf_filter` setting. {issue}2660[2660] | ||
| - Fix compile issues for OpenBSD. {pull}1347[1347] | ||
|
|
||
|
|
||
| *Winlogbeat* | ||
|
|
||
| - Fix panic when reading messages larger than 32K characters on Windows XP and 2003. {pull}1498[1498] | ||
| - Fix panic that occurs when reading a large events on Windows Vista and newer. {pull}1499[1499] | ||
| - Adding missing argument to the "Stop processing" log message. {pull}1590[1590] | ||
| - Fix issue with rendering forwarded event log records. {pull}1891[1891] | ||
| - Fix potential data loss between Winlogbeat restarts, reporting unpublished lines as published. {issue}2041[2041] | ||
| - Fix corrupt registry file that occurs on power loss by disabling file write caching. {issue}2313[2313] | ||
|
|
||
|
|
||
|
|
||
| ==== Added | ||
|
|
||
| *Affecting all Beats* | ||
|
|
||
| - Update to Go 1.7. {pull}2306[2306] | ||
| - Add option to Elasticsearch output to pass http parameters in index operations {issue}805[805] | ||
| - Improve Logstash and Elasticsearch backoff behavior. {pull}927[927] | ||
| - Add Kafka output. {pull}942[942] {pull}2188[2188] {pull}2190[2190] {pull}2284[2284] | ||
| - Add config file option to configure GOMAXPROCS. {pull}969[969] | ||
| - Improve shutdown handling in libbeat. {pull}1075[1075] | ||
| - Log total non-zero internal metrics on shutdown. {pull}2349[2349] | ||
| - Add `fields` and `fields_under_root` options under the `shipper` configuration {pull}1092[1092] | ||
| - Add the ability to use a SOCKS5 proxy with the Logstash output {issue}823[823] | ||
| - The `-configtest` flag will now print "Config OK" to stdout on success {pull}1249[1249] | ||
| - Add support for TLS to Redis output. {pull}1353[1353] | ||
| - Add SOCKS5 proxy support to Redis output. {pull}1353[1353] | ||
| - Failover and load balancing support in Redis output. {pull}1353[1353] | ||
| - Multiple-worker per host support for Redis output. {pull}1353[1353] | ||
| - Added ability to escape `${x}` in config files to avoid environment variable expansion {pull}1389[1389] | ||
| - Configuration options and CLI flags for setting the home, data and config paths. {pull}1373[1373] | ||
| - Configuration options and CLI flags for setting the default logs path. {pull}1437[1437] | ||
| - Add Elasticsearch template files compatible with Elasticsearch 2.x. {pull}1501[1501] | ||
| - Load the mapping template depending on the Elasticsearch version. {pull}1993[1993] | ||
| - Add conditions to processors. {pull}1623[1623] | ||
| - Enhance `contains` condition to work on fields that are arrays of strings. {issue}2237[2237] | ||
| - Add OR/AND/NOT to the condition associated with the processors. {pull}1983[1983] | ||
| - Improve error message if compiling regular expression from config files fails. {pull}1900[1900] | ||
| - Compression support in the Elasticsearch output. {pull}1835[1835] | ||
| - Periodically log internal metrics. {pull}1955[1955] | ||
| - Add `enabled` setting to all output modules. {pull}1987[1987] | ||
| - Command line flag `-c` can be used multiple times. {pull}1985[1985] | ||
| - Add `-E` CLI flag for overwriting single config options via command line. {pull}1986[1986] | ||
| - Check stdout being available when console output is configured. {issue}2035[2035] | ||
| - Add script to generate the Kibana index-pattern from fields.yml. {pull}2122[2122] | ||
| - Enhance Redis output key selection based on format string. {pull}2169[2169] | ||
| - Configurable Redis `keys` using filters and format strings. {pull}2169[2169] | ||
| - Lookup the configuration file relative to the `-path.config` CLI flag. {pull}2245[2245] | ||
| - Re-write `import_dashboards` script in Golang. {pull}2155[2155] | ||
| - Add support for encrypted private key files by introducing `ssl.key_passphrase` setting. {pull}2330[2330] | ||
| - Add `beat.version` fields to all events. | ||
| - Make sure Beats sent always float values when they are defined as float by sending 5.00000 instead of 5. {pull}2627[2627] | ||
|
|
||
|
|
||
| *Filebeat* | ||
|
|
||
| - Add the ability to set a list of tags for each prospector {pull}1092[1092] | ||
| - Add JSON decoding support {pull}1143[1143] | ||
| - The registry format was changed to an array instead of dict. The migration to the new format will happen automatically at the first startup. {pull}1703[1703] | ||
| - Introduce `close_removed` and `close_renamed` harvester options. {issue}1600[1600] | ||
| - Introduce `close_eof` harvester option. {issue}1600[1600] | ||
| - Add `clean_removed` and `clean_inactive` config option. {issue}1600[1600] | ||
| - Introduce `close_timeout` harvester options {issue}1926[1926] | ||
| - Strip BOM from first message in case of BOM files {issue}2351[2351] | ||
| - Add `harvester_limit` option {pull}2417[2417] | ||
|
|
||
|
|
||
| *Packetbeat* | ||
|
|
||
| - Change the DNS library used throughout the dns package to github.com/miekg/dns. {pull}803[803] | ||
| - Add support for NFS v3 and v4. {pull}1231[1231] | ||
| - Add support for EDNS and DNSSEC. {pull}1292[1292] | ||
| - Add `enabled` setting to Packetbeat protocols. {pull}1988[1988] | ||
| - Add `enabled` setting to Packetbeat network flows configuration. {pull}1988[1988] | ||
| - Add Cassandra protocol analyzer to Packetbeat. {pull}1959[1959] | ||
| - Match connections with IPv6 addresses to processes {pull}2254[2254] | ||
| - Add IP address to `-devices` command output {pull}2327[2327] | ||
| - Add configuration option for the maximum message size. Used to be hard-coded to 10 MB. {pull}2470[2470] | ||
|
|
||
| *Metricbeat* | ||
|
|
||
| - First public release, containing the following modules: system, Apache, MySQL, PostgreSQL, Nginx, Redis, MongoDB, and Zookeeper. | ||
| Metricbeat is replacing Topbeat in 5.0, and its default behaviour is to export system statistics like CPU usage, | ||
| memory usage, Disk IO similar with what Topbeat 1.x is doing. | ||
| - Add `pgid` field to process information. {pull} 2021[2021] | ||
| - Use the new `scaled_float` Elasticsearch type for the percentage values. {pull}2156[2156] | ||
| - Improve mapping by converting `half_float` to `scaled_float` and integers to long. {pull}2430[2430] | ||
| - Add experimental `cgroup` metrics to the system/process MetricSet. {pull}2184[2184] | ||
| - Add experimental `haproxy` module. {pull}2384[2384] | ||
|
|
||
|
|
||
| *Winlogbeat* | ||
|
|
||
| - Add caching of event metadata handles and the system render context for the wineventlog API {pull}888[888] | ||
| - Improve config validation by checking for unknown top-level YAML keys. {pull}1100[1100] | ||
| - Add the ability to set `tags`, `fields`, and `fields_under_root` as options for each event log {pull}1092[1092] | ||
| - Add additional data to the events published by Winlogbeat. The new fields are `activity_id`, | ||
| `event_data`, `keywords`, `opcode`, `process_id`, `provider_guid`, `related_activity_id`, | ||
| `task`, `thread_id`, `user_data`, and `version`. {issue}1053[1053] | ||
| - Add `event_id`, `level`, and `provider` configuration options for filtering events {pull}1218[1218] | ||
| - Add `include_xml` configuration option for including the raw XML with the event {pull}1218[1218] | ||
|
|
||
|
|
||
| ==== Deprecated | ||
|
|
||
| *Affecting all Beats* | ||
|
|
||
| - The support for doing GeoIP lookups is deprecated and will be removed in version 6.0. {pull}1601[1601] | ||
| - Topology map is deprecated. This applies to the settings: `refresh_topology_freq`, `topology_expire`, `save_topology`, | ||
| `host_topology`, `password_topology`, `db_topology`. | ||
| - Setting `port` has been deprecated in Redis and Logstash outputs. {pull}2620[2620] | ||
|
|
||
| *Filebeat* | ||
|
|
||
| - Deprecate `close_older` option and replace it with `close_inactive`. {issue}2051[2051] | ||
| - Deprecate `force_close_files` option and replace it with `close_removed` and `close_renamed`. {issue}1600[1600] | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters