Beats version 6.4.0

Known issue

Due to a packaging mistake, the modules.d configuration directory is installed in the wrong path in the Metricbeat DEB and RPM packages. This issue results in an empty list when you run metricbeat modules list and failures when you try to enable or disable modules. To work around this issue, run the following command:

sudo cp -r /usr/share/metricbeat/modules.d /etc/metricbeat/

This issue affects all new installations on DEB and RPM. Upgrades will run, but use old configurations defined in the modules.d directory from the previous installation.

The issue will be fixed in the 6.4.1 release.

Breaking changes

Affecting all Beats

Set default kafka version to 1.0.0 in kafka output. Older versions are still supported by configuring the version setting. Minimally supported version is 0.11 (older versions might work, but are untested). 7025

Heartbeat

Rename http.response.status to http.response.status_code to align with ECS. 7274
Remove type field as not needed. 7307

Metricbeat

Fixed typo in values for state_container status.phase, from terminate to terminated. 6916
RabbitMQ management plugin path is now configured at the module level instead of having to do it in each of the metricsets. New management_path_prefix option should be used now 7074
RabbitMQ node metricset only collects metrics of the instance it connects to, node.collect: cluster can be used to collect all nodes as before. 6556 6971
Change http/server metricset to put events by default under http.server and prefix config options with server.. 7100
Disable dedotting in docker module configuration. This will change the out-of-the-box behaviour, but not the one of already configured instances. 7485
Fix typo in etcd/self metricset fields from *.bandwithrate to *.bandwidthrate. 7456
Changed the definition of the system.cpu.total.pct and system.cpu.total.norm.cou fields to exclude the IOWait time. 7691

Bugfixes

Affecting all Beats

Error out on invalid Autodiscover template conditions settings. 7200
Allow to override the ignore_above option when defining new field with the type keyword. 7238
Fix a panic on the Dissect processor when we have data remaining after the last delimiter. 7449
When we fail to build a Kubernetes' indexer or matcher we produce a warning but we don’t add them to the execution. 7466
Fix default value for logging.files.keepfiles. It was being set to 0 and now it’s set to the documented value of 7. 7494
Retain compatibility with older Docker server versions. 7542
Fix errors unpacking configs modified via CLI by ignoring -E key=value pairs with missing value. 7599

Auditbeat

Allow auditbeat setup to run without requiring elevated privileges for the audit client. 7111
Fix goroutine leak that occurred when the auditd module was stopped. 7163

Filebeat

Fix a data race between stopping and starting of the harvesters. 6879
Fix an issue when parsing ISO8601 dates with timezone definition 7367
Fix Grok pattern of MongoDB module. 7568
Fix registry duplicates and log resending on upgrade. 7634

Metricbeat

Fix Windows service metricset when using a 32-bit binary on a 64-bit OS. 7294
Do not report Metricbeat container host as hostname in Kubernetes deployment. 7199
Ensure metadata updates don’t replace existing pod metrics. 7573
Fix kubernetes pct fields reporting. 7677
Add support for new kube_node_status_condition in Kubernetes state_node. 7699

Added

Affecting all Beats

Add dissect processor. 6925
Add IP-addresses and MAC-addresses to add_host_metadata. 6878
Added a seccomp (secure computing) filter on Linux that whitelists the necessary system calls used by each Beat. 5213
Ship fields.yml as part of the binary 4834
Added options to dev-tools/cmd/dashboards/export_dashboard.go: -indexPattern to include index-pattern in output, -quiet to be quiet. 7101
Add Indexer indexing by pod uid. Enable pod uid metadata gathering in add_kubernetes_metadata. Extended Matcher log_path matching to support volume mounts 7072
Add default_fields to Elasticsearch template when connecting to Elasticsearch >= 7.0. 7015
Add support for loading a template.json file directly instead of using fields.yml. 7039
Add support for keyword multifields in field.yml. 7131
Add experimental Jolokia Discovery autodiscover provider. 7141
Add owner object info to Kubernetes metadata. 7231
Add Beat export dashboard command. 7239
Add support for docker autodiscover to monitor containers on host network 6708
Add ability to define input configuration as stringified JSON for autodiscover. 7372
Add processor definition support for hints builder 7386
Add support to disable html escaping in outputs. 7445
Refactor error handing in schema.Apply(). 7335
Add additional types to Kubernetes metadata 7457
Add module state reporting for Beats Monitoring. 7075
Release the rename processor as GA. 7656
Add support for Openstack Nova in add_cloud_metadata processor. 7663
Add support to set Beats services to automatic-delayed start on Windows. 8711

Auditbeat

Added XXH64 hash option for file integrity checks. 7311
Added the show auditd-rules and show auditd-status commands to show kernel rules and status. 7114
Add Kubernetes specs for auditbeat file integrity monitoring 7642

Filebeat

Add Kibana module with log fileset. 7052
Support MySQL 5.7.19 by mysql/slowlog 6969
Correctly join partial log lines when using docker input. 6967
Add support for TLS with client authentication to the TCP input 7056
Converted part of pipeline from treafik/access metricSet to dissect to improve efficiency. 7209
Add GC fileset to the Elasticsearch module. 7305
Add Audit log fileset to the Elasticsearch module. 7365
Add Slow log fileset to the Elasticsearch module. 7473
Add deprecation fileset to the Elasticsearch module. 7474
Add convert_timezone option to Kafka module to convert dates to UTC. 7546 7578
Add patterns for kafka 1.1 logs. 7608
Move debug messages in tcp input source 7712

Metricbeat

Add experimental Elasticsearch index metricset. 6881
Add dashboards and visualizations for haproxy metrics. 6934
Add Jolokia agent in proxy mode. 6475
Add message rates to the RabbitMQ queue metricset 6442 6606
Add exchanges metricset to the RabbitMQ module 6442 6607
Add Elasticsearch index_summary metricset. 6918
Add shard metricset to Elasticsearch module. 7006
Add apiserver metricset to Kubernetes module. 7059
Add maxmemory to redis info metricset. 7127
Set guest as default user in RabbitMQ module. 7107
Add postgresql statement metricset. 7048 7060
Update state_container metricset to support latest kube-state-metrics version. 7216
Add TLS support to MongoDB module. 7401
Added Traefik module with health metricset. 7413
Add Elasticsearch ml_job metricsets. 7196
Add support for bearer token files to HTTP helper. 7527
Add Elasticsearch index recovery metricset. 7225
Add locks, global_locks, oplatencies and process fields to status metricset of MongoDB module. 7613
Run Kafka integration tests on version 1.1.0 7616
Release raid and socket metricset from system module as GA. 7658
Release elasticsearch module and all its metricsets as beta. 7662
Release munin and traefik module as beta. 7660
Add envoyproxy module. 7569
Release prometheus collector metricset as GA. 7660
Add Elasticsearch cluster_stats metricset. 7638
Added basepath setting for HTTP-based metricsets 7700

Packetbeat

The process monitor now reports the command-line for all processes, under Linux and Windows. 7135
Updated the TLS protocol parser with new cipher suites added to TLS 1.3. 7455
Flows are enriched with process information using the process monitor. 7507
Added UDP support to process monitor. 7571

Deprecated

Metricbeat

Kubernetes state_container cpu.limit.nanocores and cpu.request.nanocores have been deprecated in favor of cpu.*.cores. 6916

Beats version 6.3.1

View commits

Bugfixes

Affecting all Beats

Allow index-pattern only setup when setup.dashboards.only_index=true. 7285
Preserve the event when source matching fails in add_docker_metadata. 7133
Negotiate Docker API version from our client instead of using a hardcoded one. 7165
Fix duplicating dynamic_fields in template when overwriting the template. 7352

Auditbeat

Fixed parsing of AppArmor audit messages. 6978

Filebeat

Comply with PostgreSQL database name format 7198
Optimize PostgreSQL ingest pipeline to use anchored regexp and merge multiple regexp into a single expression. 7269
Keep different registry entry per container stream to avoid wrong offsets. 7281
Fix offset field pointing at end of a line. 6514
Commit registry writes to stable storage to avoid corrupt registry files. 6792

Metricbeat

Fix field mapping for the system process CPU ticks fields. 7230
Ensure canonical naming for JMX beans is disabled in Jolokia module. 7047
Fix Jolokia attribute mapping when using wildcards and MBean names with multiple properties. 7321

Packetbeat

Fix an out of bounds access in HTTP parser caused by malformed request. 6997
Fix missing type for http.response.body field. 7169

Added

Auditbeat

Added caching of UID and GID values to auditd module. 6978
Updated syscall tables for Linux 4.16. 6978
Added better error messages for when the auditd module fails due to the Linux kernel not supporting auditing (CONFIG_AUDIT=n). 7012

Metricbeat

Collect accumulated docker network metrics and mark old ones as deprecated. 7253

Beats version 6.3.0

View commits

Breaking changes

Affecting all Beats

De dot keys of labels and annotations in kubernetes meta processors to prevent collisions. 6203
Rename beat.cpu..time metrics to beat.cpu..time.ms. 6449
Add host.name field to all events, to avoid mapping conflicts. This could be breaking Logstash configs if you rely on the host field being a string. 7051

Filebeat

Add validation for Stdin, when Filebeat is configured with Stdin and any other inputs, Filebeat will now refuse to start. 6463
Mark system.syslog.message and system.auth.message as text instead of keyword. 6589

Metricbeat

De dot keys in kubernetes/event metricset to prevent collisions. 6203
Add config option for windows/perfmon metricset to ignore non existent counters. 6432
Refactor docker CPU calculations to be more consistent with docker stats. 6608
Update logstash.node_stats metricset to write data under logstash.node.stats.*. 6714

Bugfixes

Affecting all Beats

Fix panic when Events containing a float32 value are normalized. 6129
Fix setup.dashboards.always_kibana when using Kibana 5.6. 6090
Fix for Kafka logger. 6430
Remove double slashes in Windows service script. 6491
Ensure Kubernetes labels/annotations don’t break mapping 6490
Ensure that the dashboard zip files can’t contain files outside of the kibana directory. 6921
Fix map overwrite panics by cloning shared structs before doing the update. 6947
Fix delays on autodiscovery events handling caused by blocking runner stops. 7170
Do not emit Kubernetes autodiscover events for Pods without IP address. 7235
Fix self metrics when containerized 6641

Auditbeat

Add hex decoding for the name field in audit path records. 6687
Fixed a deadlock in the file_integrity module under Windows. 6864
Fixed parsing of AppArmor audit messages. 6978
Allow auditbeat setup to run without requiring elevated privileges for the audit client. 7111
Fix goroutine leak that occurred when the auditd module was stopped. 7163

Filebeat

Fix panic when log prospector configuration fails to load. 6800
Fix memory leak in log prospector when files cannot be read. 6797
Add raw JSON to message field when JSON parsing fails. 6516
Commit registry writes to stable storage to avoid corrupt registry files. 6877
Fix a parsing issue in the syslog input for RFC3339 timestamp and time with nanoseconds. 7046
Fix an issue with an overflowing wait group when using the TCP input. 7202

Heartbeat

Fix race due to updates of shared a map, that was not supposed to be shared between multiple go-routines. 6616

Metricbeat

Fix the default configuration for Logstash to include the default port. 6279
Fix dealing with new process status codes in Linux kernel 4.14+. 6306
Add filtering option by exact device names in system.diskio. diskio.include_devices. 6085
Add connections metricset to RabbitMQ module 6548
Fix panic in http dependent modules when invalid config was used. 6205
Fix system.filesystem.used.pct value to match what df reports. 5494
Fix namespace disambiguation in Kubernetes state_* metricsets. 6281
Fix Windows perfmon metricset so that it sends metrics when an error occurs. 6542
Fix Kubernetes calculated fields store. https://github.com/elastic/beats/pull/6564{6564}
Exclude bind mounts in fsstat and filesystem metricsets. 6819
Don’t stop Metricbeat if aerospike server is down. 6874
disk reads and write count metrics in RabbitMQ queue metricset made optional. 6876
Add mapping for docker metrics per cpu. 6843

Winlogbeat

Fixed a crash under Windows 2003 and XP when an event had less insert strings than required by its format string. 6247

Added

Affecting all Beats

Update Golang 1.9.4 6326
Add the ability to log to the Windows Event Log. 5913
The node name can be discovered automatically by machine-id matching when beat deployed outside Kubernetes cluster. 6146
Panics will be written to the logger before exiting. 6199
Add builder support for autodiscover and annotations builder 6408
Add plugin support for autodiscover builders, providers 6457
Preserve runtime from container statuses in Kubernetes autodiscover 6456
Experimental feature setup.template.append_fields added. 6024
Add appender support to autodiscover 6469
Add add_host_metadata processor 5968
Retry configuration to load dashboards if Kibana is not reachable when the beat starts. 6560
Add has_fields conditional to filter events based on the existence of all the given fields. 6285 6653
Add support for spooling to disk to the beats event publishing pipeline. 6581
Added logging of system info at Beat startup. 5946
Do not log errors if X-Pack Monitoring is enabled but Elastisearch X-Pack is not. 6627
Add rename processor. 6292
Allow override of dynamic template match_mapping_type for fields with object_type. 6691

Filebeat

Add IIS module to parse access log and error log. 6127
Renaming of the prospector type to the input type and all prospectors are now moved to the input folder, to maintain backward compatibility type aliasing was used to map the old type to the new one. This change also affect YAML configuration. 6078
Addition of the TCP input 6700
Add option to convert the timestamps to UTC in the system module. 5647
Add Logstash module support for main log and the slow log, support the plain text or structured JSON format 5481
Add stream filtering when using docker prospector. 6057
Add support for CRI logs format. 5630
Add json.ignore_decoding_error config to not log json decoding erors. 6547
Make registry file permission configurable. 6455
Add MongoDB module. 6238
Add Ingest pipeline loading to setup. 6814
Add support of log_format combined to NGINX access logs. 6858
Release config reloading feature as GA.
Add support human friendly size for the UDP input. 6886
Add Syslog input to ingest RFC3164 Events via TCP and UDP 6842
Remove the undefined username option from the Redis input and clarify the documentation. 6662

Heartbeat

Made the URL field of Heartbeat aggregateable. 6263
Use match.Matcher for checking Heartbeat response bodies with regular expressions. 6539

Metricbeat

Support apache status pages for versions older than 2.4.16. 6450
Add support for huge pages on Linux. 6436
Support to optionally 'de dot' keys in http/json metricset to prevent collisions. 5970
Add graphite protocol metricbeat module. 4734
Add http server metricset to support push metrics via http. 4770
Make config object public for graphite and http server 4820
Add system uptime metricset. 4848
Add experimental queue metricset to RabbitMQ module. 4788
Add additional php-fpm pool status kpis for Metricbeat module 5287
Add etcd module. 4970
Add ip address of docker containers to event. 5379
Add ceph osd tree information to metricbeat 5498
Add ceph osd_df to metricbeat 5606
Add basic Logstash module. 5540
Add dashboard for Windows service metricset. 5603
Add pct calculated fields for Pod and container CPU and memory usages. 6158
Add statefulset support to Kubernetes module. 6236
Refactor prometheus endpoint parsing to look similar to upstream prometheus 6332
Making the http/json metricset GA. 6471
Add support for array in http/json metricset. 6480
Making the jolokia/jmx module GA. 6143
Making the MongoDB module GA. 6554
Allow to disable labels dedot in Docker module, in favor of a safe way to keep dots. 6490
Add experimental module to collect metrics from munin nodes. 6517
Add support for wildcards and explicit metrics grouping in jolokia/jmx. 6462
Set collector as default metricset in Prometheus module. 6636 6747
Set mntr as default metricset in Zookeeper module. 6674
Set default metricsets in vSphere module. 6676
Set status as default metricset in Apache module. 6673
Set namespace as default metricset in Aerospike module. 6669
Set service as default metricset in Windows module. 6675
Set all metricsets as default metricsets in uwsgi module. 6688
Allow autodiscover to monitor unexposed ports 6727
Mark kubernetes.event metricset as beta. 6715
Set all metricsets as default metricsets in couchbase module. 6683
Mark uwsgi module and metricset as beta. 6717
Mark Golang module and metricsets as beta. 6711
Mark system.raid metricset as beta. 6710
Mark http.server metricset as beta. 6712
Mark metricbeat logstash module and metricsets as beta. 6713
Set all metricsets as default metricsets in Ceph module. 6676
Set container, cpu, diskio, healthcheck, info, memory and network in docker module as default. 6718
Set cpu, load, memory, network, process and process_summary as default metricsets in system module. 6689
Set collector as default metricset in Dropwizard module. 6669
Set info and keyspace as default metricsets in redis module. 6742
Set connection as default metricset in rabbitmq module. 6743
Set all metricsets as default metricsets in Elasticsearch module. 6755
Set all metricsets as default metricsets in Etcd module. 6756
Set server metricsets as default in Graphite module. 6757
Set all metricsets as default metricsets in HAProxy module. 6758
Set all metricsets as default metricsets in Kafka module. 6759
Set all metricsets as default metricsets in postgresql module. 6761
Set status metricsets as default in Kibana module. 6762
Set all metricsets as default metricsets in Logstash module. 6763
Set container, node, pod, system, volume as default in Kubernetes module. https://github.com/elastic/beats/pull/ 6764[6764]
Set stats as default in memcached module. 6765
Set all metricsets as default metricsets in Mongodb module. 6766
Set pool as default metricset for php_fpm module. 6768
Set status as default metricset for mysql module. https://github.com/elastic/beats/pull/ 6769[6769]
Set stubstatus as default metricset for nginx module. 6770
Added support for haproxy 1.7 and 1.8. 6793
Add accumulated I/O stats to diskio in the line of docker stats. 6701
Ignore virtual filesystem types by default in system module. 6819
Release config reloading feature as GA. 6891
Kubernetes deployment: Add ServiceAccount config to system metricbeat. 6824
Kubernetes deployment: Add DNS Policy to system metricbeat. 6656

Packetbeat

Add support for condition on bool type 5659 5954
Fix high memory usage on HTTP body if body is not published. 6680
Allow to capture the HTTP request or response bodies independently. 6784
HTTP publishes an Error event for unmatched requests or responses. 6794

Winlogbeat

Use bookmarks to persist the last published event. 6150

Beats version 6.2.3

View commits

Breaking changes

Affecting all Beats

Fix conditions checking on autodiscover Docker labels. 6412

Bugfixes

Affecting all Beats

Avoid panic errors when processing nil Pod events in add_kubernetes_metadata. 6372
Fix infinite failure on Kubernetes watch 6504

Metricbeat

Fix Kubernetes overview dashboard views for non default time ranges. https://github.com/elastic/beats/issues/6395{6395}

Beats version 6.2.2

View commits

Bugfixes

Affecting all Beats

Add logging when monitoring cannot connect to Elasticsearch. 6365
Fix infinite loop when event unmarshal fails in Kubernetes pod watcher. 6353

Filebeat

Fix a conversion issue for time related fields in the Logstash module for the slowlog fileset. 6317

Beats version 6.2.1

View commits

No changes in this release.

Beats version 6.2.0

View commits

Breaking changes

Affecting all Beats

The log format may differ due to logging library changes. 5901
The default value for pipelining is reduced to 2 to avoid high memory in the Logstash beats input. 6250

Auditbeat

Split the audit.kernel and audit.file metricsets into their own modules named auditd and file_integrity, respectively. This change requires existing users to update their config. 5422
Renamed file_integrity module fields. 5423 5995
Renamed auditd module fields. 5423 6080

Metricbeat

Rename golang.heap.system.optained field to golang.heap.system.obtained. 5703
De dot keys in jolokia/jmx metricset to prevent collisions. 5957

Bugfixes

Auditbeat

Fixed an issue where the proctitle value was being truncated. 6080
Fixed an issue where values were incorrectly interpreted as hex data. 6080
Fixed parsing of the key value when multiple keys are present. 6080
Fix possible resource leak if file_integrity module is used with config reloading on Windows or Linux. 6198

Filebeat

Fix variable name for convert_timezone in the system module. 5936

Metricbeat

Fix error datastore '*' not found in Vsphere module. 4879
Fix error NotAuthenticated in Vsphere module. 4673
Fix mongodb session consistency mode to allow command execution on secondary nodes. 4689
Fix kubernetes state_pod status.phase so that the active phase is returned instead of unknown. 5980
Fix error collecting network_names in Vsphere module. 5962
Fix process cgroup memory metrics for memsw, kmem, and kmem_tcp. 6033
Fix kafka OffsetFetch request missing topic and partition parameters. 5880

Packetbeat

Fix mysql SQL parser to trim \r from Windows Server SELECT\r\n\t1. 5572

Added

Affecting all Beats

Adding a local keystore to allow user to obfuscate password 5687
Add autodiscover for kubernetes. 6055
Add Beats metrics reporting to Xpack. 3422
Update the command line library cobra and add support for zsh completion 5761
Update to Golang 1.9.2
Moved ip_port indexer for add_kubernetes_metadata to all beats. 5707
ip_port indexer now index both IP and IP:port pairs. 5721
Add the ability to write structured logs. 5901
Use structured logging for the metrics that are periodically logged via the logging.metrics feature. 5915
Improve Elasticsearch output metrics to count number of dropped and duplicate (if event ID is given) events. 5811
Add the ability for the add_docker_metadata process to enrich based on process ID. 6100
The add_docker_metadata and add_kubernetes_metadata processors are now GA, instead of Beta. 6105
Update go-ucfg library to support top level key reference and cyclic key reference for the keystore 6098

Auditbeat

Auditbeat is marked as GA, no longer Beta. 5432
Add support for BLAKE2b hash algorithms to the file integrity module. 5926
Add support for recursive file watches. 5575 5833

Filebeat

Add Osquery module. 5971
Add stream filtering when using docker prospector. 6057

Metricbeat

Add ceph osd_df to metricbeat 5606
Add field network_names of hosts and virtual machines. 5646
Add experimental system/raid metricset. 5642
Add a dashboard for the Nginx module. 5991
Add experimental mongodb/collstats metricset. 5852
Update the MySQL dashboard to use the Time Series Visual Builder. 5996
Add experimental uwsgi module. 6006
Docker and Kubernetes modules are now GA, instead of Beta. 6105
Support haproxy stats gathering using http (additionally to tcp socket). 5819
Support to optionally 'de dot' keys in http/json metricset to prevent collisions. 5957

Packetbeat

Configure good defaults for add_kubernetes_metadata. 5707

Beats version 6.1.3

View commits

No changes in this release.

Beats version 6.1.2

View commits

Bugfixes

Auditbeat

Add an error check to the file integrity scanner to prevent a panic when there is an error reading file info via lstat. 6005

Added

Filebeat

Switch to docker prospector in sample manifests for Kubernetes deployment 5963

Beats version 6.1.1

View commits

No changes in this release.

Beats version 6.1.0

View commits

Breaking changes

Auditbeat

Changed audit.file.path to be a multi-field so that path is searchable. 5625

Metricbeat

Rename heap_init field to heap.init in the Elasticsearch module. 5320
Rename http.response.status_code field to http.response.code in the HTTP module. 5521

Bugfixes

Affecting all Beats

Remove ID() from Runner interface 5153
Correctly send configured Host header to the remote server. 4842
Change add_kubernetes_metadata to attempt detection of namespace. 5482
Avoid double slash when join url and path 5517
Fix console color output for Windows. 5611
Fix logstash output debug message. https://github.com/elastic/beats/pull/5799{5799]
Fix isolation of modules when merging local and global field settings. 5795
Report ephemeral ID and uptime in monitoring events on all platforms 6501

Filebeat

Add support for adding string tags 5395
Fix race condition when limiting the number of harvesters running in parallel 5458
Fix relative paths in the prospector definitions. 5443
Fix recursive_globe.enabled option. 5443

Metricbeat

Change field type of http header from nested to object 5258
Fix the fetching of process information when some data is missing under MacOS X. 5337
Change MySQL active connections visualization title to MySQL total connections. 4812
Fix ProcState on Linux and FreeBSD when process names contain parentheses. 5775
Fix incorrect Mem.Used calculation under linux. 5775
Fix open_file_descriptor_count and max_file_descriptor_count lost in zookeeper module 5902
Fix system process metricset for kernel processes. 5700
Change kubernetes.node.cpu.allocatable.cores to float. 6130

Packetbeat

Fix http status phrase parsing not allow spaces. 5312
Fix http parse to allow to parse get request with space in the URI. 5495
Fix mysql SQL parser to trim \r from Windows Server SELECT\r\n\t1. 5572
Fix corruption when parsing repeated headers in an HTTP request or response. 6325
Fix panic when parsing partial AMQP messages. 6384
Fix out of bounds access to slice in MongoDB parser. 6256
Fix sniffer hanging on exit under Linux. 6535
Fix bounds check error in http parser causing a panic. 6750

Winlogbeat

Fix the registry file. It was not correctly storing event log names, and upon restart it would begin reading at the start of each event log. 5813
Fix config validation to allow event_logs.processors. [pull]6217[6217]

Added

Affecting all Beats

Support dashboard loading without Elasticsearch 5653
Changed the hashbang used in the beat helper script from /bin/bash to /usr/bin/env bash. 5051
Changed beat helper script to use exec when running the beat. 5051
Fix reloader error message to only print on actual error 5066
Add support for enabling TLS renegotiation. 4386
Add Azure VM support for add_cloud_metadata processor 5355
Add output.file.permission config option. 4638
Refactor add_kubernetes_metadata to support autodiscovery 5434
Improve custom flag handling and CLI flags usage message. 5543
Add number_of_routing_shards config set to 30 5570
Set log level for kafka output. 5397
Move TCP UDP start up into server.Start() 4903
Update to Golang 1.9.2

Auditbeat

Add support for SHA3 hash algorithms to the file integrity module. 5345
Add dashboards for Linux audit framework events (overview, executions, sockets). 5516

Filebeat

Add PostgreSQL module with slowlog support. 4763
Add Kafka log module. 4885
Add support for /var/log/containers/ log path in add_kubernetes_metadata processor. 4981
Remove error log from runnerfactory as error is returned by API. 5085
Add experimental Docker json-file prospector . 5402
Add experimental Docker autodiscover functionality. 5245
Add option to convert the timestamps to UTC in the system module. 5647
Add Logstash module support for main log and the slow log, support the plain text or structured JSON format 5481

Metricbeat

Add graphite protocol metricbeat module. 4734
Add http server metricset to support push metrics via http. 4770
Make config object public for graphite and http server 4820
Add system uptime metricset. 4848
Add experimental queue metricset to RabbitMQ module. 4788
Add additional php-fpm pool status kpis for Metricbeat module 5287
Add etcd module. 4970
Add ip address of docker containers to event. 5379
Add ceph osd tree information to Metricbeat 5498
Add basic Logstash module. 5540
Add dashboard for Windows service metricset. 5603
Add experimental Docker autodiscover functionality. 5245
Add Windows service metricset in the windows module. 5332
Update gosigar to v0.6.0. 5775

Packetbeat

Add support for decoding the TLS envelopes. 5476
HTTP parses successfully on empty status phrase. 6176
HTTP parser supports broken status line. 6631

Beats version 6.0.1

View commits

Bugfixes

Affecting all Beats

Fix documentation links in README.md files. 5710
Fix add_docker_metadata dropping some containers. 5788

Heartbeat

Fix the "HTTP up status" visualization. 5564

Metricbeat

Fix map overwrite in docker diskio module. 5582
Fix connection leak in mongodb module. 5688
Fix the include top N processes feature for cases where there are fewer processes than N. 5729

libbeat/docs/release-notes/6.0.0.asciidoc

Beats version 6.0.0-GA

View commits

The list below covers the changes between 6.0.0-rc2 and 6.0.0 GA only.

Bugfixes

Filebeat

Fix machine learning jobs setup for dynamic modules. 5509

Packetbeat

Fix missing length check in the PostgreSQL module. 5457
Fix panic in ACK handler if event is dropped on blocked queue 5524

Added

Filebeat

Add Kubernetes manifests to deploy Filebeat. 5349
Add container short ID matching to add_docker_metadata. 6172

Metricbeat

Add Kubernetes manifests to deploy Metricbeat. 5349

Beats version 6.0.0-rc2

View commits

Breaking changes

Packetbeat

Remove not-working runoptions.uid and runoptions.gid options in Packetbeat. 5261

Bugfixes

Affecting all Beats

Fix data race accessing watched containers. 5147
Do not require template if index change and template disabled 5319
Fix missing ACK in redis output. 5404

Filebeat

Fix default paths for redis 4.0.1 logs on macOS 5173
Fix Filebeat not starting if command line and modules configs are used together. 5376
Fix double @timestamp field when JSON decoding was used. 5436

Metricbeat

Use beat.name instead of beat.hostname in the Host Overview dashboard. 5340
Fix the loading of 5.x dashboards. 5277

Added

Metricbeat

Auto-select a hostname (based on the host on which the Beat is running) in the Host Overview dashboard. 5340

Deprecated

Filebeat

The filebeat.config_dir option is deprecated. Use filebeat.config.prospector options instead. 5321

Beats version 6.0.0-rc1

View commits

Bugfixes

Affecting all Beats

Fix the /usr/bin/beatname script to accept -d "*" as a parameter. 5040
Combine fields.yml properties when they are defined in different sources. 5075
Keep Docker & Kubernetes pod metadata after container dies while they are needed by processors. 5084
Fix fields.yml lookup when using export template with a custom path.config param. 5089
Remove runner creation from every reload check 5141
Fix add_kubernetes_metadata matcher registry lookup. 5159

Metricbeat

Fix a memory allocation issue where more memory was allocated than needed in the windows-perfmon metricset. 5035
Don’t start metricbeat if external modules config is wrong and reload is disabled 5053
The MongoDB module now connects on each fetch, to avoid stopping the whole Metricbeat instance if MongoDB is not up when starting. 5120
Fix kubernetes events module to be able to index time fields properly. 5093
Fixed cmd_set and cmd_get being mixed in the Memcache module. 5189

Added

Affecting all Beats

Enable flush timeout by default. 5150
Add @metadata.version to events send to Logstash. 5166

Auditbeat

Changed the number of shards in the default configuration to 3. 5095
Add support for receiving audit events using a multicast socket. 4850

Filebeat

Changed the number of shards in the default configuration to 3. 5095
Don’t start filebeat if external modules/prospectors config is wrong and reload is disabled 5053
Add filebeat.registry_flush setting, to delay the registry updates. 5146

Heartbeat

Changed the number of shards in the default configuration to 1. 5095

Packetbeat

Changed the number of shards in the default configuration to 3. 5095

Winlogbeat

Changed the number of shards in the default configuration to 3. 5095

Beats version 6.0.0-beta2

View commits

Breaking changes

Affecting all Beats

The log directory (path.log) for Windows services is now set to C:\ProgramData\[beatname]\logs. 4764
The _all field is disabled in Elasticsearch 6.0. This means that searching by individual words only work on text fields. 4901
Fail if removed setting output.X.flush_interval is explicitly configured.
Rename the /usr/bin/beatname.sh script (e.g. metricbeat.sh) to /usr/bin/beatname. 4933
Beat does not start if elasticsearch index pattern was modified but not the template name and pattern. 4769
Fail if removed setting output.X.flush_interval is explicitly configured. 4880

Bugfixes

Affecting all Beats

Register kubernetes field_format matcher and remove logger in Encode API 4888
Fix go plugins not loaded when beat starts 4799
Add support for initContainers in add_kubernetes_metadata processor. 4825
Eliminate deprecated default mapping in 6.x 4864
Fix pod name indexer to use both namespace, pod name to frame index key 4775

Filebeat

Fix issue where the fileset.module could have the wrong value. 4761

Heartbeat

Fix monitor.name being empty by default. 4852
Fix wrong event timestamps. 4851

Metricbeat

Added missing mongodb configuration file to the modules.d folder. 4870
Fix wrong MySQL CRUD queries timelion visualization 4857
Add new metrics to CPU metricset 4969

Packetbeat

Update flow timestamp on each packet being received. 4895

Added

Affecting all Beats

Add setting to enable/disable the slow start in logstash output. 4972
Update init scripts to use the test config subcommand instead of the deprecated -configtest flag. 4600
Get by default the credentials for connecting to Kibana from the Elasticsearch output configuration. 4867
Added cloud.id and cloud.auth settings, for simplifying using Beats with the Elastic Cloud. 4959
Add lz4 compression support to kafka output. 4977
Add newer kafka versions to kafka output. 4977
Configure the index name when loading the dashboards and the index pattern. 4949

Metricbeat

Add filesystem.ignore_types to system module for ignoring filesystem types. 4685
Add support to exclude labels from kubernetes pod metadata. 4757

Beats version 6.0.0-beta1

View commits

Breaking changes

Affecting all Beats

Rename kubernetes processor to add_kubernetes_metadata. 4473
Rename .full.yml config files to .reference.yml. 4563
The scripts/import_dashboards is removed from packages. Use the setup command instead. 4586
Change format of the saved kibana dashboards to have a single JSON file for each dashboard 4413
Rename configtest command to test config. 4590
Remove setting queue_size and bulk_queue_size. 4650
Remove setting dashboard.snapshot and dashboard.snapshot_url. They are no longer needed because the dashboards are included in the packages by default. 4675
Beats can no longer be launched from Windows Explorer (GUI), command line is required. 4420

Auditbeat

Changed file metricset config to make file.paths a list instead of a dictionary. 4796

Heartbeat

Renamed the heartbeat RPM/DEB name to heartbeat-elastic. 4601

Metricbeat

Change all system.cpu.*.pct metrics to be scaled by the number of CPU cores. This will make the CPU usage percentages from the system cpu metricset consistent with the system process metricset. The documentation for these metrics already stated that on multi-core systems the percentages could be greater than 100%. 4544
Remove filters setting from metricbeat modules. 4699
Added type field to filesystem metrics. 4717

Packetbeat

Remove the already unsupported pf_ring sniffer option. 4608

Bugfixes

Affecting all Beats

Don’t stop with error loading the ES template if the ES output is not enabled. 4436
Fix race condition in internal logging rotator. 4519
Normalize all times to UTC to ensure proper index naming. 4569
Fix issue with loading dashboards to ES 6.0 when .kibana index did not already exist. 4659

Auditbeat

Fix file.max_file_size config option for the audit file metricset. 4796

Filebeat

Fix issue where the fileset.module could have the wrong value. 4761

Metricbeat

Fix issue affecting Windows services timing out at startup. 4491
Fix incorrect docker.diskio.total metric calculation. 4507
Vsphere module: used memory field corrected. 4461

Packetbeat

Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing. 4442

Winlogbeat

Removed validation of top-level config keys. This behavior was inconsistent with other Beats and caused maintainability issues. 4657

Added

Affecting all Beats

New cli subcommands interface. 4420
Allow source path matching in add_docker_metadata processor. 4495
Add support for analyzers and multifields in fields.yml. 4574
Add support for JSON logging. 4523
Add test output command, to test Elasticsearch and Logstash output settings. 4590
Introduce configurable event queue settings: queue.mem.events, queue.mem.flush.min_events and queue.mem.flush.timeout. 4650
Enable pipelining in Logstash output by default. 4650
Added 'result' field to Elasticsearch QueryResult struct for compatibility with 6.x Index and Delete API responses. {issue]4661[4661]
The sample dashboards are now included in the Beats packages. 4675
Add pattern option to be used in the fields.yml to specify the pattern for a number field. 4731

Auditbeat

Added file.hash_types config option for controlling the hash types. 4796
Added the ability to specify byte unit suffixes to file.max_file_size. 4796

Filebeat

Add experimental Redis module. 4441
Nginx module: use the first not-private IP address as the remote_ip. 4417
Load Ingest Node pipelines when the Elasticsearch connection is established, instead of only once at startup. 4479
Add support for loading Xpack Machine Learning configurations from the modules, and added sample configurations for the Nginx module. 4506 4609
Add udp prospector type. 4452
Enabled Cgo which means libc is dynamically compiled. 4546
Add Beta module config reloading mechanism 4566
Remove spooler and publisher components and settings. 4644

Heartbeat

Enabled Cgo which means libc is dynamically compiled. 4546

Metricbeat

Add random startup delay to each metricset to avoid the thundering herd problem. 4010
Add the ability to configure audit rules to the kernel module. 4482
Add the ability to configure kernel’s audit failure mode. 4516
Add experimental Aerospike module. 4560
Vsphere module: collect custom fields from virtual machines. 4464
Add test modules command, to test modules expected output. 4656
Add processors setting to metricbeat modules. 4699
Support npipe protocol (Windows) in Docker module. 4751

Winlogbeat

Add the ability to use LevelRaw if Level isn’t populated in the event XML. 4257

Auditbeat

Add file integrity metricset to the audit module. 4486

Beats version 6.0.0-alpha2

View commits

Breaking changes

Filebeat

Rename input_type field to prospector.type 4294
The @metadata.type field, added by the Logstash output, is now hardcoded to doc and will be removed in future versions. 4331.

Bugfixes

Affecting all Beats

Fix importing the dashboards when the limit for max open files is too low. 4244
Fix configuration documentation for kubernetes processor 4313
Fix misspelling in add_locale configuration option for abbreviation.

Filebeat

Fix race condition on harvester stopping with reloading enabled. 3779
Fix recursive glob config parsing and resolution across restarts. 4269
Allow string characters in user agent patch version (NGINX and Apache) 4415
Fix grok pattern in filebeat module system/auth without hostname. 4224

Metricbeat

Set correct format for percent fields in memory module. 4619
Fix a debug statement that said a module wrapper had stopped when it hadn’t. 4264
Use MemAvailable value from /proc/meminfo on Linux 3.14. 4316
Fix panic when events were dropped by filters. 4327
Add filtering to system filesystem metricset to remove relative mountpoints like those from Linux network namespaces. 4370
Remove unnecessary print statement in schema apis. 4355
Fix type of field haproxy.stat.check.health.last. 4407

Packetbeat - Enable memcache filtering only if a port is specified in the config file. 4335 - Enable memcache filtering only if a port is specified in the config file. 4335

Added

Affecting all Beats

Upgraded to Golang 1.8.3. 4401
Added the possibility to set Elasticsearch mapping template settings from the Beat configuration file. 4284 4317
Add a variable to the SysV init scripts to make it easier to change the user. 4340
Add the option to write the generated Elasticsearch mapping template into a file. 4323
Add instance_name in GCE add_cloud_metadata processor. 4414
Add add_docker_metadata processor. 4352
Add logging.files permissions option. 4295

Filebeat - Added ability to sort harvested files. 4374 - Add experimental Redis slow log prospector type. 4180

Metricbeat

Add macOS implementation of the system diskio metricset. 4144
Add process_summary metricset that records high level metrics about processes. 4231
Add kube-state-metrics based metrics to kubernetes module 4253
Add debug logging to Jolokia JMX metricset. 4341
Add events metricset for kubernetes metricbeat module 4315
Change Metricbeat default configuration file to be better optimized for most users. 4329
Add experimental RabbitMQ module. 4394
Add Kibana dashboard for the Kubernetes modules. 4138

Packetbeat

Winlogbeat

Deprecated

Affecting all Beats

The @metadata.type field, added by the Logstash output, is deprecated, hardcoded to doc and will be removed in future versions. 4331.

Filebeat

Deprecate input_type prospector config. Use type config option instead. 4294

Known Issue

If the Elasticsearch output is not enabled, but setup.template options are present (like it’s the case in the default Metricbeat configuration), the Beat stops with an error: "Template loading requested but the Elasticsearch output is not configured/enabled". To avoid this error, disable the template loading explicitly setup.template.enabled: false.

Beats version 6.0.0-alpha1

View commits

Breaking changes

Affecting all Beats

Introduce beat version in the Elasticsearch index and mapping template 3527
Usage of field _type is now ignored and hardcoded to doc. 3757
Change vendor manager from glide to govendor. 3851
Rename error field to error.message. 3987
Change dashboards. config options to setup.dashboards.. 3921
Change outputs.elasticsearch.template.* to `setup.template.* 4080

Filebeat

Remove code to convert states from 1.x. 3767
Remove deprecated config options force_close_files and close_older. 3768
Change clean_removed behaviour to also remove states for files which cannot be found anymore under the same name. 3827
Remove document_type config option. Use fields instead. 4204
Move json_error under error.message and error.key. 4167

Packetbeat

Remove deprecated geoip. 3766
Replace waitstop command line argument by shutdown_timeout in configuration file. 3588

Winlogbeat

Remove metrics endpoint. Replaced by http endpoint in libbeat (see #3717). 3901

Bugfixes

Affecting all Beats

Add _id, _type, _index and _score fields in the generated index pattern. 3282

Filebeat

Fix the Mysql slowlog parsing of IP addresses. 4183
Fix issue that new prospector was not reloaded on conflict 4128

Heartbeat

Use IP type of elasticsearch for ip field. 3926

Metricbeat

Support common.Time in mapstriface.toTime() 3812
Fix MongoDB dbstats fields mapping. 4025
Fixing prometheus collector to aggregate metrics based on metric family. 4075
Fixing multiEventFetch error reporting when no events are returned 4153

Added

Affecting all Beats

Initialize a beats UUID from file on startup. 3615
Add new add_locale processor to export the local timezone with an event. 3902
Add http endpoint. 3717
Updated to Go 1.8.1. 4033
Add kubernetes processor 3888
Add support for include_labels and include_annotations in kubernetes processor 4043
Support new index_patterns field when loading templates for Elasticsearch >= 6.0 4056
Adding goimports support to make check and fmt 4114
Make kubernetes indexers/matchers pluggable 4151
Abstracting pod interface in kubernetes plugin to enable easier vendoring 4152

Filebeat

Restructure input.Event to be inline with outputs.Data 3823
Add base for supporting prospector level processors 3853
Add filebeat.config.path as replacement for config_dir. 4051
Add a recursive_glob.enabled setting to expand ** in patterns. 3980
Add Icinga module. 3904
Add ability to parse nginx logs exposing the X-Forwarded-For header instead of the remote address.

Heartbeat

Event format and field naming changes in Heartbeat and sample Dashboard. 4091

Metricbeat

Add experimental metricset perfmon to Windows module. 3758
Add memcached module with stats metricset. 3693
Add the process.cmdline.cache.enabled config option to the System Process Metricset. 3891
Add new MetricSet interfaces for developers (Closer, ReportingFetcher, and PushMetricSet). 3908
Add kubelet module 3916
Add dropwizard module 4022
Adding query APIs for metricsets and modules from metricbeat registry 4102
Fixing nil pointer on prometheus collector when http response is nil 4119
Add http module with json metricset. 4092
Add the option to the system module to include only the first top N processes by CPU and memory. 4127.
Add experimental Vsphere module. 4028
Add experimental Elasticsearch module. 3903
Add experimental Kibana module. 3895
Move elasticsearch metricset node_stats under node.stats namespace. 4142
Make IP port indexer constructor public 4434

Packetbeat

Add fields and fields_under_root to Packetbeat protocols configurations. 3518
Add list style Packetbeat protocols configurations. This change supports specifying multiple configurations of the same protocol analyzer. 3518

Winlogbeat

Deprecated

Affecting all Beats

Usage of field _type is deprecated. It should not be used in queries or dashboards. 3409

Packetbeat

Deprecate dictionary style protocols configuration. 3518

Winlogbeat

Known Issue

Filebeat

Prospector reloading only works properly with new files. 3546

Beats version 5.6.2

View commits

No changes in this release.

Beats version 5.6.1

View commits

No changes in this release.

Beats version 5.6.0

View commits

Breaking changes

Affecting all Beats

The _all.norms setting in the Elasticsearch template is no longer disabled. This increases the storage size with one byte per document, but allows for a better upgrade experience to 6.0. 4901

Bugfixes

Filebeat

Fix issue where the fileset.module could have the wrong value. 4761

Packetbeat

Update flow timestamp on each packet being received. 4895

Metricbeat

Fix a debug statement that said a module wrapper had stopped when it hadn’t. 4264
Use MemAvailable value from /proc/meminfo on Linux 3.14. 4316
Fix panic when events were dropped by filters. 4327

Added

Affecting all Beats

Add option to the import_dashboards script to load the dashboards via Kibana API. 4682

Filebeat

Add support for loading Xpack Machine Learning configurations from the modules, and added sample configurations for the Nginx module. 4506 4609
Add ability to parse nginx logs exposing the X-Forwarded-For header instead of the remote address. 4351

Metricbeat

Add filesystem.ignore_types to system module for ignoring filesystem types. 4685

Deprecated

Affecting all Beats

Loading more than one output is deprecated and will be removed in 6.0. 4907

Beats version 5.5.3

View commits

No changes in this release.

Beats version 5.5.2

View commits

No changes in this release.

Beats version 5.5.1

View commits

Bugfixes

Affecting all Beats

Normalize all times to UTC to ensure proper index naming. 4569

Beats version 5.5.0

View commits

Breaking changes

Affecting all Beats

Usage of field _type is now ignored and hardcoded to doc. 3757

Metricbeat - Change all system.cpu.*.pct metrics to be scaled by the number of CPU cores. This will make the CPU usage percentages from the system cpu metricset consistent with the system process metricset. The documentation for these metrics already stated that on multi-core systems the percentages could be greater than 100%. 4544

Bugfixes

Affecting all Beats

Fix console output. 4045

Filebeat

Allow string characters in user agent patch version (NGINX and Apache) 4415

Metricbeat

Fix type of field haproxy.stat.check.health.last. 4407

Packetbeat

Fix packetbeat.interface options that contain underscores (e.g. with_vlans or bpf_filter). 4378
Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing. 4442

Deprecated

Filebeat

Deprecate document_type prospector config option as _type is removed in elasticsearch 6.0. Use fields instead. 4225

Winlogbeat

Deprecated metrics endpoint. It is superseded by a libbeat feature that can serve metrics on an HTTP endpoint. 4145

Beats version 5.4.2

View commits

Bugfixes

Affecting all Beats

Removed empty sections from the template files, causing indexing errors for array objects. 4488

Metricbeat

Fix issue affecting Windows services timing out at startup. 4491
Add filtering to system filesystem metricset to remove relative mountpoints like those from Linux network namespaces. 4370

Packetbeat

Clean configured geoip.paths before attempting to open the database. 4306

Beats version 5.4.1

View commits

Bugfixes

Affecting all Beats

Fix importing the dashboards when the limit for max open files is too low. 4244
Fix console output. 4045

Filebeat

Fix issue that new prospector was not reloaded on conflict. 4128
Fix grok pattern in filebeat module system/auth without hostname. 4224
Fix the Mysql slowlog parsing of IP addresses. 4183

Added

Affecting all Beats

Binaries upgraded to Go 1.7.6 which contains security fixes. 4400

Winlogbeat

Add the ability to use LevelRaw if Level isn’t populated in the event XML. 4257

Beats version 5.4.0

View commits

Bugfixes

Affecting all Beats

Improve error message when downloading the dashboards fails. 3805
Fix potential Elasticsearch output URL parsing error if protocol scheme is missing. 3671
Downgrade Elasticsearch per batch item failure log to debug level. 3953
Make @timestamp accessible from format strings. 3721

Filebeat

Allow log lines without a program name in the Syslog fileset. 3944
Don’t stop Filebeat when modules are used with the Logstash output. 3929

Metricbeat

Fixing panic on the Prometheus collector when label has a comma. 3947
Make system process metricset honor the cpu_ticks config option. 3590

Winlogbeat

Fix null terminators include in raw XML string when include_xml is enabled. 3943

Added

Affecting all Beats

Update index mappings to support future Elasticsearch 6.X. 3778

Filebeat

Add auditd module for reading audit logs on Linux. 3750 3941
Add fileset for the Linux authorization logs. 3669

Heartbeat

Add default ports in HTTP monitor. 3924

Metricbeat

Add beta Jolokia module. 3844
Add dashboard for the MySQL module. 3716
Module configuration reloading is now beta instead of experimental. 3841
Marked http fields from the HAProxy module optional to improve compatibility with 1.5. 3788
Add support for custom HTTP headers and TLS for the Metricbeat modules. 3945

Packetbeat

Add DNS dashboard for an overview the DNS traffic. 3883
Add DNS Tunneling dashboard to highlight domains with large numbers of subdomains or high data volume. 3884

Beats version 5.3.2

View commits

Bugfixes

Filebeat

Properly shut down crawler in case one prospector is misconfigured. 4037
Fix panic in JSON decoding code if the input line is "null". 4042

Beats version 5.3.1

View commits

Bugfixes

Affecting all Beats

Fix panic when testing regex-AST to match against date patterns. 3889
Fix panic due to race condition in kafka output. 4098

Filebeat

Fix modules default file permissions. 3879
Allow - in Apache access log byte count. 3863

Metricbeat

Avoid errors when some Apache status fields are missing. 3074

Beats version 5.3.0

View commits

Breaking changes

Affecting all Beats

Configuration files must be owned by the user running the Beat or by root, and they must not be writable by others. 3544 3689
Change Beat generator. Use $GOPATH/src/github.com/elastic/beats/script/generate.py to generate a beat. 3452

Filebeat

Always use absolute path for event and registry. This can lead to issues when relative paths were used before. 3328

Metricbeat

Linux cgroup metrics are now enabled by default for the system process metricset. The configuration option for the feature was renamed from cgroups to process.cgroups.enabled. 3519
Change field names couchbase.node.couch..actual_disk_size. to couchbase.node.couch..disk_size. 3545

Bugfixes

Affecting all Beats

Add _id, _type, _index and _score fields in the generated index pattern. 3282

Filebeat - Always use absolute path for event and registry. 3328 - Raise an exception in case there is a syntax error in one of the configuration files available under filebeat.config_dir. 3573 - Fix empty registry file on machine crash. 3537

Metricbeat

Add error handling to system process metricset for when Linux cgroups are missing from the kernel. 3692
Add labels to the Docker healthcheck metricset output. 3707

Winlogbeat

Fix handling of empty strings in event_data. 3705

Added

Affecting all Beats

Files created by Beats (logs, registry, file output) will have 0600 permissions. 3387.
RPM/deb packages will now install the config file with 0600 permissions. 3382
Add the option to pass custom HTTP headers to the Elasticsearch output. 3400
Unify regexp and contains conditionals, for both to support array of strings and convert numbers to strings if required. 3469
Add the option to load the sample dashboards during the Beat startup phase. 3506
Disabled date detection in Elasticsearch index templates. Date fields must be explicitly defined in index templates. 3528
Using environment variables in the configuration file is now GA, instead of experimental. 3525

Filebeat

Add Filebeat modules for system, apache2, mysql, and nginx. 3159
Add the pipeline config option at the prospector level, for configuring the Ingest Node pipeline ID. 3433
Update regular expressions used for matching file names or lines (multiline, include/exclude functionality) to new matchers improving performance of simple string matches. 3469
The symlinks and harvester_limit settings are now GA, instead of experimental. 3525
close_timeout is also applied when the output is blocking. 3511
Improve handling of different path variants on Windows. 3781
Add multiline.flush_pattern option, for specifying the 'end' of a multiline pattern 4019

Heartbeat

Add tags, fields and fields_under_root in monitors configuration. 3623

Metricbeat

Add experimental dbstats metricset to MongoDB module. 3228
Use persistent, direct connections to the configured nodes for MongoDB module. 3228
Add dynamic configuration reloading for modules. 3281
Add docker health metricset 3357
Add docker image metricset 3467
System module uses new matchers for white-listing processes. 3469
Add Beta CEPH module with health metricset. 3311
Add Beta php_fpm module with pool metricset. 3415
The Docker, Kafka, and Prometheus modules are now Beta, instead of experimental. 3525
The HAProxy module is now GA, instead of experimental. 3525
Add the ability to collect the environment variables from system processes. 3337

Deprecated

Affecting all Beats

Usage of field _type is deprecated. It should not be used in queries or dashboards. 3409

Filebeat

The experimental publish_async option is now deprecated and is planned to be removed in 6.0. 3525

Beats version 5.2.2

View commits

Metricbeat

Fix bug docker module hanging when docker container killed. 3610
Set timeout to period instead of 1s by default as documented. 3612

Beats version 5.2.1

View commits

Bugfixes

Metricbeat

Fix go routine leak in docker module. 3492

Packetbeat

Fix error in the NFS sample dashboard. 3548

Winlogbeat

Fix error in the Winlogbeat sample dashboard. 3548

Beats version 5.2.0

View commits

Bugfixes

Affecting all Beats

Fix overwriting explicit empty config sections. 2918

Filebeat

Fix alignment issue were Filebeat compiled with Go 1.7.4 was crashing on 32 bits system. 3273

Metricbeat

Fix service times-out at startup. 3056
Kafka module case sensitive host name matching. 3193
Fix interface conversion panic in couchbase module 3272

Packetbeat

Fix issue where some Cassandra visualizations were showing data from all protocols. 3314

Added

Affecting all Beats

Add support for passing list and dictionary settings via -E flag.
Support for parsing list and dictionary setting from environment variables.
Added new flags to import_dashboards (-cacert, -cert, -key, -insecure). 3139 3163
The limit for the number of fields is increased via the mapping template. 3275
Updated to Go 1.7.4. 3277
Added a NOTICE file containing the notices and licenses of the dependencies. 3334.

Heartbeat

First release, containing monitors for ICMP, TCP, and HTTP.

Filebeat

Add enabled config option to prospectors. 3157
Add target option for decoded_json_field. 3169

Metricbeat

Kafka module broker matching enhancements. 3129
Add a couchbase module with metricsets for node, cluster and bucket. 3081
Export number of cores for CPU module. 3192
Experimental Prometheus module. 3202
Add system socket module that reports all TCP sockets. 3246
Kafka consumer groups metricset. 3240
Add jolokia module with dynamic jmx metricset. 3570

Winlogbeat

Reduced amount of memory allocated while reading event log records. 3113 3118

Beats version 5.1.2

View commits

Bugfixes

Filebeat

Fix registry migration issue from old states where files were only harvested after second restart. 3322

Packetbeat

Fix error on importing dashboards due to colons in the Cassandra dashboard. 3140
Fix error on importing dashboards due to the wrong type for the geo_point fields. 3147

Winlogbeat

Fix for "The array bounds are invalid" error when reading large events. 3076

Beats version 5.1.1

View commits

Breaking changes

Metricbeat

Change data structure of experimental haproxy module. 3003

Filebeat

If a file is falling under ignore_older during startup, offset is now set to end of file instead of 0. With the previous logic the whole file was sent in case a line was added and it was inconsistent with files which were harvested previously. 2907
tail_files is now only applied on the first scan and not for all new files. 2932

Bugfixes

Affecting all Beats

Fix empty benign errors logged by processor actions. 3046

Metricbeat

Calculate the fsstat values per mounting point, and not filesystem. 2777

Added

Affecting all Beats

Add add_cloud_metadata processor for collecting cloud provider metadata. 2728
Added decode_json_fields processor for decoding fields containing JSON strings. 2605
Add Tencent Cloud provider for add_cloud_metadata processor. 4023
Add Alibaba Cloud provider for add_cloud_metadata processor. 4111

Metricbeat

Add experimental Docker module. Provided by Ingensi and @douaejeouit based on dockbeat.
Add a sample Redis Kibana dashboard. 2916
Add support for MongoDB 3.4 and WiredTiger metrics. 2999
Add experimental kafka module with partition metricset. 2969
Add raw config option for mysql/status metricset. 3001
Add command fields for mysql/status metricset. 3251

Filebeat

Add command line option -once to run Filebeat only once and then close. 2456
Only load matching states into prospector to improve state handling 2840
Reset all states ttl on startup to make sure it is overwritten by new config 2840
Persist all states for files which fall under ignore_older to have consistent behaviour 2859
Improve shutdown behaviour with large number of files. 3035

Winlogbeat

Add event_logs.batch_read_size configuration option. 2641

Beats version 5.1.0 (skipped)

Version 5.1.0 doesn’t exist because, for a short period of time, the Elastic Yum and Apt repositories included unreleased binaries labeled 5.1.0. To avoid confusion and upgrade issues for the people that have installed these without realizing, we decided to skip the 5.1.0 version and release 5.1.1 instead.

Beats version 5.0.2

View commits

Bugfixes

Metricbeat

Fix the password option in the MongoDB module. 2995

Beats version 5.0.1

View commits

Bugfixes

Metricbeat

Fix system.process.start_time on Windows. 2848
Fix system.process.ppid on Windows. 2860
Fix system process metricset for Windows XP and 2003. cmdline will be unavailable. 1704
Fix access denied issues in system process metricset by enabling SeDebugPrivilege on Windows. 1897
Fix system diskio metricset for Windows XP and 2003. 2885

Packetbeat

Fix 'index out of bounds' bug in Packetbeat DNS protocol plugin. 2872

Filebeat

Fix registry cleanup issue when files falling under ignore_older after restart. 2818

Added

Metricbeat

Add username and password config options to the PostgreSQL module. 2890
Add username and password config options to the MongoDB module. 2889
Add system core metricset for Windows. 2883

Packetbeat

Define client_geoip.location as geo_point in the mappings to be used by the GeoIP processor in the Ingest Node pipeline. 2795

Filebeat

Stop Filebeat on registrar loading error. 2868

libbeat/docs/release-notes/5.0.0.asciidoc

Beats version 5.0.0-GA

View commits

The list below covers the changes between 5.0.0-rc1 and 5.0.0 GA only.

Bugfixes

Affecting all Beats

Fix kafka output re-trying batches with too large events. 2735
Fix kafka output protocol error if version: 0.10 is configured. 2651
Fix kafka output connection closed by broker on SASL/PLAIN. 2717

Metricbeat

Fix high CPU usage on macOS when encountering processes with long command lines. 2747
Fix high value of system.memory.actual.free and system.memory.actual.used. 2653
Change several OpenProcess calls on Windows to request the lowest possible access privilege. 1897
Fix system.memory.actual.free high value on Windows. 2653

Filebeat

Fix issue when clean_removed and clean_inactive were used together that states were not directly removed from the registry.
Fix issue where upgrading a 1.x registry file resulted in duplicate state entries. 2792

Added

Affecting all Beats

Add beat.version fields to all events.

Beats version 5.0.0-rc1

View commits

Breaking changes

Affecting all Beats

A dynamic mapping rule is added to the default Elasticsearch template to treat strings as keywords by default. 2688

Bugfixes

Affecting all Beats

Make sure Beats sent always float values when they are defined as float by sending 5.00000 instead of 5. 2627
Fix ignoring all fields from drop_fields in case the first field is unknown. 2685
Fix dynamic configuration int/uint to float type conversion. 2698
Fix primitive types conversion if values are read from environment variables. 2698

Metricbeat

Fix default configuration file on Windows to not enabled the load metricset. 2632

Packetbeat

Fix the bpf_filter setting. 2660

Filebeat

Fix input buffer on encoding problem. 2416

Deprecated

Affecting all Beats

Setting port has been deprecated in Redis and Logstash outputs. 2620

Beats version 5.0.0-beta1

View commits

Breaking changes

Affecting all Beats

Change Elasticsearch output index configuration to be based on format strings. If index has been configured, no date will be appended anymore to the index name. 2119
Replace output.kafka.use_type by output.kafka.topic accepting a format string. 2188
If the path specified by the -c flag is not absolute and -path.config is not specified, it is considered relative to the current working directory. 2245
rename tls configurations section to ssl. 2330
rename certificate_key configuration to key. 2330
replace tls.insecure with ssl.verification_mode setting. 2330
replace tls.min/max_version with ssl.supported_protocols setting requiring full protocol name. 2330

Metricbeat

Change field type system.process.cpu.start_time from keyword to date. 1565
redis/info metricset fields were renamed up according to the naming conventions.

Packetbeat

Group HTTP fields under http.request and http.response 2167
Export http.request.body and http.response.body when configured under include_body_for 2167
Move ignore_outgoing config to packetbeat.ignore_outgoing 2393

Filebeat

Set close_inactive default to 5 minutes (was 1 hour before)
Set clean_removed and close_removed to true by default

Bugfixes

Affecting all Beats

Fix logstash output handles error twice when asynchronous sending fails. 2441
Fix Elasticsearch structured error response parsing error. 2229
Fixed the run script to allow the overriding of the configuration file. 2171
Fix logstash output crash if no hosts are configured. 2325
Fix array value support in -E CLI flag. 2521
Fix merging array values if -c CLI flag is used multiple times. 2521
Fix beats failing to start due to invalid duplicate key error in configuration file. 2521
Fix panic on non writable logging directory. 2571

Metricbeat

Fix module filters to work properly with drop_event filter. 2249

Packetbeat

Fix mapping for some Packetbeat flow metrics that were not marked as being longs. 2177
Fix handling of messages larger than the maximum message size (10MB). 2470

Filebeat

Fix processor failure in Filebeat when using regex, contain, or equals with the message field. 2178
Fix async publisher sending empty events 2455
Fix potential issue with multiple harvester per file on large file numbers or slow output 2541

Winlogbeat

Fix corrupt registry file that occurs on power loss by disabling file write caching. 2313

Added

Affecting all Beats

Add script to generate the Kibana index-pattern from fields.yml. 2122
Enhance Redis output key selection based on format string. 2169
Configurable Redis keys using filters and format strings. 2169
Add format string support to output.kafka.topic. 2188
Add output.kafka.topics for more advanced kafka topic selection per event. 2188
Add support for Kafka 0.10. 2190
Add SASL/PLAIN authentication support to kafka output. 2190
Make Kafka metadata update configurable. 2190
Add Kafka version setting (optional) enabling kafka broker version support. 2190
Add Kafka message timestamp if at least version 0.10 is configured. 2190
Add configurable Kafka event key setting. 2284
Add settings for configuring the kafka partitioning strategy. 2284
Add partitioner settings reachable_only to ignore partitions not reachable by network. 2284
Enhance contains condition to work on fields that are arrays of strings. 2237
Lookup the configuration file relative to the -path.config CLI flag. 2245
Re-write import_dashboards.sh in Golang. 2155
Update to Go 1.7. 2306
Log total non-zero internal metrics on shutdown. 2349
Add support for encrypted private key files by introducing ssl.key_passphrase setting. 2330
Add experimental symlink support with symlinks config 2478
Improve validation of registry file on startup.

Metricbeat

Use the new scaled_float Elasticsearch type for the percentage values. 2156
Add experimental cgroup metrics to the system/process MetricSet. 2184
Added a PostgreSQL module. 2253
Improve mapping by converting half_float to scaled_float and integers to long. 2430
Add experimental haproxy module. 2384
Add Kibana dashboard for cgroups data 2555

Packetbeat

Add Cassandra protocol analyzer to Packetbeat. 1959
Match connections with IPv6 addresses to processes 2254
Add IP address to -devices command output 2327
Add configuration option for the maximum message size. Used to be hard-coded to 10 MB. 2470

Filebeat

Introduce close_timeout harvester options 1926
Strip BOM from first message in case of BOM files 2351
Add harvester_limit option 2417

Deprecated

Affecting all Beats

Topology map is deprecated. This applies to the settings: refresh_topology_freq, topology_expire, save_topology, host_topology, password_topology, db_topology.

Beats version 5.0.0-alpha5

View commits

Breaking changes

Affecting all Beats

Rename the filters section to processors. 1944
Introduce the condition with when in the processor configuration. 1949
The Elasticsearch template is now loaded by default. 1993
The Redis output index setting is renamed to key. index still works but it’s deprecated. 2077
The undocumented file output index setting was removed. Use filename instead. 2077

Metricbeat

Create a separate metricSet for load under the system module and remove load information from CPU stats. 2101
Add system.load.norm.1, system.load.norm.5 and system.load.norm.15. 2101
Add threads fields to mysql module. 2484

Packetbeat

Set enabled ` in packetbeat.protocols.icmp configuration to true by default. 1988

Bugfixes

Affecting all Beats

Fix sync publisher PublishEvents return value if client is closed concurrently. 2046

Metricbeat

Do not send zero values when no value was present in the source. 1972

Filebeat

Fix potential data loss between Filebeat restarts, reporting unpublished lines as published. 2041
Fix open file handler issue. 2028 2020
Fix filtering of JSON events when using integers in conditions. 2038

Winlogbeat

Fix potential data loss between Winlogbeat restarts, reporting unpublished lines as published. 2041

Added

Affecting all Beats

Periodically log internal metrics. 1955
Add enabled setting to all output modules. 1987
Command line flag -c can be used multiple times. 1985
Add OR/AND/NOT to the condition associated with the processors. 1983
Add -E CLI flag for overwriting single config options via command line. 1986
Choose the mapping template file based on the Elasticsearch version. 1993
Check stdout being available when console output is configured. 2035

Metricbeat

Add pgid field to process information. https://github.com/elastic/beats/pull/ 2021[2021]

Packetbeat

Add enabled setting to Packetbeat protocols. 1988
Add enabled setting to Packetbeat network flows configuration. 1988

Filebeat

Introduce close_removed and close_renamed harvester options. 1600
Introduce close_eof harvester option. 1600
Add clean_removed and clean_inactive config option. 1600

Deprecated

Filebeat

Deprecate close_older option and replace it with close_inactive. 2051
Deprecate force_close_files option and replace it with close_removed and close_renamed. 1600

Beats version 5.0.0-alpha4

View commits

Breaking changes

Affecting all Beats

The topology_expire option of the Elasticsearch output was removed. 1907

Filebeat

Stop following symlink. Symlinks are now ignored: 1686

Bugfixes

Affecting all Beats

Reset backoff factor on partial ACK. 1803
Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. 1829
Fix logstash output with pipelining mode enabled not reconnecting. 1876
Empty configuration sections become merge-able with variables containing full path. 1900
Fix error message about required fields missing not printing the missing field name. 1900

Metricbeat

Fix the CPU values returned for each core. 1863

Packetbeat

Add missing nil-check to memcached GapInStream handler. 1162
Fix NFSv4 Operation returning the first found first-class operation available in compound requests. 1821
Fix TCP overlapping segments not being handled correctly. 1898

Winlogbeat

Fix issue with rendering forwarded event log records. 1891

Added

Affecting all Beats

Improve error message if compiling regular expression from config files fails. 1900
Compression support in the Elasticsearch output. 1835

Metricbeat

Add MongoDB module. 1837

Beats version 5.0.0-alpha3

View commits

Breaking changes

Affecting all Beats

All configuration settings under shipper: are moved to be top level configuration settings. I.e. shipper.name: becomes name: in the configuration file. 1570

Topbeat

Topbeat is replaced by Metricbeat.

Filebeat

The state for files which fall under ignore_older is not stored anymore. This has the consequence, that if a file which fell under ignore_older is updated, the whole file will be crawled.

Bugfixes

Winlogbeat

Adding missing argument to the "Stop processing" log message. 1590

Added

Affecting all Beats

Add conditions to generic filtering. 1623

Metricbeat

First public release, containing the following modules: apache, mysql, nginx, redis, system, and zookeeper.

Filebeat

The registry format was changed to an array instead of dict. The migration to the new format will happen automatically at the first startup. 1703

Deprecated

Affecting all Beats

The support for doing GeoIP lookups is deprecated and will be removed in version 6.0. 1601

Beats version 5.0.0-alpha2

View commits

Breaking changes

Affecting all Beats

On DEB/RPM installations, the binary files are now found under /usr/share/{{beat_name}}/bin, not in /usr/bin. 1385
The logs are written by default to self rotating files, instead of syslog. 1371
Remove deprecated host option from elasticsearch, logstash and redis outputs. 1474

Packetbeat

Configuration of redis topology support changed. 1353
Move all Packetbeat configuration options under the packetbeat namespace 1417

Filebeat

Default location for the registry file was changed to be data/registry from the binary directory, rather than .filebeat in the current working directory. This affects installations for zip/tar.gz/source, the location for DEB and RPM packages stays the same. 1373

Bugfixes

Affecting all Beats

Drain response buffers when pipelining is used by Redis output. 1353
Unterminated environment variable expressions in config files will now cause an error 1389
Fix issue with the automatic template loading when Elasticsearch is not available on Beat start. 1321
Fix bug affecting -cpuprofile, -memprofile, and -httpprof CLI flags 1415
Fix race when multiple outputs access the same event with logstash output manipulating event 1410 1428
Seed random number generator using crypto.rand package. https://github.com/elastic/beats/pull/1503{1503]
Fix beats hanging in -configtest 1213
Fix kafka log message output 1516

Filebeat

Improvements in registrar dealing with file rotation. 1281
Fix issue with JSON decoding where @timestamp or type keys with the wrong type could cause Filebeat to crash. 1378
Fix issue with JSON decoding where values having null as values could crash Filebeat. 1466
Multiline reader normalizing newline to use \n. 1552

Winlogbeat

Fix panic when reading messages larger than 32K characters on Windows XP and 2003. 1498
Fix panic that occurs when reading a large events on Windows Vista and newer. 1499

Added

Affecting all Beats

Add support for TLS to Redis output. 1353
Add SOCKS5 proxy support to Redis output. 1353
Failover and load balancing support in redis output. 1353
Multiple-worker per host support for redis output. 1353
Added ability to escape ${x} in config files to avoid environment variable expansion 1389
Configuration options and CLI flags for setting the home, data and config paths. 1373
Configuration options and CLI flags for setting the default logs path. 1437
Update to Go 1.6.2 1447
Add Elasticsearch template files compatible with Elasticsearch 2.x. 1501
Add scripts for managing the dashboards of a single Beat 1359

Packetbeat

Fix compile issues for OpenBSD. 1347

Topbeat

Updated elastic/gosigar version so Topbeat can compile on OpenBSD. 1403

Beats version 5.0.0-alpha1

View commits

Breaking changes

libbeat

Run function to start a Beat now returns an error instead of directly exiting. 771
The method signature of HandleFlags() was changed to allow returning an error 1249
Require braces for environment variable expansion in config files 1304

Packetbeat

Rename output fields in the dns package. Former flag recursion_allowed becomes recursion_available. 803 Former SOA field ttl becomes minimum. 803
The fully qualified domain names which are part of output fields values of the dns package now terminate with a dot. 803
Remove the count field from the exported event 1210

Topbeat

Rename proc.cpu.user_p with proc.cpu.total_p as it includes CPU time spent in kernel space 631
Remove count field from the exported fields 1207
Rename input top level config option to topbeat

Filebeat

Scalar values in used in the fields configuration setting are no longer automatically converted to strings. 1092
Count field was removed from event as not used in filebeat 778

Winlogbeat

The message_inserts field was replaced with the event_data field 1053
The category field was renamed to task to better align with the Windows Event Log API naming 1053
Remove the count field from the exported event 1218

Bugfixes

Affecting all Beats

Logstash output will not retry events that are not JSON-encodable 927

Packetbeat

Create a proper BPF filter when ICMP is the only enabled protocol 757
Check column length in pgsql parser. 565
Harden pgsql parser. 565

Topbeat

Fix issue with cpu.system_p being greater than 1 on Windows 1128

Filebeat

Stop filebeat if started without any prospectors defined or empty prospectors 644 647
Improve shutdown of crawler and prospector to wait for clean completion 720
Omit fields from Filebeat events when null 899

Winlogbeat

Added

Affecting all Beats

Update builds to Golang version 1.6
Add option to Elasticsearch output to pass http parameters in index operations 805
Improve Logstash and Elasticsearch backoff behavior. 927
Add experimental Kafka output. 942
Add config file option to configure GOMAXPROCS. 969
Improve shutdown handling in libbeat. 1075
Add fields and fields_under_root options under the shipper configuration 1092
Add the ability to use a SOCKS5 proxy with the Logstash output 823
The -configtest flag will now print "Config OK" to stdout on success 1249

Packetbeat

Change the DNS library used throughout the dns package to github.com/miekg/dns. 803
Add support for NFS v3 and v4. 1231
Add support for EDNS and DNSSEC. 1292

Topbeat

Add username to processes 845

Filebeat

Add the ability to set a list of tags for each prospector 1092
Add JSON decoding support 1143

Winlogbeat

Add caching of event metadata handles and the system render context for the wineventlog API 888
Improve config validation by checking for unknown top-level YAML keys. 1100
Add the ability to set tags, fields, and fields_under_root as options for each event log 1092
Add additional data to the events published by Winlogbeat. The new fields are activity_id, event_data, keywords, opcode, process_id, provider_guid, related_activity_id, task, thread_id, user_data, and version. 1053
Add event_id, level, and provider configuration options for filtering events 1218
Add include_xml configuration option for including the raw XML with the event 1218

Known issues

All Beats can hang or panic on shutdown if the next server in the pipeline (e.g. Elasticsearch or Logstash) is not reachable. 1319
When running the Beats as a service on Windows, you need to manually load the Elasticsearch mapping template. 1315
The ES template automatic load doesn’t work if Elasticsearch is not available when the Beat is starting. 1321

Beats version 1.3.1

View commits

Bugfixes

Filebeat

Fix a concurrent bug on filebeat startup with a large number of prospectors defined. 2509

Packetbeat

Fix description for the -I CLI flag. 2480

Winlogbeat

Fix corrupt registry file that occurs on power loss by disabling file write caching. 2313

Beats version 1.3.0

View commits

Deprecated

Filebeat

Undocumented support for following symlinks is deprecated. Filebeat will not follow symlinks in version 5.0. 1767

Bugfixes

Affecting all Beats

Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. 1829
Fix output modes backoff counter reset. 1803 1814 1818
Set logstash output default bulk_max_size to 2048. 1662
Seed random number generator using crypto.rand package. 1503
Check stdout being available when console output is configured. 2063

Packetbeat

Add missing nil-check to memcached GapInStream handler. 1162
Fix NFSv4 Operation returning the first found first-class operation available in compound requests. 1821
Fix TCP overlapping segments not being handled correctly. 1917

Added

Affecting all Beats

Updated to Go 1.7

Beats version 1.2.3

View commits

Bugfixes

Topbeat

Fix high CPU usage when using filtering under Windows. 1598

Filebeat

Fix rotation issue with ignore_older. 1528

Winlogbeat

Fix panic when reading messages larger than 32K characters on Windows XP and 2003. 1498

Added

Filebeat

Prevent file opening for files which reached ignore_older. 1649

Beats version 1.2.2

View commits

Bugfixes

Affecting all Beats

Fix race when multiple outputs access the same event with Logstash output manipulating event. 1410
Fix go-daemon (supervisor used in init scripts) hanging when executed over SSH. 1394

Filebeat

Improvements in registrar dealing with file rotation. 1281

Beats version 1.2.1

View commits

Breaking changes

Affecting all Beats

Require braces for environment variable expansion in config files 1304
Removed deprecation warning for the Redis output. 1282

Topbeat

Fixed name of the setting stats.proc to stats.process in the default configuration file. 1343
Fix issue with cpu.system_p being greater than 1 on Windows 1128

Added

Topbeat

Add username to processes 845

Beats version 1.2.0

View commits

Breaking changes

Filebeat

Default config for ignore_older is now infinite instead of 24h, means ignore_older is disabled by default. Use close_older to only close file handlers.

Bugfixes

Packetbeat

Split real_ip_header value when it contains multiple IPs 1241

Winlogbeat

Fix invalid event_id on Windows XP and Windows 2003 1227

Added

Affecting all Beats

Add ability to override configuration settings using environment variables 114
Libbeat now always exits through a single exit method for proper cleanup and control 736
Add ability to create Elasticsearch mapping on startup 639

Topbeat

Add the command line used to start processes 533

Filebeat

Add close_older configuration option to complete ignore_older 181

Beats version 1.1.2

View commits

Bugfixes

Filebeat

Fix registrar bug for rotated files 1010

Beats version 1.1.1

View commits

Bugfixes

Affecting all Beats

Fix logstash output loop hanging in infinite loop on too many output errors. 944
Fix critical bug in filebeat and winlogbeat potentially dropping events. 953

Beats version 1.1.0

View commits

Bugfixes

Affecting all Beats

Fix logging issue with file based output where newlines could be misplaced during concurrent logging 650
Reduce memory usage by separate queue sizes for single events and bulk events. 649 516
Set default default bulk_max_size value to 2048 628

Packetbeat

Fix setting direction to out and use its value to decide when dropping events if ignore_outgoing is enabled 557
Fix logging issue with file-based output where newlines could be misplaced during concurrent logging 650
Reduce memory usage by having separate queue sizes for single events and bulk events. 649 516
Set default bulk_max_size value to 2048 628
Fix logstash window size of 1 not increasing. 598

Packetbeat

Fix the condition that determines whether the direction of the transaction is set to "outgoing". Packetbeat uses the direction field to determine which transactions to drop when dropping outgoing transactions. 557
Allow PF_RING sniffer type to be configured using pf_ring or pfring 671

Filebeat

Set spool_size default value to 2048 628

Added

Affecting all Beats

Add include_fields and drop_fields as part of generic filtering 1120
Make logstash output compression level configurable. 630
Some publisher options refactoring in libbeat 684
Move event preprocessor applying GeoIP to packetbeat 772

Packetbeat

Add support for capturing DNS over TCP network traffic. 486 554

Topbeat

Group all CPU usage per core statistics and export them optionally if cpu_per_core is configured 496

Filebeat

Add multiline support for combining multiple related lines into one event. 461
Add exclude_lines and include_lines options for regexp based line filtering. 430
Add exclude_files configuration option. 563
Add experimental option to enable filebeat publisher pipeline to operate asynchronously 782

Winlogbeat

First public release of Winlogbeat

Beats version 1.0.1

Check 1.0.1 diff

Bugfixes

Filebeat

Fix force_close_files in case renamed file appeared very fast. 302

Packetbeat

Improve MongoDB message correlation. 377
Improve redis parser performance. 422
Fix panic on nil in redis protocol parser. 384
Fix errors redis parser when messages are split in multiple TCP segments. 402
Fix errors in redis parser when length prefixed strings contain sequences of CRLF. 402
Fix errors in redis parser when dealing with nested arrays. 402

Beats version 1.0.0

Check 1.0.0 diff

Breaking changes

Topbeat

Change proc type to process #138

Bugfixes

Affecting all Beats

Fix random panic on shutdown by calling shutdown handler only once. elastic/filebeat#204
Fix credentials are not send when pinging an elasticsearch host. elastic/filebeat#287

Filebeat

Fix problem that harvesters stopped reading after some time and filebeat stopped processing events #257
Fix line truncating by internal buffers being reused by accident #258
Set default ignore_older to 24 hours #282

Beats version 1.0.0-rc2

Check 1.0.0-rc2 diff

Breaking changes

Affecting all Beats

The shipper output field is renamed to beat.name. #285
Use of enabled as a configuration option for outputs (elasticsearch, logstash, etc.) has been removed. #264
Use of disabled as a configuration option for tls has been removed. #264
The -test command line flag was renamed to -configtest. #264
Disable geoip by default. To enable it uncomment in config file. #305

Filebeat

Removed utf-16be-bom encoding support. Support will be added with fix for #205
Rename force_close_windows_files to force_close_files and make it available for all platforms.

Bugfixes

Affecting all Beats

Disable logging to stderr after configuration phase. #276
Set the default file logging path when not set in config. #275
Fix bug silently dropping records based on current window size. elastic/filebeat#226
Fix direction field in published events. #300
Fix elasticsearch structured errors breaking error handling. #309

Packetbeat

Packetbeat will now exit if a configuration error is detected. #357
Fixed an issue handling DNS requests containing no questions. #369

Topbeat

Fix leak of Windows handles. #98
Fix memory leak of process information. #104

Filebeat

Filebeat will now exit if a configuration error is detected. #198
Fix to enable prospector to harvest existing files that are modified. #199
Improve line reading and encoding to better keep track of file offsets based on encoding. #224
Set input_type by default to "log"

Added

Affecting all Beats

Added beat.hostname to contain the hostname where the Beat is running on as returned by the operating system. #285
Added timestamp for file logging. #291

Filebeat

Handling end of line under windows was improved #233

Beats version 1.0.0-rc1

Check 1.0.0-rc1 diff

Breaking changes

Affecting all Beats

Rename timestamp field with @timestamp. #237

Packetbeat

Rename timestamp field with @timestamp. #343

Topbeat

Rename timestamp field with @timestamp for a better integration with Logstash. #80

Filebeat

Rename the timestamp field with @timestamp #168
Rename tail_on_rotate prospector config to tail_files
Removal of line field in event. Line number was not correct and does not add value. #217

Bugfixes

Affecting all Beats

Use stderr for console log output. #219
Handle empty event array in publisher. #207
Respect '*' debug selector in IsDebug. #226 (elastic#339)
Limit number of workers for Elasticsearch output. elastic#226
On Windows, remove service related error message when running in the console. #242
Fix waitRetry no configured in single output mode configuration. elastic/filebeat#144
Use http as the default scheme in the elasticsearch hosts #253
Respect max bulk size if bulk publisher (collector) is disabled or sync flag is set.
Always evaluate status code from Elasticsearch responses when indexing events. #192
Use bulk_max_size configuration option instead of bulk_size. #256
Fix max_retries=0 (no retries) configuration option. #266
Filename used for file based logging now defaults to beat name. #267

Packetbeat

Close file descriptors used to monitor processes. #337
Remove old RPM spec file. It moved to elastic/beats-packer. #334

Topbeat

Don’t wait for one period until shutdown #75

Filebeat

Omit 'fields' from event JSON when null. #126
Make offset and line value of type long in elasticsearch template to prevent overflow. #140
Fix locking files for writing behaviour. #156
Introduce 'document_type' config option per prospector to define document type for event stored in elasticsearch. #133
Add 'input_type' field to published events reporting the prospector type being used. #133
Fix high CPU usage when not connected to Elasticsearch or Logstash. #144
Fix issue that files were not crawled anymore when encoding was set to something other then plain. #182

Added

Affecting all Beats

Add Console output plugin. #218
Add timestamp to log messages #245
Send @metadata.beat to Logstash instead of @metadata.index to prevent possible name clashes and give user full control over index name used for Elasticsearch
Add logging messages for bulk publishing in case of error #229
Add option to configure number of parallel workers publishing to Elasticsearch or Logstash.
Set default bulk size for Elasticsearch output to 50.
Set default http timeout for Elasticsearch to 90s.
Improve publish retry if sync flag is set by retrying only up to max bulk size events instead of all events to be published.

Filebeat

Introduction of backoff, backoff_factor, max_backoff, partial_line_waiting, force_close_windows_files config variables to make crawling more configurable.
All Godeps dependencies were updated to master on 2015-10-21 [#122]
Set default value for ignore_older config to 10 minutes. #164
Added the fields_under_root setting to optionally store the custom fields top level in the output dictionary. #188
Add more encodings by using x/text/encodings/htmlindex package to select encoding by name.

Beats version 1.0.0-beta4

Check 1.0.0-beta4 diff

Breaking changes

Affecting all Beats

Update tls config options naming from dash to underline #162
Feature/output modes: Introduction of PublishEvent(s) to be used by beats #118 #115

Packetbeat

Renamed http module config file option 'strip_authorization' to 'redact_authorization'
Save_topology is set to false by default
Rename elasticsearch index to [packetbeat-]YYYY.MM.DD

Topbeat

Percentage fields (e.g user_p) are exported as a float between 0 and 1 #34

Bugfixes

Affecting all Beats

Determine Elasticsearch index for an event based on UTC time #81
Fixing ES output’s defaultDeadTimeout so that it is 60 seconds #103
ES outputer: fix timestamp conversion #91
Fix TLS insecure config option #239
ES outputer: check bulk API per item status code for retransmit on failure.

Packetbeat

Support for lower-case header names when redacting http authorization headers
Redact proxy-authorization if redact-authorization is set
Fix some multithreading issues #203
Fix negative response time #216
Fix memcache TCP connection being nil after dropping stream data. #299
Add missing DNS protocol configuration to documentation #269

Topbeat

Don’t divide the reported memory by an extra 1024 #60

Added

Affecting all Beats

Add logstash output plugin #151
Integration tests for Beat → Logstash → Elasticsearch added #195 #188 #168 #137 #128 #112
Large updates and improvements to the documentation
Add direction field to publisher output to indicate inbound/outbound transactions #150
Add tls configuration support to elasticsearch and logstash outputers #139
All external dependencies were updated to the latest version. Update to Golang 1.5.1 #162
Guarantee ES index is based in UTC time zone #164
Cache: optional per element timeout #144
Make it possible to set hosts in different ways. #135
Expose more TLS config options #124
Use the Beat name in the default configuration file path #99

Packetbeat

add [.editorconfig file](http://editorconfig.org/)
add (experimental/unsupported?) saltstack files
Sample config file cleanup
Moved common documentation to [libbeat repository](https://github.com/elastic/libbeat)
Update build to go 1.5.1
Adding device descriptions to the -device output.
Generate coverage for system tests
Move go-daemon dependency to beats-packer
Rename integration tests to system tests
Made the -devices option more user friendly in case sudo is not used. Issue #296.
Publish expired DNS transactions #301
Update protocol guide to libbeat changes
Add protocol registration to new protocol guide
Make transaction timeouts configurable #300
Add direction field to the exported fields #317

Topbeat

Document fields in a standardized format (etc/fields.yml) #34
Updated to use new libbeat Publisher #37 #41
Update to go 1.5.1 #43
Updated configuration files with comments for all options #65
Documentation improvements

Deprecated

Affecting all Beats

Redis output was deprecated #169 #145
Host and port configuration options are deprecated. They are replaced by the hosts configuration option. #141

Files

CHANGELOG.asciidoc

Latest commit

History

CHANGELOG.asciidoc

File metadata and controls

Beats version 6.4.0

Known issue

Breaking changes

Bugfixes

Added

Deprecated

Beats version 6.3.1

Bugfixes

Added

Beats version 6.3.0

Breaking changes

Bugfixes

Added

Beats version 6.2.3

Breaking changes

Bugfixes

Beats version 6.2.2

Bugfixes

Beats version 6.2.1

Beats version 6.2.0

Breaking changes

Bugfixes

Added

Beats version 6.1.3

Beats version 6.1.2

Bugfixes

Added

Beats version 6.1.1

Beats version 6.1.0

Breaking changes

Bugfixes

Added

Beats version 6.0.1

Bugfixes

Beats version 6.0.0-GA

Bugfixes

Added

Beats version 6.0.0-rc2

Breaking changes

Bugfixes

Added

Deprecated

Beats version 6.0.0-rc1

Bugfixes

Added

Beats version 6.0.0-beta2

Breaking changes

Bugfixes

Added

Beats version 6.0.0-beta1

Breaking changes

Bugfixes

Added

Beats version 6.0.0-alpha2

Breaking changes

Bugfixes

Added

Deprecated

Known Issue

Beats version 6.0.0-alpha1

Breaking changes

Bugfixes

Added

Deprecated

Known Issue

Beats version 5.6.2

Beats version 5.6.1

Beats version 5.6.0

Breaking changes

Bugfixes

Added

Deprecated

Beats version 5.5.3

Beats version 5.5.2