CHANGELOG.next.asciidoc

Beats version HEAD

Check the HEAD diff

Breaking changes

Affecting all Beats

• Remove the non-ECS agent.hostname field. Use the agent.name or agent.id fields for an identifier. 16377 18328

• Make error message about locked data path actionable. 18667

• Remove the deprecated xpack.monitoring. settings. Going forward only monitoring. settings may be used. 9424 18608

• Skip add_kubernetes_metadata processor when kubernetes metadata are already present 27689

• Remove deprecated/undocumented IncludeCreatorMetadata setting from kubernetes metadata config options 28006

• Remove deprecated fields from kubernetes module 28046

• Remove deprecated config option aws_partition. 28120

• Improve stats API 27963

• Enable IMDSv2 support for add_cloud_metadata processor on AWS. 22101 28285

• Update kubernetes.namespace from keyword to group field and add name, labels, annotations, uuid as its fields 27917

• Libbeat: logp package forces ECS compliant logs. Logs are JSON formatted. Options to enable ECS/JSON have been removed. 15544 28573

• Previously, RE2 and thus Golang had a bug where (|a)* matched more characters than (|a)+. To stay consistent with PCRE, the bug was fixed. Configurations that rely on the old, buggy behaviour has to be adjusted. See more about Golang bug: golang/go#46123 27543

• Update docker client. 28716

• Remove auto from the available options of setup.ilm.enabled and set the default value to true. 28671

• add_process_metadata processor: Replace usage of deprecated process.ppid field with process.parent.pid. 28620

• add_docker_metadata processor: Replace usage of deprecated process.ppid field with process.parent.pid. 28620

• Index template’s default_fields setting is only populated with ECS fields. 28596 28215

Auditbeat

• File integrity dataset (macOS): Replace unnecessary file.origin.raw (type keyword) with file.origin.text (type text). 12423 15630

• Change event.kind=error to event.kind=event to comply with ECS. 18870 20685

• File integrity dataset: Remove non-ECS hash. fields. Hashes are under file.hash.. 19039 28378

• Auditd dataset: Removes the authentication_success and authentication_failure event.type values for user logins. 19039 28378

• Fix handling of long file names on Windows. 25334 28517

• System/socket dataset: Fix uninstallation of return kprobes. 28608 28609

• Replace usage of deprecated process.ppid field with process.parent.pid. 28620

• Fix auditbeat tracing struct decoding. 28580

Filebeat

• Fix parsing of Elasticsearch node name by elasticsearch/slowlog fileset. 14547

• With the default configuration the following modules will no longer send the host field that contains information about the host on which Filebeat is running. You can revert this change by configuring tags for the module and omitting forwarded from the list. 13920

• Preserve case of http.request.method. ECS prior to 1.6 specified normalizing to lowercase, which lost information. Affects filesets: apache/access, elasticsearch/audit, iis/access, iis/error, nginx/access, nginx/ingress_controller, aws/elb, suricata/eve, zeek/http. 18154 18359

• With the default configuration the cloud modules (aws, azure, googlecloud, o365, okta)

• With the default configuration the cef and panw modules will no longer send the host

• Preserve case of http.request.method. ECS prior to 1.6 specified normalizing to lowercase, which lost information. Affects filesets: apache/access, elasticsearch/audit, iis/access, iis/error, nginx/access, nginx/ingress_controller, aws/elb, suricata/eve, zeek/http. 18154 18359

• Add while_pattern type to multiline reader. 19662

• Add support for GMT timezone offsets in decode_cef. 20993

• Fix parsing of Elasticsearch node name by elasticsearch/slowlog fileset. 14547

• Removes old module aliases for googlecloud (moved to gcp) and apache2 (moved to apache). 27919

• Removes old module name aliases (gsuite) and removing old cyberark module in favor of the new cyberarkpashttps://github.com/elastic/pull/27915[27915]

• Only filesets that are explicitly configured will be enabled. 17256 27526

• All filesets are disabled in the default configuration. 17256 27762

• Remove deprecated fields in Kafka module. 27938

• Remove deprecated fields in coredns module. 28196

• Remove old httpjson config implementation. 28054

• Added dataset threatq to the threatintel module to ingest indicators from ThreatQ 27423

• Fail to start Filebat if none between queue_url, bucket_arn or non_aws_bucket_name is set for a configured aws-s3 input 13911 28666

• All modules: Replace usages of deprecated ECS fields process.ppid and log.original with process.parent.pid and event.original. 28620

• Replace usages of host.user. fields with user. in cisco, microsoft and oracle modules. 28620

• Remove docker input. Please use filestream input with container parser or container input. 28817

Heartbeat

Journalbeat

• Rename field journald.process.capabilites to journald.process.capabilities to fix spelling. 28065

• Rename field log.syslog.facility.name to log.syslog.facility.code because the value is numeric rather than the facility name. 28065

Metricbeat

• Add Linux pressure metricset 27355

• Add User-Agent header to HTTP requests. 18160 27509

• Errors should be thrown as errors. Metricsets inside Metricbeat will now throw errors as the error log level. 27804

• Remove deprecated fields in Docker module. 11835 27933

• Remove deprecated fields in Kafka module. 27938

• Remove deprecated config option default_region from aws module. 28120

• Remove network and diskio metrics from ec2 metricset. 28316

• Rename read/write_io.ops_per_sec to read/write.iops in rds metricset. 28350

• Remove linux-only metrics from diskio, memory 28292

• Remove deprecated config option perfmon.counters from windows/perfmon metricset. 28282

• Remove deprecated fields in Redis module. 11835 28246

• Align fields to Beats naming conventions in GCP module. 27231 27974

• system/process metricset: Replace usage of deprecated process.ppid field with process.parent.pid. 28620

Packetbeat

• Redis: fix incorrectly handle with two-words redis command. 14872 14873

• event.category no longer contains the value network_traffic because this is not a valid ECS event category value. 20556

• Remove deprecated TLS fields in favor of tls.server.x509 and tls.client.x509 ECS fields. 28487

• HTTP: The field http.request.method will maintain its original case. 28620

Winlogbeat

• Add support to Sysmon file delete events (event ID 23). 18094

• Improve ECS field mappings in Sysmon module. related.hash, related.ip, and related.user are now populated. 18364

• Improve ECS field mappings in Sysmon module. Hashes are now also populated to the corresponding process.hash, process.pe.imphash, file.hash, or file.pe.imphash. 18364

• Improve ECS field mappings in Sysmon module. file.name, file.directory, and file.extension are now populated. 18364

• Improve ECS field mappings in Sysmon module. rule.name is populated for all events when present. 18364

• Fix unprefixed fields in fields.yml for Powershell module 18984

• Remove top level hash property from sysmon events 20653

Functionbeat

• Support for Google Cloud Functions have been removed, as it has been in Beta for a long time and been broken for a few releases. Please use other tools provided by Elastic to fetch data from GCP (e.g. Filebeat).

Bugfixes

Affecting all Beats

• Fix a race condition with the Kafka pipeline client, it is possible that Close() get called before Connect() . 11945

• Allow users to configure only cluster_uuid setting under monitoring namespace. 14338

• Update replicaset group to apps/v1 15802

• Fix missing output in dockerlogbeat 15719

• Fix issue where TLS settings would be ignored when a forward proxy was in use. 15516

• Update replicaset group to apps/v1 15802

• Add ssl.ca_sha256 option to the supported TLS option, this allow to check that a specific certificate is used as part of the verified chain. 15717

• Improve some logging messages for add_kubernetes_metadata processor https://github.com/elastic/beats/pull/16866{16866}

• Do not rotate log files on startup when interval is configured and rotateonstartup is disabled. 17613

• Fix setup.dashboards.index setting not working. 17749

• Fix Elasticsearch license endpoint URL referenced in error message. 17880 18030

• Change decode_json_fields processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

• Gives monitoring reporter hosts, if configured, total precedence over corresponding output hosts. 17937 17991

• Change decode_json_fields processor, to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

• [Autodiscover] Check if runner is already running before starting again. 18564

• Fix an issue where error messages are not accurate in mapstriface. 18662 18663

• Fix regression in add_kubernetes_metadata, so configured indexers and matchers are used if defaults are not disabled. 18481 18818

• Fix the translate_sid processor’s handling of unconfigured target fields. 18990 18991

• Fixed a service restart failure under Windows. 18914 18916

• Fix terminating pod autodiscover issue. 20084

• Fix seccomp policy for calls to chmod and chown. 20054

• Output errors when Kibana index pattern setup fails. 20121

• Fix issue in autodiscover that kept inputs stopped after config updates. 20305

• Add service resource in k8s cluster role. 20546

• Periodic metrics in logs will now report libbeat.output.events.active and beat.memstats.rss

• Allows disable pod events enrichment with deployment name 28521

• Fix fingerprint processor to give it access to the @timestamp field. 28683

• Fix the wrong beat name on monitoring and state endpoint 27755

Auditbeat

• system/package: Fix parsing of Installed-Size field of DEB packages. 16661 17188

• system module: Fix panic during initialisation when /proc/stat can’t be read. 17569

• system/package: Fix an error that can occur while trying to persist package metadata. 18536 18887

• Fix handling of root and relative paths 24430 28354

Filebeat

• cisco/asa fileset: Fix parsing of 302021 message code. 14519

• Fix filebeat azure dashboards, event category should be Alert. 14668

• Fix s3 input with cloudtrail fileset reading json file. 16374 16441

• Add queue_url definition in manifest file for aws module. 16640

• Add queue_url definition in manifest file for aws module. https://github.com/elastic/beats/pull/16640{16640}

• Fix elasticsearch.gc fileset to not collect all logs when Elasticsearch is running in Docker. 13164 16583 17164

• Fixed a mapping exception when ingesting CEF logs that used the spriv or dpriv extensions. 17216 17220

• Remove migrationVersion map 7.7.0 reference from Kibana dashboard file to fix backward compatibility issues. 17425

• Fix issue 17734 to retry on rate-limit error in the Filebeat httpjson input. 17734 17735

• Fixed cloudfoundry.access to have the correct cloudfoundry.app.id contents. 17847

• Fixing ingress_controller. fields to be of type keyword instead of text. 17834

• Fixed typo in log message. 17897

• Fix o365 module ignoring var.api settings. 18948

• Fix netflow module to support 7 bytepad for IPFIX template. 18098

• Update container name for the azure filesets. 19899

• Fix o365 module ignoring var.api settings. 18948

• Fix S3 input to trim delimiter /n from each log line. 19972

• Fix s3 input parsing json file without expand_event_list_from_field. 19902 19962 20370

• Fix millisecond timestamp normalization issues in CrowdStrike module 20035, 20138

• Fix support for message code 106100 in Cisco ASA and FTD. 19350 20245

• Fix fortinet setting event.timezone to the system one when no tz field present 20273

• Fix okta geoip lookup in pipeline for destination.ip 20454

• Fix mapping exception in the googlecloud/audit dataset pipeline. 18465 20465

• Fix cisco asa and ftd parsing of messages 106102 and 106103. 20469

• Update indentation for azure filebeat configuration. 26604

• Add support for passing a prefix on S3 bucket list mode for AWS-S3 input 28252 27965

• Resolve issue with @timestamp for defender_atp. 28272

• Tolerate faults when Windows Event Log session is interrupted 27947 28191

• Add support for username in cisco asa security negotiation logs 26975

• Relax time parsing and capture group and session type in Cisco ASA module 24710 28325

• Correctly track bytes read when max_bytes is exceeded. 28317 28352

• Upgrade azure-eventhub sdk reference, contains potential checkpoint fixes. 28919

• Revert usageDetails api version to 2019-01-01. 28995

• Fix in aws-s3 input regarding provider discovery through endpoint 28963

Heartbeat

• Fix broken seccomp filtering and improve security via setcap and setuid when running as root on linux in containers. 27878

• Log browser zip_url download failures as warn instead of as info. 28440

• Properly locate base stream in fleet configs. 28455

• Stop logging params values. 28774

• Remove accidentally included cups library in docker images. pull

• Fix broken monitors with newer versions of image relying on dup3. pull

Journalbeat

Metricbeat

• Fix checking tagsFilter using length in cloudwatch metricset. 14525

• Log bulk failures from bulk API requests to monitoring cluster. 14303 14356

• Fix skipping protocol scheme by light modules. pull

• Revert changes in docker module: add size flag to docker.container. 16600

• Fix detection and logging of some error cases with light modules. 14706

• Fix imports after PR was merged before rebase. 16756

• Reduce memory usage in elasticsearch/index metricset. 16503 16538

• Fix issue in Jolokia module when mbean contains multiple quoted properties. 17375 17374

• Fix issue in Jolokia module when mbean contains multiple quoted properties. 17375 17374

• Fix azure storage dashboards. 17590

• Metricbeat no longer needs to be started strictly after Logstash for logstash-xpack module to report correct data. 17261 17497

• Fix pubsub metricset to collect all GA stage metrics from gcp stackdriver. 17154 17600

• Add privileged option so as mb to access data dir in Openshift. 17606

• Fix "ID" event generator of Google Cloud module 17160 17608

• Add privileged option for Auditbeat in Openshift 17637

• Fix storage metricset to allow config without region/zone. 17623 17624

• Fix overflow on Prometheus rates when new buckets are added on the go. 17753

• Remove specific win32 api errors from events in perfmon. 18292 18361

• Fix application_pool metricset after pdh changes. 18477

• Fix panic on metricbeat test modules when modules are configured in metricbeat.modules. 18789 18797

• Fix getting gcp compute instance metadata with partial zone/region in config. 18757

• Add missing network.sent_packets_count metric into compute metricset in googlecloud module. 18802

• Fix compute and pubsub dashboard for googlecloud module. 18962 18980

• Fix crash on vsphere module when Host information is not available. 18996 19078

• Modify doc for app_insights metricset to contain example of config. 20185

• Add required option for metrics in app_insights. 20406

• Groups same timestamp metric values to one event in the app_insights metricset. 20403

• beat module respects basepath config option. 28162

• Fix list_docker.go 28374

• Use xpack.enabled on SM modules to write into .monitoring indices when using Metricbeat standalone 28365

• Fix in rename processor to ingest metrics for write.iops to proper field instead of write_iops in rds metricset. 28960

Packetbeat

Winlogbeat

• Add source.ip validation for event ID 4778 in the Security module. 19627

• Tolerate faults when Windows Event Log session is interrupted 27947 28191

• Add ECS 1.9 new users fields 26509

• Don’t split hyphenated tokens 28483

Functionbeat

Elastic Logging Plugin

Added

Affecting all Beats

• Decouple Debug logging from fail_on_error logic for rename, copy, truncate processors 12451

• Fingerprint processor adds a new xxhash hashing algorithm 15418

• Update RPM packages contained in Beat Docker images. 17035

• Update documentation for system.process.memory fields to include clarification on Windows os’s. 17268

• When using the decode_json_fields processor, decoded fields are now deep-merged into existing event. 17958

• Update documentation for system.process.memory fields to include clarification on Windows os’s. 17268

• Add keystore support for autodiscover static configurations. {pull]16306[16306]

• When using the decode_json_fields processor, decoded fields are now deep-merged into existing event. 17958

• Add keystore support for autodiscover static configurations. {pull]16306[16306]

• Add TLS support to Kerberos authentication in Elasticsearch. 18607

• Add config option rotate_on_startup to file output 19150 19347

• Set index.max_docvalue_fields_search in index template to increase value to 200 fields. 20215

• Allow non-padded base64 data to be decoded by decode_base64_field 27311, 27021

• The Kafka support library Sarama has been updated to 1.29.1. 27717

• Kafka is now supported up to version 2.8.0. 27720

• Add Huawei Cloud provider to add_cloud_metadata. 27607

• Add default seccomp policy for linux arm64. 27955

• Add cluster level add_kubernetes_metadata support for centralized enrichment 24621

• Update ECS to 1.12.0. 27770

• Fields mapped as match_only_text will automatically fallback to a text mapping when using Elasticsearch versions that do not support match_only_text. 27770

• Update cloud.google.com/go library. 28229

• Add additional metadata to the root HTTP endpoint. 28265

• Upgrade k8s.io/client-go library. 28228

• Upgrade prometheus library. 28716

• Name all k8s workqueue. 28085

• Update kubernetes scheduler and controllermanager endpoints in elastic-agent-standalone-kubernetes.yaml with secure ports 28675

• Add options to configure k8s client qps/burst. 28151

• Update to ECS 8.0 fields. 28620

• Add http.pprof.enabled option to libbeat to allow http/pprof endpoints on the socket that libbeat creates for metrics. 21965

• Support custom analyzers in fields.yml. 28540 28926

Auditbeat

• Reference kubernetes manifests include configuration for auditd and enrichment with kubernetes metadata. 17431

Filebeat

• container and docker inputs now support reading of labels and env vars written by docker JSON file logging driver. 8358

• Add index option to all inputs to directly set a per-input index value. 14010

• move create-[module,fileset,fields] to mage and enable in x-pack/filebeat 15836

• Work on e2e ACK’s for the azure-eventhub input 15671 16215

• Add a TLS test and more debug output to httpjson input 16315

• Add an SSL config example in config.yml for filebeat MISP module. 16320

• Update filebeat httpjson input to support pagination via Header and Okta module. 16354

• Add a TLS test and more debug output to httpjson input 16315

• Add an SSL config example in config.yml for filebeat MISP module. 16320

• Added documentation for running Filebeat in Cloud Foundry. 17275

• Release Google Cloud module as GA. 17511

• Improve ECS categorization field mappings for nats module. 16173 17550

• Enhance elasticsearch/slowlog fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17729

• Added documentation for running Filebeat in Cloud Foundry. 17275

• Release Google Cloud module as GA. 17511

• Update filebeat httpjson input to support pagination via Header and Okta module. 16354

• Change the json.* input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. 17958

• Add support for array parsing in azure-eventhub input. 18585

• Add support for array parsing in azure-eventhub input. 18585

• Improved performance of PANW sample dashboards. 19031 19032

• Add event.ingested for CrowdStrike module 20138

• Add support for additional fields and FirewallMatchEvent type events in CrowdStrike module 20138

• Add timezone config option to the decode_cef processor. 27232 27727

• Add timezone config option to the syslog input. 27727

• Added support for parsing syslog dates containing a leading 0 (e.g. Sep 01) rather than a space. 27775

• Add base64 Encode functionality to httpjson input. 27681

• Add join and sprintf functions to httpjson input. 27735

• Improve memory usage of line reader of log and filestream input. 27782

• Add ignore_empty_value flag to httpjson split processor. 27880

• Update Cisco ASA/FTD ingest pipeline grok/dissect patterns for multiple message IDs. 26869 26879

• Add write access to url.value from request.transforms in httpjson input. 27937

• Add Base64 encoded HMAC and UUID template functions to httpjson input 27873

• Release checkpoint module as GA. 27814

• Make aws-cloudwatch input GA. 28161

• Move processing to ingest node for AWS vpcflow fileset. 28168

• Release zoom module as GA. 28106

• Add support for secondary object attribute handling in ThreatIntel MISP module 28124

• Azure signinlogs - Add support for ManagedIdentitySignInLogs, NonInteractiveUserSignInLogs, and ServicePrincipalSignInLogs. 23653

• Add base64Decode and base64DecodeNoPad functions to httpsjon templates. 28385

• Add 'early_limit' config option for Rate-Limiting httpjson. Default rate-limiting for Okta will start when remaining is 1. 28513

• Add latency config option for aws-cloudwatch input. 28509

• Added proxy support to threatintel/malwarebazaar. 28533

• Add text/csv decoder to httpjson input 28564

• Update aws-s3 input to connect to non AWS S3 buckets 28222 28234

• Sophos UTM: Support logs containing hostname in syslog header. 28638

• Moving Oracle Filebeat module to GA. 28754

• Add support for '/var/log/pods/' path for add_kubernetes_metadata processor with resource_type: pod. 28868

• Add documentation for add_kubernetes_metadata processors log_path matcher. 28868

• Add support in aws-s3 input for s3 notification from SNS to SQS. 28800

Heartbeat

• Support JSON expressions / validation of JSON arrays. 28073

• Experimental 'run once' mode. 25972

• Add keyword multi-field mapping for synthetics.step.name. 28452

Journalbeat

Metricbeat

• Move the windows pdh implementation from perfmon to a shared location in order for future modules/metricsets to make use of. 15503

• Add database_account azure metricset. 15758

• Add database_account azure metricset. 15758

• Release Zookeeper/connection module as GA. 14281 17043

• Add dashboard for pubsub metricset in googlecloud module. 17161

• Added documentation for running Metricbeat in Cloud Foundry. 17275

• Added documentation for running Metricbeat in Cloud Foundry. 17275

• Remove required for region/zone and make stackdriver a metricset in googlecloud. 16785 18398

• Add memory metrics into compute googlecloud. 18802

• Enable journald input type in Filebeat. 7955 27351

• Added a new beta enterprisesearch module for Elastic Enterprise Search 27549

• Preliminary AIX support 27954

• Register additional name for storage metricset in the azure module. 28447

• Update reference to gosigar pacakge for filesystem windows fix. 28909

Packetbeat

Functionbeat

• Add support for AWS Kinesis record deaggregation 28241

Winlogbeat

• Add more DNS error codes to the Sysmon module. 15685

• Add support for event language selection from config file 19818

Elastic Log Driver

• Fixed docs for hosts 23644

Deprecated

Affecting all Beats

Filebeat

• Deprecate log input in favour of filestream input. 28623

Heartbeat

Journalbeat

Metricbeat

Packetbeat

Winlogbeat

Functionbeat

Known Issue

Journalbeat