Merge branch '7.x' into mergify/bp/7.x/pr-26158

CHANGELOG.next.asciidoc

@@ -218,6 +218,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix bug in aws-s3 input where the end of gzipped log files might have been discarded. {pull}26260[26260]
 - Fix bug in `httpjson` that prevented `first_event` getting updated. {pull}26407[26407]
 - Fix bug in the Syslog input that misparsed rfc5424 days starting with 0. {pull}26419[26419]
+- Do not close filestream harvester if an unexpected error is returned when close.on_state_change.* is enabled. {pull}26411[26411]
 
 *Filebeat*
 
@@ -298,10 +299,12 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix Nginx module pipelines. {issue}19088[19088] {pull}24699[24699]
 - Remove space from field `sophos.xg.trans_src_ ip`. {issue}25154[25154] {pull}25250[25250]
 - Fix `checkpoint.action_reason` when its a string, not a Long. {issue}25575[25575] {pull}25609[25609]
+- Fix `fortinet.firewall.addr` when its a string, not an IP address. {issue}25585[25585] {pull}25608[25608]
 - Fix incorrect field name appending to `related.hash` in `threatintel.abusechmalware` ingest pipeline. {issue}25151[25151] {pull}25674[25674]
 - Add improvements to the azure activitylogs and platformlogs ingest pipelines. {pull}26148[26148]
 - Fix `kibana.log` pipeline when `event.duration` calculation becomes a Long. {issue}24556[24556] {pull}25675[25675]
 - Clone value when copy fields in processors to avoid crash. {issue}19206[19206] {pull}20500[20500]
+- Removed incorrect `http.request.referrer` field from `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
 
 *Heartbeat*
 
@@ -591,6 +594,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Update Okta module to parse additional fields to `okta.debug_context.debug_data`. {issue}25689[25689] {pull}25818[25818]
 - Added dataset `anomalithreatstream` to the `threatintel` module to ingest indicators from Anomali ThreatStream {pull}26350[26350]
 
+- Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457]
+- Add `uri_parts` and `user_agent` ingest processors to `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
+
 *Heartbeat*
 
 - Bundle synthetics deps with heartbeat docker image. {pull}23274[23274]

filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl

@@ -292,6 +292,16 @@ filebeat.inputs:
   # original for harvesting but will report the symlink name as source.
   #prospector.scanner.symlinks: false
 
+  ### Log rotation
+
+  # When an external tool rotates the input files with copytruncate strategy
+  # use this section to help the input find the rotated files.
+  #rotation.external.strategy.copytruncate:
+  # Regex that matches the rotated files.
+  #  suffix_regex: \.\d$
+  # If the rotated filename suffix is a datetime, set it here. 
+  #  dateformat: -20060102
+
   ### State options
 
   # Files for the modification data is older then clean_inactive the state from the registry is removed

filebeat/docs/fields.asciidoc

@@ -107782,6 +107782,279 @@ Specifies the sub type of the log
 Virtual system instance
 
 
+type: keyword
+
+--
+
+*`panw.panos.client_os_ver`*::
++
+--
+The client device’s OS version.
+
+
+type: keyword
+
+--
+
+*`panw.panos.client_os`*::
++
+--
+The client device’s OS version.
+
+
+type: keyword
+
+--
+
+*`panw.panos.client_ver`*::
++
+--
+The client’s GlobalProtect app version.
+
+
+type: keyword
+
+--
+
+*`panw.panos.stage`*::
++
+--
+A string showing the stage of the connection
+
+
+type: keyword
+
+example: before-login
+
+--
+
+*`panw.panos.actionflags`*::
++
+--
+A bit field indicating if the log was forwarded to Panorama.
+
+
+type: keyword
+
+--
+
+*`panw.panos.error`*::
++
+--
+A string showing that error that has occurred in any event.
+
+
+type: keyword
+
+--
+
+*`panw.panos.error_code`*::
++
+--
+An integer associated with any errors that occurred.
+
+
+type: integer
+
+--
+
+*`panw.panos.repeatcnt`*::
++
+--
+The number of sessions with the same source IP address, destination IP address, application, and subtype that GlobalProtect has detected within the last five seconds.An integer associated with any errors that occurred.
+
+
+type: integer
+
+--
+
+*`panw.panos.serial_number`*::
++
+--
+The serial number of the user’s machine or device.
+
+
+type: keyword
+
+--
+
+*`panw.panos.auth_method`*::
++
+--
+A string showing the authentication type
+
+
+type: keyword
+
+example: LDAP
+
+--
+
+*`panw.panos.datasource`*::
++
+--
+Source from which mapping information is collected.
+
+
+type: keyword
+
+--
+
+*`panw.panos.datasourcetype`*::
++
+--
+Mechanism used to identify the IP/User mappings within a data source.
+
+
+type: keyword
+
+--
+
+*`panw.panos.datasourcename`*::
++
+--
+User-ID source that sends the IP (Port)-User Mapping.
+
+
+type: keyword
+
+--
+
+*`panw.panos.factorno`*::
++
+--
+Indicates the use of primary authentication (1) or additional factors (2, 3).
+
+
+type: integer
+
+--
+
+*`panw.panos.factortype`*::
++
+--
+Vendor used to authenticate a user when Multi Factor authentication is present.
+
+
+type: keyword
+
+--
+
+*`panw.panos.factorcompletiontime`*::
++
+--
+Time the authentication was completed.
+
+
+type: date
+
+--
+
+*`panw.panos.ugflags`*::
++
+--
+Displays whether the user group that was found during user group mapping. Supported values are:
+User Group Found—Indicates whether the user could be mapped to a group.
+Duplicate User—Indicates whether duplicate users were found in a user group. Displays N/A if no user group is found.
+
+
+type: keyword
+
+--
+
+[float]
+=== device_group_hierarchy
+
+A sequence of identification numbers that indicate the device group’s location within a device group hierarchy. The firewall (or virtual system) generating the log includes the identification number of each ancestor in its device group hierarchy. The shared device group (level 0) is not included in this structure. If the log values are 12, 34, 45, 0, it means that the log was generated by a firewall (or virtual system) that belongs to device group 45, and its ancestors are 34, and 12.
+
+
+
+*`panw.panos.device_group_hierarchy.level_1`*::
++
+--
+A sequence of identification numbers that indicate the device group’s location within a device group hierarchy. The firewall (or virtual system) generating the log includes the identification number of each ancestor in its device group hierarchy. The shared device group (level 0) is not included in this structure. If the log values are 12, 34, 45, 0, it means that the log was generated by a firewall (or virtual system) that belongs to device group 45, and its ancestors are 34, and 12.
+
+
+type: keyword
+
+--
+
+*`panw.panos.device_group_hierarchy.level_2`*::
++
+--
+A sequence of identification numbers that indicate the device group’s location within a device group hierarchy. The firewall (or virtual system) generating the log includes the identification number of each ancestor in its device group hierarchy. The shared device group (level 0) is not included in this structure. If the log values are 12, 34, 45, 0, it means that the log was generated by a firewall (or virtual system) that belongs to device group 45, and its ancestors are 34, and 12.
+
+
+type: keyword
+
+--
+
+*`panw.panos.device_group_hierarchy.level_3`*::
++
+--
+A sequence of identification numbers that indicate the device group’s location within a device group hierarchy. The firewall (or virtual system) generating the log includes the identification number of each ancestor in its device group hierarchy. The shared device group (level 0) is not included in this structure. If the log values are 12, 34, 45, 0, it means that the log was generated by a firewall (or virtual system) that belongs to device group 45, and its ancestors are 34, and 12.
+
+
+type: keyword
+
+--
+
+*`panw.panos.device_group_hierarchy.level_4`*::
++
+--
+A sequence of identification numbers that indicate the device group’s location within a device group hierarchy. The firewall (or virtual system) generating the log includes the identification number of each ancestor in its device group hierarchy. The shared device group (level 0) is not included in this structure. If the log values are 12, 34, 45, 0, it means that the log was generated by a firewall (or virtual system) that belongs to device group 45, and its ancestors are 34, and 12.
+
+
+type: keyword
+
+--
+
+*`panw.panos.timeout`*::
++
+--
+Timeout after which the IP/User Mappings are cleared.
+
+
+type: integer
+
+--
+
+*`panw.panos.vsys_id`*::
++
+--
+A unique identifier for a virtual system on a Palo Alto Networks firewall.
+
+
+type: keyword
+
+--
+
+*`panw.panos.vsys_name`*::
++
+--
+The name of the virtual system associated with the session; only valid on firewalls enabled for multiple virtual systems.
+
+
+type: keyword
+
+--
+
+*`panw.panos.description`*::
++
+--
+Additional information for any event that has occurred.
+
+
+type: keyword
+
+--
+
+*`panw.panos.tunnel_type`*::
++
+--
+The type of tunnel (either SSLVPN or IPSec).
+
+
 type: keyword
 
 --

filebeat/docs/inputs/input-filestream-file-options.asciidoc

@@ -482,3 +482,56 @@ Set the location of the marker file the following way:
 ----
 file_identity.inode_marker.path: /logs/.filebeat-marker
 ----
+
+=== Log rotation
+
+As log files are constantly written, they must be rotated and purged to prevent
+the logger application from filling up the disk. Rotation is done by an external
+application, thus, {beatname_uc} needs information how to cooperate with it.
+
+When reading from rotating files make sure the paths configuration includes
+both the active file and all rotated files.
+
+By default, {beatname_uc} is able to track files correctly in the following strategies:
+* create: new active file with a unique name is created on rotation
+* rename: rotated files are renamed
+
+However, in case of copytruncate strategy, you should provide additional configuration
+to {beatname_uc}.
+
+[float]
+==== rotation.external.strategy.copytruncate
+
+experimental[]
+
+If the log rotating application copies the contents of the active file and then
+truncates the original file, use these options to help {beatname_uc} to read files
+correctly.
+
+Set the option `suffix_regex` so {beatname_uc} can tell active and rotated files apart. There are
+two supported suffix types in the input: numberic and date.
+
+==== Numeric suffix
+
+If your rotated files have an incrementing index appended to the end of the filename, e.g.
+active file `apache.log` and the rotated files are named `apache.log.1`, `apache.log.2`, etc,
+use the following configuration.
+
+[source,yaml]
+---
+rotation.external.strategy.copytruncate:
+  suffix_regex: \.\d$
+---
+
+==== Date suffix
+
+If the rotation date is appended to the end of the filename, e.g. active file `apache.log` and the
+rotated files are named `apache.log-20210526`, `apache.log-20210527`, etc. use the following configuration:
+
+[source,yaml]
+---
+rotation.external.strategy.copytruncate:
+  suffix_regex: \-\d{6}$
+  dateformat: -20060102
+---
+