Cherry-pick elastic#14553 to 7.6: New fileset for googlecloud firewal…

…l logs (elastic#15621)

* New fileset for googlecloud firewall logs (elastic#14553)

This PR adds a new fileset, firewall, to the googlecloud module in Filebeat. It helps
parsing firewall logs generated by rules under VPC Network -> Firewall Rules.

Note that GCP only logs firewall events under the following conditions:
- Logging needs to be enabled for each individual rule in order to log.
- Only TCP and UDP rules can be logged.

(cherry picked from commit 4a66f0b)

* googlecloud/vpcflow fileset: Populate additional log fields (elastic#14608)

To keep the vpcflow fileset of the googlecloud module aligned with the
new firewall fileset, a `var.keep_original_message` option is added.
Also the log.logger ECS field is now filled.

(cherry picked from commit 22fb66d)

CHANGELOG.next.asciidoc

@@ -148,6 +148,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Integrate the azure-eventhub with filebeat azure module (replace the kafka input). {pull}15480[15480]
 - Release aws s3access fileset to GA. {pull}15431[15431] {issue}15430[15430]
 - Add cloudtrail fileset to AWS module. {issue}14657[14657] {pull}15227[15227]
+- New fileset googlecloud/firewall for ingesting Google Cloud Firewall logs. {pull}14553[14553]
 
 *Heartbeat*
 

filebeat/docs/fields.asciidoc

@@ -11952,6 +11952,154 @@ Fields from Google Cloud logs.
 
 
 
+[float]
+=== destination.instance
+
+If the destination of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project.
+
+
+
+*`googlecloud.destination.instance.project_id`*::
++
+--
+ID of the project containing the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.destination.instance.region`*::
++
+--
+Region of the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.destination.instance.zone`*::
++
+--
+Zone of the VM.
+
+
+type: keyword
+
+--
+
+[float]
+=== destination.vpc
+
+If the destination of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project.
+
+
+
+*`googlecloud.destination.vpc.project_id`*::
++
+--
+ID of the project containing the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.destination.vpc.vpc_name`*::
++
+--
+VPC on which the VM is operating.
+
+
+type: keyword
+
+--
+
+*`googlecloud.destination.vpc.subnetwork_name`*::
++
+--
+Subnetwork on which the VM is operating.
+
+
+type: keyword
+
+--
+
+[float]
+=== source.instance
+
+If the source of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project.
+
+
+
+*`googlecloud.source.instance.project_id`*::
++
+--
+ID of the project containing the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.source.instance.region`*::
++
+--
+Region of the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.source.instance.zone`*::
++
+--
+Zone of the VM.
+
+
+type: keyword
+
+--
+
+[float]
+=== source.vpc
+
+If the source of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project.
+
+
+
+*`googlecloud.source.vpc.project_id`*::
++
+--
+ID of the project containing the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.source.vpc.vpc_name`*::
++
+--
+VPC on which the VM is operating.
+
+
+type: keyword
+
+--
+
+*`googlecloud.source.vpc.subnetwork_name`*::
++
+--
+Subnetwork on which the VM is operating.
+
+
+type: keyword
+
+--
+
 [float]
 === audit
 
@@ -12165,177 +12313,147 @@ type: keyword
 --
 
 [float]
-=== vpcflow
+=== firewall
 
-Fields for Google Cloud VPC flow logs.
+Fields for Google Cloud Firewall logs.
 
 
 
-*`googlecloud.vpcflow.reporter`*::
-+
---
-The side which reported the flow. Can be either 'SRC' or 'DEST'.
+[float]
+=== rule_details
 
+Description of the firewall rule that matched this connection.
 
-type: keyword
 
---
 
-*`googlecloud.vpcflow.rtt.ms`*::
+*`googlecloud.firewall.rule_details.priority`*::
 +
 --
-Latency as measured (for TCP flows only) during the time interval. This is the time elapsed between sending a SEQ and receiving a corresponding ACK and it contains the network RTT as well as the application related delay.
-
+The priority for the firewall rule.
 
 type: long
 
 --
 
-[float]
-=== destination.instance
-
-If the destination of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project.
-
-
-
-*`googlecloud.vpcflow.destination.instance.project_id`*::
+*`googlecloud.firewall.rule_details.action`*::
 +
 --
-ID of the project containing the VM.
-
+Action that the rule performs on match.
 
 type: keyword
 
 --
 
-*`googlecloud.vpcflow.destination.instance.region`*::
+*`googlecloud.firewall.rule_details.direction`*::
 +
 --
-Region of the VM.
-
+Direction of traffic that matches this rule.
 
 type: keyword
 
 --
 
-*`googlecloud.vpcflow.destination.instance.zone`*::
+*`googlecloud.firewall.rule_details.reference`*::
 +
 --
-Zone of the VM.
-
+Reference to the firewall rule.
 
 type: keyword
 
 --
 
-[float]
-=== destination.vpc
-
-If the destination of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project.
-
-
-
-*`googlecloud.vpcflow.destination.vpc.project_id`*::
+*`googlecloud.firewall.rule_details.source_range`*::
 +
 --
-ID of the project containing the VM.
-
+List of source ranges that the firewall rule applies to.
 
 type: keyword
 
 --
 
-*`googlecloud.vpcflow.destination.vpc.vpc_name`*::
+*`googlecloud.firewall.rule_details.destination_range`*::
 +
 --
-VPC on which the VM is operating.
-
+List of destination ranges that the firewall applies to.
 
 type: keyword
 
 --
 
-*`googlecloud.vpcflow.destination.vpc.subnetwork_name`*::
+*`googlecloud.firewall.rule_details.source_tag`*::
 +
 --
-Subnetwork on which the VM is operating.
+List of all the source tags that the firewall rule applies to.
 
 
 type: keyword
 
 --
 
-[float]
-=== source.instance
+*`googlecloud.firewall.rule_details.target_tag`*::
++
+--
+List of all the target tags that the firewall rule applies to.
 
-If the source of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project.
 
+type: keyword
 
+--
 
-*`googlecloud.vpcflow.source.instance.project_id`*::
+*`googlecloud.firewall.rule_details.ip_port_info`*::
 +
 --
-ID of the project containing the VM.
+List of ip protocols and applicable port ranges for rules.
 
 
-type: keyword
+type: array
 
 --
 
-*`googlecloud.vpcflow.source.instance.region`*::
+*`googlecloud.firewall.rule_details.source_service_account`*::
 +
 --
-Region of the VM.
+List of all the source service accounts that the firewall rule applies to.
 
 
 type: keyword
 
 --
 
-*`googlecloud.vpcflow.source.instance.zone`*::
+*`googlecloud.firewall.rule_details.target_service_account`*::
 +
 --
-Zone of the VM.
+List of all the target service accounts that the firewall rule applies to.
 
 
 type: keyword
 
 --
 
 [float]
-=== source.vpc
-
-If the source of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project.
-
-
-
-*`googlecloud.vpcflow.source.vpc.project_id`*::
-+
---
-ID of the project containing the VM.
+=== vpcflow
 
+Fields for Google Cloud VPC flow logs.
 
-type: keyword
 
---
 
-*`googlecloud.vpcflow.source.vpc.vpc_name`*::
+*`googlecloud.vpcflow.reporter`*::
 +
 --
-VPC on which the VM is operating.
+The side which reported the flow. Can be either 'SRC' or 'DEST'.
 
 
 type: keyword
 
 --
 
-*`googlecloud.vpcflow.source.vpc.subnetwork_name`*::
+*`googlecloud.vpcflow.rtt.ms`*::
 +
 --
-Subnetwork on which the VM is operating.
+Latency as measured (for TCP flows only) during the time interval. This is the time elapsed between sending a SEQ and receiving a corresponding ACK and it contains the network RTT as well as the application related delay.
 
 
-type: keyword
+type: long
 
 --