[Filebeat] Add network.direction by specifying internal_networks to gcp module

[andrewstucki]

What does this PR do?

This uses the new add_network_direction processor to override how we calculate the event network.direction. It allows users to specify what CIDR block ranges/ip address types consist of their "internal" network regardless of how GCP characterizes traffic.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Relates [ECS] Upgrade modules to 1.7 #21674

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #23081 updated

• Start Time: 2020-12-10T22:02:38.204+0000

• Duration: 76 min 16 sec

Test stats 🧪

Test	Results
Failed	0
Passed	17405
Skipped	1379
Total	18784

Steps errors

Expand to view the steps failures

Terraform Apply on x-pack/metricbeat/module/aws

• Took 0 min 20 sec . View more details on here

Terraform Apply on x-pack/metricbeat/module/aws

• Took 0 min 16 sec . View more details on here

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	17405
Skipped	1379
Total	18784

[andrewstucki]

To show this as working. Making this change to the default firewall fileset configuration:

diff --git a/x-pack/filebeat/module/gcp/firewall/manifest.yml b/x-pack/filebeat/module/gcp/firewall/manifest.yml
index d5f684c87..00084d649 100644
--- a/x-pack/filebeat/module/gcp/firewall/manifest.yml
+++ b/x-pack/filebeat/module/gcp/firewall/manifest.yml
@@ -18,6 +18,7 @@ var:
   - name: tags
     default: [forwarded]
   - name: internal_networks
+    default: ["10.128.0.0/8"]

 ingest_pipeline: ingest/pipeline.yml
 input: config/input.yml

diff --git a/x-pack/filebeat/module/gcp/firewall/test/test.log-expected.json b/x-pack/filebeat/module/gcp/firewall/test/test.log-expected.json
index 908b2436b..9b1b1fe50 100644
--- a/x-pack/filebeat/module/gcp/firewall/test/test.log-expected.json
+++ b/x-pack/filebeat/module/gcp/firewall/test/test.log-expected.json
@@ -977,7 +977,7 @@
         "log.logger": "projects/test-beats/logs/compute.googleapis.com%2Ffirewall",
         "log.offset": 14407,
         "network.community_id": "1:DAX43chSGct8LhjTchX9JgmQSEE=",
-        "network.direction": "internal",
+        "network.direction": "inbound",
         "network.iana_number": 6,
         "network.name": "default",
         "network.transport": "tcp",

And to for the vpcflow fileset (don't worry about the weird CIDRs, it's just to show the override):

diff --git a/x-pack/filebeat/module/gcp/vpcflow/manifest.yml b/x-pack/filebeat/module/gcp/vpcflow/manifest.yml
index 1f67548e0..a11478136 100644
--- a/x-pack/filebeat/module/gcp/vpcflow/manifest.yml
+++ b/x-pack/filebeat/module/gcp/vpcflow/manifest.yml
@@ -16,6 +16,7 @@ var:
   - name: tags
     default: [forwarded]
   - name: internal_networks
+    default: [203.0.0.0/8]

diff --git a/x-pack/filebeat/module/gcp/vpcflow/test/vpc-flow-log-entries.json.log-expected.json b/x-pack/filebeat/modul
e/gcp/vpcflow/test/vpc-flow-log-entries.json.log-expected.json
index b9d0250b9..813ce69da 100644
--- a/x-pack/filebeat/module/gcp/vpcflow/test/vpc-flow-log-entries.json.log-expected.json
+++ b/x-pack/filebeat/module/gcp/vpcflow/test/vpc-flow-log-entries.json.log-expected.json
@@ -32,7 +32,7 @@
         "log.offset": 0,
         "network.bytes": 1776,
         "network.community_id": "1:Eav+HA4T0zQk7MDzMdHH6Hhsx2A=",
-        "network.direction": "outbound",
+        "network.direction": "inbound",
         "network.iana_number": "6",
         "network.name": "default",
         "network.packets": 7,

[leehinman]

for firewall, this would be a breaking change, but should we switch the default inbound/outbound to ingress/egress? the original event uses ingress/egress

[andrewstucki]

@leehinman so, I'd stick with inbound/outbound for this one since it seems to be less about data coming into a firewall host as a final destination (i.e. ingress) v. inbound traffic into a network perimeter that the firewall is sitting in front of.