[Filebeat] Add network direction processor to Zeek and Suricata modules

[legoguy1000]

What does this PR do?

Adds the add_network_direction processor to the Zeek & Suricata module filesets using the internal_networks variable. The internal_networks variable is set to default to [ private ] and the add_network_direction will only run if that varaible is defined.

Why is it important?

The add_network_direction process adds the network.direction to documents which allows users to easily filter on traffic based on direction without needing to know specific IP subnets.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

cd beats/x-pack/filebeat
GENERATE=true TESTING_FILEBEAT_MODULES=suricata,zeek mage -v pythonIntegTest

Related issues

N/A

Use cases

Screenshots

Logs

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: P1llus commented: /test

• Start Time: 2021-06-29T11:55:40.436+0000

• Duration: 115 min 44 sec

• Commit: 28e08cc

Test stats 🧪

Test	Results
Failed	0
Passed	14229
Skipped	2311
Total	16540

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	14229
Skipped	2311
Total	16540

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[legoguy1000]

@andrewkroh I think I'm almost done with this PR but I think I found a bug and want to check with you first. I noticed on the Snort and Sonicwall JS scripts, for source and dest IP it was using the fld_append function. This causes the field to be an array/list even if its a single value. When I tried to add the add_network_direction processor, it didn't work until I change the JS function to fld_set so its just a single value. My thought is that if it is an array/list, the processor should just use the first value. Thoughts??

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-network-directions upstream/add-network-directions
git merge upstream/master
git push upstream add-network-directions

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-network-directions upstream/add-network-directions
git merge upstream/master
git push upstream add-network-directions

[P1llus]

run tests

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-network-directions upstream/add-network-directions
git merge upstream/master
git push upstream add-network-directions

[P1llus]

/test

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-network-directions upstream/add-network-directions
git merge upstream/master
git push upstream add-network-directions

[P1llus]

/test

[P1llus]

Is this okay to merge now @andrewkroh? We moved everything out of ingest pipelines and into beats, so that they offer backwards compatibility as well.

[andrewkroh]

Modifying snort and sonicwall will cause issues in the future since those are generated modules from https://github.com/adriansr/nwdevice2filebeat. This feature could be added to the generator then all of the RSA would get this processor. IMO I think we should remove the changes from those two modules and proceed with this PR.

[andrewkroh]

Why set and not append? Normally related.ip is an array. This seems unrelated to adding network direction.

[legoguy1000]

i don't recall removing "daddr_v6": {convert: to_ip, to:[{field: "destination.ip", setter: fld_append},{field: "related.ip", setter: fld_append}]},. That is probably an accident. As for the change from {field: "destination.ip", setter: fld_append} to {field: "destination.ip", setter: fld_set}, is because the network_direction processor requires a string field not an array.

[legoguy1000]

The append for related.ip is unchanged.

[legoguy1000]

I am reverting the changes for Snort and Sonicwall.

[legoguy1000]

removed

[andrewkroh]

Thanks for pointing out at source.ip and destination.ip should be scalars. I think we should make that change just not in this PR. Would you please open an issue for this? It does look like you reverted most of the snort/sonicwall changes, but a small part of snort is still changed.

[P1llus]

/test