[FileBeat] GCP module enhancement - Populate orchestrator.* fields for K8S logs

[TonioRyo]

What does this PR do?

This PR add the ability to populate orchestrator.* fields for GCP K8S logs. The cluster_name field from the original message was not kept in the logs, this field is important to analyze security events. I also tried to populate other orchestrator.* fields with information available in other fields of the original message.

Why is it important?

The cluster_name field from the original message was not kept in the logs, this field is important to analyze security events.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #25368 updated

• Start Time: 2021-06-01T11:52:52.571+0000

• Duration: 102 min 53 sec

• Commit: 9b88e3d

Test stats 🧪

Test	Results
Failed	0
Passed	7194
Skipped	1193
Total	8387

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	7194
Skipped	1193
Total	8387

[TonioRyo]

A discussion is opened on discuss.elastic.co regarding this PR: https://discuss.elastic.co/t/gcp-module-add-cluster-name-for-gke-k8s-io-logs/271278

[legoguy1000]

This should also be done for the other modules that can get K8S logs.

[legoguy1000]

You also need to add sample K8S logs to the x-pack/filebeat/module/gcp/audit/test/audit-log-entries.json.log file and then run the below to update the generated data.

cd beats/x-pack/filebeat
GENERATE=true TESTING_FILEBEAT_MODULES=gcp TESTING_FILEBEAT_FILESETS=audit mage -v pythonIntegTest

[TonioRyo]

You also need to add sample K8S logs to the x-pack/filebeat/module/gcp/audit/test/audit-log-entries.json.log file and then run the below to update the generated data.

cd beats/x-pack/filebeat
GENERATE=true TESTING_FILEBEAT_MODULES=gcp TESTING_FILEBEAT_FILESETS=audit mage -v pythonIntegTest

I tried to do that but got an error FAILED tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_0_gcp - Exception: Key 'orchestrator.cluster.name' found in event is not documented!.
I think it's because the orchestrator.* field set is not available in ECS 1.9, it will only be available in ECS 1.10, and FileBeat is not yet build with this ECS version.
Do you have any direction on how I should handle that ?

[TonioRyo]

This should also be done for the other modules that can get K8S logs.

Agreed but I do not have access to K8S systems from other providers (Azure Kubernetes Service, Amazon EKS, on premise K8S...) to do that.

[legoguy1000]

You also need to add sample K8S logs to the x-pack/filebeat/module/gcp/audit/test/audit-log-entries.json.log file and then run the below to update the generated data.

cd beats/x-pack/filebeat
GENERATE=true TESTING_FILEBEAT_MODULES=gcp TESTING_FILEBEAT_FILESETS=audit mage -v pythonIntegTest

I tried to do that but got an error FAILED tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_0_gcp - Exception: Key 'orchestrator.cluster.name' found in event is not documented!.
I think it's because the orchestrator.* field set is not available in ECS 1.9, it will only be available in ECS 1.10, and FileBeat is not yet build with this ECS version.
Do you have any direction on how I should handle that ?

Thats a good question. for the development team. I've never tried to use fields in a future ECS spec version.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b master upstream/master
git merge upstream/master
git push upstream master

[sorantis]

cc @kaiyan-sheng, @jsoriano, @exekias. Looks like we need to align with the Inventory schema work here.

[kaiyan-sheng]

@sorantis Thanks for pinging here! orchestrator.* fields will be published in ECS 1.10 and the three fields we defined for orchestrator in inventory schema already matches the ones in ECS.

[andrewkroh]

This change will also need applied to elastic/integrations.

[ChrsMark]

Thanks for your contribution @TonioRyo! Regarding the ECS version how about trying to update it at

beats/x-pack/filebeat/module/gcp/audit/config/input.yml

Line 37 in 83f248e

ecs.version: 1.9.0

?

@kaiyan-sheng @andrewkroh this change looks good to me regarding the orchestrator.* fields that it adds. I'm ok with merging this when golden files are updated too. We will need to open a separate issue for elastic/integrations too.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b master upstream/master
git merge upstream/master
git push upstream master

[P1llus]

@TonioRyo Would you be able to merge with master because I added the ECS fields to the beats definition today. If you could also bump the ECS version to 1.10.0 that would be great!

[TonioRyo]

Thanks @P1llus, I was able to generate the expected results based on sample logs thanks to your work!

[P1llus]

Removing integration sync as I added the same changes to packages here: elastic/integrations#1045

[P1llus]

Awesome @TonioRyo . Il check to see if someone can merge it then!

[P1llus]

jenkins run tests

[ChrsMark]

LGTM, thanks!