[Filebeat] Improve ECS categorization field mappings for azure module #19376

leehinman · 2020-06-24T18:14:35Z

What does this PR do?

Improve ECS categorization field mappings for azure module.
Specifically:

activitylogs
- add azure.activitylogs.result_type
- set default_field: false
- populate event.outcome with allowed values
- set event.action
- populate event.category with allowed values
- set event.kind
- set event.type
- add support tickets example
- add geoip for source.ip
- add AS info for source.ip
- add user.name
- add user.full_name
- add user.domain
auditlogs
- set default_field: false
- add azure.auditlogs.category
- populate event.outcome with allowed values
- set event.action
- set event.kind
signinlogs
- set default_field: false
- set event.action
- populate event.category with allowed values
- set event.type
- populate event.outcome with allowed values
- add azure.signinlogs.category
- add azure.signinlogs.result_type
- set user.name
- set user.domain
- set user.full_name
- set user.id
- add geoip for source.ip
- add AS info for source.ip

Why is it important?

ECS categorization fields allow cross correlation between filesets.

Checklist

My code follows the style guidelines of this project
~~- [ ] I have commented my code, particularly in hard-to-understand areas~~
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

TESTING_FILEBEAT_MODULES=azure mage -v pythonIntegTest

Related issues

Closes 16155

elasticmachine · 2020-06-24T18:14:38Z

Pinging @elastic/siem (Team:SIEM)

elasticmachine · 2020-06-24T18:19:49Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #19376 updated]
Start Time: 2020-07-07T21:08:03.894+0000
Duration: 56 min 5 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4213
Skipped	677
Total	4890

andrewkroh · 2020-06-28T20:43:07Z

run tests

andrewkroh · 2020-06-28T20:49:00Z

x-pack/filebeat/module/azure/signinlogs/test/signinlogs.log-expected.json

 "event.dataset": "azure.signinlogs",
- "event.duration": 0.0,


That type checking PR you opened is looking like a very good idea 😄 .

- activitylogs + convert pipeline to yml - auditlogs + convert pipeline to yml - signinlogs + convert pipeline to yml Closes elastic#16155

- activitylogs + add azure.activitylogs.result_type + set default_field: false + populate event.outcome with allowed values + set event.action + populate event.category with allowed values + set event.kind + set event.type + add support tickets example + add geoip for source.ip + add AS info for source.ip + add user.name + add user.full_name + add user.domain - auditlogs + set default_field: false + add azure.auditlogs.category + populate event.outcome with allowed values + set event.action + set event.kind - signinlogs + set default_field: false + set event.action + populate event.category with allowed values + set event.type + populate event.outcome with allowed values + add azure.signinlogs.category + add azure.signinlogs.result_type + set user.name + set user.domain + set user.full_name + set user.id + add geoip for source.ip + add AS info for source.ip

narph

Might worth checking against the existing dashboards, I think they contain some of the fields that have been replaced

leehinman · 2020-07-07T21:08:48Z

@narph tested on azure and with changes to the dashboards everything looks ok. anything else you can think of to check?

narph · 2020-07-08T11:25:07Z

@narph tested on azure and with changes to the dashboards everything looks ok. anything else you can think of to check?

LGTM, thanks

…elastic#19376) * Improve ECS categorization field mappings in azure module - activitylogs + convert pipeline to yml + add azure.activitylogs.result_type + set default_field: false + populate event.outcome with allowed values + set event.action + populate event.category with allowed values + set event.kind + set event.type + add support tickets example + add geoip for source.ip + add AS info for source.ip + add user.name + add user.full_name + add user.domain + update dashboards - auditlogs + convert pipeline to yml + set default_field: false + add azure.auditlogs.category + populate event.outcome with allowed values + set event.action + set event.kind + update dashboards - signinlogs + convert pipeline to yml + set default_field: false + set event.action + populate event.category with allowed values + set event.type + populate event.outcome with allowed values + add azure.signinlogs.category + add azure.signinlogs.result_type + set user.name + set user.domain + set user.full_name + set user.id + add geoip for source.ip + add AS info for source.ip + update dashboards Closes elastic#16155 (cherry picked from commit 00a274e)

…#19376) (#19737) * Improve ECS categorization field mappings in azure module - activitylogs + convert pipeline to yml + add azure.activitylogs.result_type + set default_field: false + populate event.outcome with allowed values + set event.action + populate event.category with allowed values + set event.kind + set event.type + add support tickets example + add geoip for source.ip + add AS info for source.ip + add user.name + add user.full_name + add user.domain + update dashboards - auditlogs + convert pipeline to yml + set default_field: false + add azure.auditlogs.category + populate event.outcome with allowed values + set event.action + set event.kind + update dashboards - signinlogs + convert pipeline to yml + set default_field: false + set event.action + populate event.category with allowed values + set event.type + populate event.outcome with allowed values + add azure.signinlogs.category + add azure.signinlogs.result_type + set user.name + set user.domain + set user.full_name + set user.id + add geoip for source.ip + add AS info for source.ip + update dashboards Closes #16155 (cherry picked from commit 00a274e)

kevinserafin · 2020-07-19T21:53:05Z

Pardon the question but I was attempting to integrate some of these pipeline changes in now and I am wondering if I am not understanding how the error_code set processors are supposed to function.

set:
field: event.outcome
value: success
if: "ctx?.azure?.signinlogs?.properties?.status?.error_code == null || ctx.azure.signinlogs.properties.status.error_code == 0"
set:
field: event.outcome
value: failure
if: "ctx?.azure?.signinlogs?.properties?.status?.error_code != null || ctx.azure.signinlogs.properties.status.error_code > 0"

So if I pass a zero for azure.signinlogs.properties.status.error_code, is it not evaluated as true for both the success and failure processor? All of my testing kept events showing up as failures so I started poking around, changing the failure processor to Boolean And seemed to fix it but I wanted to see if I was missing something first.

leehinman · 2020-07-27T20:53:27Z

@kevinserafin you are correct that was a bug. PR #20254 should fix

…elastic#19376) * Improve ECS categorization field mappings in azure module - activitylogs + convert pipeline to yml + add azure.activitylogs.result_type + set default_field: false + populate event.outcome with allowed values + set event.action + populate event.category with allowed values + set event.kind + set event.type + add support tickets example + add geoip for source.ip + add AS info for source.ip + add user.name + add user.full_name + add user.domain + update dashboards - auditlogs + convert pipeline to yml + set default_field: false + add azure.auditlogs.category + populate event.outcome with allowed values + set event.action + set event.kind + update dashboards - signinlogs + convert pipeline to yml + set default_field: false + set event.action + populate event.category with allowed values + set event.type + populate event.outcome with allowed values + add azure.signinlogs.category + add azure.signinlogs.result_type + set user.name + set user.domain + set user.full_name + set user.id + add geoip for source.ip + add AS info for source.ip + update dashboards Closes elastic#16155

leehinman added enhancement Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. Team:SIEM labels Jun 24, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 24, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 24, 2020

leehinman requested a review from narph June 24, 2020 18:18

leehinman changed the title ~~[Filebeat] 16155 azure ecs 1.4~~ [Filebeat] Improve ECS categorization field mappings for azure module Jun 25, 2020

andrewkroh approved these changes Jun 28, 2020

View reviewed changes

leehinman force-pushed the 16155_azure_ecs_1.4 branch from d52bbe8 to f4ab282 Compare July 1, 2020 14:50

leehinman added 3 commits July 1, 2020 13:22

Improve ECS categorization field mappings in azure module

a5e490e

- activitylogs + convert pipeline to yml - auditlogs + convert pipeline to yml - signinlogs + convert pipeline to yml Closes elastic#16155

Update changelog

80e67c8

leehinman force-pushed the 16155_azure_ecs_1.4 branch from f4ab282 to 80e67c8 Compare July 1, 2020 18:22

narph reviewed Jul 6, 2020

View reviewed changes

update dashboards

bd9066b

narph approved these changes Jul 8, 2020

View reviewed changes

leehinman merged commit 00a274e into elastic:master Jul 8, 2020

leehinman mentioned this pull request Jul 8, 2020

Cherry-pick #19376 to 7.x: [Filebeat] Improve ECS categorization field mappings for azure module #19737

Merged

5 tasks

leehinman added v7.9.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jul 8, 2020

This was referenced Sep 23, 2020

Azure module event.category incorrectly mapped #21259

Closed

[Rule Tuning] Update Azure rules to remove event.category elastic/detection-rules#333

Closed

[Rule Tuning] Update event.category for Azure rules elastic/detection-rules#335

Merged

leehinman deleted the 16155_azure_ecs_1.4 branch October 5, 2020 19:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Improve ECS categorization field mappings for azure module #19376

[Filebeat] Improve ECS categorization field mappings for azure module #19376

leehinman commented Jun 24, 2020 •

edited

Loading

elasticmachine commented Jun 24, 2020

elasticmachine commented Jun 24, 2020 •

edited

Loading

Build stats

Test stats 🧪

andrewkroh commented Jun 28, 2020

andrewkroh Jun 28, 2020

narph left a comment

leehinman commented Jul 7, 2020

narph commented Jul 8, 2020

kevinserafin commented Jul 19, 2020

leehinman commented Jul 27, 2020

[Filebeat] Improve ECS categorization field mappings for azure module #19376

[Filebeat] Improve ECS categorization field mappings for azure module #19376

Conversation

leehinman commented Jun 24, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Jun 24, 2020

elasticmachine commented Jun 24, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

andrewkroh commented Jun 28, 2020

andrewkroh Jun 28, 2020

Choose a reason for hiding this comment

narph left a comment

Choose a reason for hiding this comment

leehinman commented Jul 7, 2020

narph commented Jul 8, 2020

kevinserafin commented Jul 19, 2020

leehinman commented Jul 27, 2020

leehinman commented Jun 24, 2020 •

edited

Loading

elasticmachine commented Jun 24, 2020 •

edited

Loading