[SIEM][CEF] Add support for Check Point devices #16907

adriansr · 2020-03-09T17:43:14Z

This PR adds a new ingest pipeline and fields to populate from Check Point CEF logs.

Additional changes to the ECS processor:

Make CEF key translation case-insensitive.
Add missing CEF extension c6a2/deviceCustomIPv6Address2.
Add deprecated flexNumber[12] extensions. Used by Check Point. Dropped from CEF after V23.
Change case of DeviceCustomNumber2 to align with the other fields.

Closes #16041

elasticmachine · 2020-03-09T17:43:17Z

Pinging @elastic/siem (Team:SIEM)

adriansr · 2020-03-09T17:46:56Z

x-pack/filebeat/module/cef/log/_meta/fields.yml

+    - name: version
+      type: keyword
+
+# TODO: Update to ECS 1.5 and remove.


Will remove and rebase once #16900 is merged

andrewkroh

Look great 👍

andrewkroh · 2020-03-10T22:17:45Z

filebeat/docs/modules/cef.asciidoc

+Check Point CEF extensions are mapped as follows:
+[options="header"]
+|=======================================================================================================================
+| CEF Extension              | CEF Label value             | ECS Fields               | Non-ECS Field                  |


This table is awesome. Helps me a lot in digesting the pipeline.

leehinman

Very Nice.

Thank you for adding comments about the field names and Checkpoint use being "unusual". The sourceNtDomain one was throwing me :-)

Do you think we could add some ECS categorization fields?
event.kind: most things would be events. could alerts be events that have rules and high risk_score or severity?
event.category: I think if the event has IP addresses we can add network, I'm not sure if we have enough info for malware or intrustion_detection
event.type: could we map event.action to allowed/denied for the network events?

adriansr · 2020-03-13T16:37:30Z

@leehinman , I've attempted some categorisation by looking at the fields present. However, it is difficult to do anything acceptable without sample logs (I couldn't find any).

For example for event.action (or outcome) I only have the value "Allowed" in the sample logs but I don't know what other possible values are there so I'm reluctant to map everything else to denied.

Have a look at the changes, I'm open to suggestions

leehinman · 2020-03-13T22:11:50Z

@leehinman , I've attempted some categorisation by looking at the fields present. However, it is difficult to do anything acceptable without sample logs (I couldn't find any).

For example for event.action (or outcome) I only have the value "Allowed" in the sample logs but I don't know what other possible values are there so I'm reluctant to map everything else to denied.

Have a look at the changes, I'm open to suggestions

changes look great. hopefully we can get more examples in the future.

There's some case inconsistency in CEF docs (i.e. C6a4Label). Better to ignore case when mapping keys to full names.

This adds: - `deviceCustomIPv6Address2(Label)`: Only 1, 3 and 4 were expected. - `flexNumber[12](Label)`: These two alternative custom numbers were dropped after V23 of the spec, but still used by some vendors. [Maybe unnecessary] changes: - Changed the case of `DeviceCustomNumber2` from uppercase as documented) to lowercase to align with the other fields.

This adds a new ingest pipeline and fields to populate from Check Point CEF logs. Closes elastic#16041

* Make CEF key name mapping case-insensitive There's some case inconsistency in CEF docs (i.e. C6a4Label). Better to ignore case when mapping keys to full names. * Add missing custom CEF extensions This adds: - `deviceCustomIPv6Address2(Label)`: Only 1, 3 and 4 were expected. - `flexNumber[12](Label)`: These two alternative custom numbers were dropped after V23 of the spec, but still used by some vendors. [Maybe unnecessary] changes: - Changed the case of `DeviceCustomNumber2` from uppercase as documented) to lowercase to align with the other fields. * CEF module: Support Check Point devices This adds a new ingest pipeline and fields to populate from Check Point CEF logs. Closes elastic#16041 (cherry picked from commit f6fde2e)

…vices (#17111) * Make CEF key name mapping case-insensitive There's some case inconsistency in CEF docs (i.e. C6a4Label). Better to ignore case when mapping keys to full names. * Add missing custom CEF extensions This adds: - `deviceCustomIPv6Address2(Label)`: Only 1, 3 and 4 were expected. - `flexNumber[12](Label)`: These two alternative custom numbers were dropped after V23 of the spec, but still used by some vendors. [Maybe unnecessary] changes: - Changed the case of `DeviceCustomNumber2` from uppercase as documented) to lowercase to align with the other fields. * CEF module: Support Check Point devices This adds a new ingest pipeline and fields to populate from Check Point CEF logs. Closes #16041 (cherry picked from commit f6fde2e)

This PR makes some changes to CEF module's custom mappings for Check Point devices to ensure compatibility with the upcoming checkpoint module. Check Point has its custom log format, for which a new module is being prepared. The idea behind this new module as well as CEF custom mappings for Check Point (this PR), is to use ECS whenever possible and map the rest under checkpoint.* using the original field name from Check Point. In the original PR for CEF, a few mistakes had been done in field names and types. Also taking the opportunity to change some ECS mappings. Related #16907 #17682

This PR makes some changes to CEF module's custom mappings for Check Point devices to ensure compatibility with the upcoming checkpoint module. Check Point has its custom log format, for which a new module is being prepared. The idea behind this new module as well as CEF custom mappings for Check Point (this PR), is to use ECS whenever possible and map the rest under checkpoint.* using the original field name from Check Point. In the original PR for CEF, a few mistakes had been done in field names and types. Also taking the opportunity to change some ECS mappings. Related elastic#16907 elastic#17682 (cherry picked from commit 4f6da4f)

) This PR makes some changes to CEF module's custom mappings for Check Point devices to ensure compatibility with the upcoming checkpoint module. Check Point has its custom log format, for which a new module is being prepared. The idea behind this new module as well as CEF custom mappings for Check Point (this PR), is to use ECS whenever possible and map the rest under checkpoint.* using the original field name from Check Point. In the original PR for CEF, a few mistakes had been done in field names and types. Also taking the opportunity to change some ECS mappings. Related #16907 #17682 (cherry picked from commit 4f6da4f)

… (elastic#17712) This PR makes some changes to CEF module's custom mappings for Check Point devices to ensure compatibility with the upcoming checkpoint module. Check Point has its custom log format, for which a new module is being prepared. The idea behind this new module as well as CEF custom mappings for Check Point (this PR), is to use ECS whenever possible and map the rest under checkpoint.* using the original field name from Check Point. In the original PR for CEF, a few mistakes had been done in field names and types. Also taking the opportunity to change some ECS mappings. Related elastic#16907 elastic#17682 (cherry picked from commit ddb92ca)

adriansr added review Team:SIEM labels Mar 9, 2020

adriansr requested a review from andrewkroh March 9, 2020 17:43

adriansr added enhancement module labels Mar 9, 2020

adriansr requested a review from leehinman March 9, 2020 17:44

adriansr commented Mar 9, 2020

View reviewed changes

andrewkroh approved these changes Mar 10, 2020

View reviewed changes

leehinman reviewed Mar 11, 2020

View reviewed changes

adriansr force-pushed the feature_checkpoint_cef branch 2 times, most recently from 2eca571 to e80d90f Compare March 13, 2020 16:33

adriansr added 11 commits March 18, 2020 11:07

Make CEF key name mapping case-insensitive

5aed151

There's some case inconsistency in CEF docs (i.e. C6a4Label). Better to ignore case when mapping keys to full names.

CEF module: Support Check Point devices

b2210e7

This adds a new ingest pipeline and fields to populate from Check Point CEF logs. Closes elastic#16041

Add docs

e3f9f86

Changelog entry

92a30c0

Add PR number

aae6071

Update decode_cef golden files

2db8b7a

Remove temporary ECS fields after rebase

ab3418c

observer.interface -> observer.ingress.interface

98f9f17

Try to populate some event categorisation fields

d17acce

Update golden files

81a7702

adriansr force-pushed the feature_checkpoint_cef branch from c43a127 to 81a7702 Compare March 18, 2020 10:07

adriansr merged commit f6fde2e into elastic:master Mar 18, 2020

adriansr mentioned this pull request Mar 19, 2020

Cherry-pick #16907 to 7.x: [SIEM][CEF] Add support for Check Point devices #17111

Merged

adriansr added the v7.7.0 label Mar 19, 2020

adriansr mentioned this pull request Apr 13, 2020

CEF CheckPoint: adjust fields for forward compatibility #17681

Merged

6 tasks

adriansr mentioned this pull request Apr 14, 2020

Cherry-pick #17681 to 7.x: CEF CheckPoint: adjust fields for forward compatibility #17711

Merged

6 tasks

adriansr mentioned this pull request Apr 14, 2020

Cherry-pick #17681 to 7.7: CEF CheckPoint: adjust fields for forward compatibility #17712

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM][CEF] Add support for Check Point devices #16907

[SIEM][CEF] Add support for Check Point devices #16907

adriansr commented Mar 9, 2020 •

edited by zube bot

Loading

elasticmachine commented Mar 9, 2020

adriansr Mar 9, 2020

andrewkroh left a comment

andrewkroh Mar 10, 2020

leehinman left a comment

adriansr commented Mar 13, 2020

leehinman commented Mar 13, 2020

[SIEM][CEF] Add support for Check Point devices #16907

[SIEM][CEF] Add support for Check Point devices #16907

Conversation

adriansr commented Mar 9, 2020 • edited by zube bot Loading

elasticmachine commented Mar 9, 2020

adriansr Mar 9, 2020

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh Mar 10, 2020

Choose a reason for hiding this comment

leehinman left a comment

Choose a reason for hiding this comment

adriansr commented Mar 13, 2020

leehinman commented Mar 13, 2020

adriansr commented Mar 9, 2020 •

edited by zube bot

Loading