Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SIEM][CEF] Add support for Check Point devices #16907

Merged
merged 11 commits into from
Mar 18, 2020

Conversation

adriansr
Copy link
Contributor

@adriansr adriansr commented Mar 9, 2020

This PR adds a new ingest pipeline and fields to populate from Check Point CEF logs.

Additional changes to the ECS processor:

  • Make CEF key translation case-insensitive.
  • Add missing CEF extension c6a2/deviceCustomIPv6Address2.
  • Add deprecated flexNumber[12] extensions. Used by Check Point. Dropped from CEF after V23.
  • Change case of DeviceCustomNumber2 to align with the other fields.

Closes #16041

@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

- name: version
type: keyword

# TODO: Update to ECS 1.5 and remove.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will remove and rebase once #16900 is merged

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look great 👍

Check Point CEF extensions are mapped as follows:
[options="header"]
|=======================================================================================================================
| CEF Extension | CEF Label value | ECS Fields | Non-ECS Field |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This table is awesome. Helps me a lot in digesting the pipeline.

Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very Nice.

Thank you for adding comments about the field names and Checkpoint use being "unusual". The sourceNtDomain one was throwing me :-)

Do you think we could add some ECS categorization fields?
event.kind: most things would be events. could alerts be events that have rules and high risk_score or severity?
event.category: I think if the event has IP addresses we can add network, I'm not sure if we have enough info for malware or intrustion_detection
event.type: could we map event.action to allowed/denied for the network events?

@adriansr adriansr force-pushed the feature_checkpoint_cef branch 2 times, most recently from 2eca571 to e80d90f Compare March 13, 2020 16:33
@adriansr
Copy link
Contributor Author

@leehinman , I've attempted some categorisation by looking at the fields present. However, it is difficult to do anything acceptable without sample logs (I couldn't find any).

For example for event.action (or outcome) I only have the value "Allowed" in the sample logs but I don't know what other possible values are there so I'm reluctant to map everything else to denied.

Have a look at the changes, I'm open to suggestions

@leehinman
Copy link
Contributor

@leehinman , I've attempted some categorisation by looking at the fields present. However, it is difficult to do anything acceptable without sample logs (I couldn't find any).

For example for event.action (or outcome) I only have the value "Allowed" in the sample logs but I don't know what other possible values are there so I'm reluctant to map everything else to denied.

Have a look at the changes, I'm open to suggestions

changes look great. hopefully we can get more examples in the future.

There's some case inconsistency in CEF docs (i.e. C6a4Label). Better to
ignore case when mapping keys to full names.
This adds:
 - `deviceCustomIPv6Address2(Label)`: Only 1, 3 and 4 were expected.
 - `flexNumber[12](Label)`: These two alternative custom numbers were
   dropped after V23 of the spec, but still used by some vendors.

[Maybe unnecessary] changes:

 - Changed the case of `DeviceCustomNumber2` from uppercase as
   documented) to lowercase to align with the other fields.
This adds a new ingest pipeline and fields to populate from Check Point
CEF logs.

Closes elastic#16041
adriansr added a commit to adriansr/beats that referenced this pull request Mar 19, 2020
* Make CEF key name mapping case-insensitive

There's some case inconsistency in CEF docs (i.e. C6a4Label). Better to
ignore case when mapping keys to full names.

* Add missing custom CEF extensions

This adds:
 - `deviceCustomIPv6Address2(Label)`: Only 1, 3 and 4 were expected.
 - `flexNumber[12](Label)`: These two alternative custom numbers were
   dropped after V23 of the spec, but still used by some vendors.

[Maybe unnecessary] changes:

 - Changed the case of `DeviceCustomNumber2` from uppercase as
   documented) to lowercase to align with the other fields.

* CEF module: Support Check Point devices

This adds a new ingest pipeline and fields to populate from Check Point
CEF logs.

Closes elastic#16041

(cherry picked from commit f6fde2e)
adriansr added a commit that referenced this pull request Mar 20, 2020
…vices (#17111)

* Make CEF key name mapping case-insensitive

There's some case inconsistency in CEF docs (i.e. C6a4Label). Better to
ignore case when mapping keys to full names.

* Add missing custom CEF extensions

This adds:
 - `deviceCustomIPv6Address2(Label)`: Only 1, 3 and 4 were expected.
 - `flexNumber[12](Label)`: These two alternative custom numbers were
   dropped after V23 of the spec, but still used by some vendors.

[Maybe unnecessary] changes:

 - Changed the case of `DeviceCustomNumber2` from uppercase as
   documented) to lowercase to align with the other fields.

* CEF module: Support Check Point devices

This adds a new ingest pipeline and fields to populate from Check Point
CEF logs.

Closes #16041

(cherry picked from commit f6fde2e)
adriansr added a commit that referenced this pull request Apr 14, 2020
This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related #16907 #17682
adriansr added a commit to adriansr/beats that referenced this pull request Apr 14, 2020
This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related elastic#16907 elastic#17682

(cherry picked from commit 4f6da4f)
adriansr added a commit to adriansr/beats that referenced this pull request Apr 14, 2020
This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related elastic#16907 elastic#17682

(cherry picked from commit 4f6da4f)
adriansr added a commit that referenced this pull request Apr 15, 2020
)

This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related #16907 #17682

(cherry picked from commit 4f6da4f)
adriansr added a commit that referenced this pull request Apr 15, 2020
)

This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related #16907 #17682

(cherry picked from commit 4f6da4f)
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
… (elastic#17712)

This PR makes some changes to CEF module's custom mappings for Check Point
devices to ensure compatibility with the upcoming checkpoint module.

Check Point has its custom log format, for which a new module is being
prepared. The idea behind this new module as well as CEF custom mappings for
Check Point (this PR), is to use ECS whenever possible and map the rest
under checkpoint.* using the original field name from Check Point.

In the original PR for CEF, a few mistakes had been done in field names and
types. Also taking the opportunity to change some ECS mappings.

Related elastic#16907 elastic#17682

(cherry picked from commit ddb92ca)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Filebeat] Add module for Checkpoint Firewall
4 participants