Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Winlogbeat] Add support for FileDelete events (event id 23) to sysmon module #18340

Merged

Conversation

marc-gr
Copy link
Contributor

@marc-gr marc-gr commented May 7, 2020

What does this PR do?

FileDelete events were added in Sysmon v11, this PR adds support for FileDelete events (event id 23) to sysmon module.

Why is it important?

Prior to this change processing such events lead to an 'unexpected sysmon event_id' error message.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

TODO

  • Diff the schema XML from v10 to v11 and make updates.
  • Empty strings are replaced with “-“ to work around a WEF bug
  • Handle Event ID 23
  • Add .evtx test files from v11.

Related issues

Closes #18094

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 7, 2020
@marc-gr marc-gr added Team:SIEM Winlogbeat and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 7, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@marc-gr marc-gr changed the title Add support for FileDelete events (event id 23) to sysmon module [Winlogbeat] Add support for FileDelete events (event id 23) to sysmon module May 7, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 7, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview stats

Expand to view the summary

Build stats

Test stats 🧪

Test Results
Failed 0
Passed 263
Skipped 1
Total 264

to: "process.executable",
},
{
from: "winlog.event_data.Archived",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The process namespace is "managed" by ECS. If archived and is_executable are not part of ECS then we should not add them to process.

Either leave them as-is or copy them over to a sysmon namespace like sysmon.is_archived. Then add an entry to this file for the field. https://github.com/elastic/beats/blob/master/x-pack/winlogbeat/module/sysmon/_meta/fields.yml The benefit of copying them is that we can set the correct data type. Leaving them in winlog.event_data means they will be a keyword in Elasticsearch rather than boolean.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added them to sysmon.file.archived and sysmon.file.is_executable 👍

@marc-gr marc-gr force-pushed the winlogbeat-add_sysmon-v11_support branch 2 times, most recently from 8dbb6cc to 0eab793 Compare May 7, 2020 15:36
FileDelete events were added in Sysmon v11.
Prior to this change processing such events lead to an 'unexpected sysmon event_id' error message.

Closes elastic#18094
@marc-gr marc-gr force-pushed the winlogbeat-add_sysmon-v11_support branch from 0eab793 to 69510ad Compare May 7, 2020 15:39
@marc-gr marc-gr marked this pull request as ready for review May 7, 2020 15:55
@marc-gr marc-gr requested a review from a team as a code owner May 7, 2020 15:55
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Congrats on your first Beats PR 🍾 .

As a follow-up pull request, I noticed a few issues with the module as a whole that we should address.

  • Set related.hash.
  • Set file.extension/name/directory.
  • hash.* is not part of ECS. It should be used as file.hash.* or process.hash.*. We can't delete the existing hash.* fields until 8.0, so for 7.x we could populate them both. And then do a breaking change for 8.0 where we drop hash.*.

@andrewkroh andrewkroh merged commit 0a327bb into elastic:master May 7, 2020
@andrewkroh andrewkroh added the needs_backport PR is waiting to be backported to other branches. label May 7, 2020
andrewkroh pushed a commit to andrewkroh/beats that referenced this pull request May 7, 2020
…stic#18340)

FileDelete events were added in Sysmon v11.
Prior to this change processing such events lead to an 'unexpected sysmon event_id' error message.

Closes elastic#18094

(cherry picked from commit 0a327bb)
@andrewkroh andrewkroh added v7.8.0 and removed needs_backport PR is waiting to be backported to other branches. labels May 7, 2020
@andrewkroh
Copy link
Member

@marc-gr I opened #18364 to track those additional changes. And I disucssed the rule.name: - with @andrewstucki and we think the field should be dropped so I added that to the issue.

Normally you will merge the PRs after they are approved and green on CI, but I wanted to get this into the 7.8 branch asap so that it can be included in 7.8.0. This still needs to be backported into the 7.x branch. Can you please use the dev-tools/cherrypick_pr script to do that (./dev-tools/cherrypick_pr --create_pr 7.x 18340 0a327bb). Read the script b/c you need to setup a github auth token first.

andrewkroh added a commit that referenced this pull request May 7, 2020
) (#18363)

FileDelete events were added in Sysmon v11.
Prior to this change processing such events lead to an 'unexpected sysmon event_id' error message.

Closes #18094

(cherry picked from commit 0a327bb)

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>
@marc-gr marc-gr deleted the winlogbeat-add_sysmon-v11_support branch May 8, 2020 06:21
marc-gr added a commit to marc-gr/beats that referenced this pull request May 8, 2020
…stic#18340)

FileDelete events were added in Sysmon v11.
Prior to this change processing such events lead to an 'unexpected sysmon event_id' error message.

Closes elastic#18094

(cherry picked from commit 0a327bb)
marc-gr added a commit that referenced this pull request May 8, 2020
) (#18371)

FileDelete events were added in Sysmon v11.
Prior to this change processing such events lead to an 'unexpected sysmon event_id' error message.

Closes #18094

(cherry picked from commit 0a327bb)
@marc-gr marc-gr restored the winlogbeat-add_sysmon-v11_support branch May 14, 2020 13:15
@marc-gr marc-gr deleted the winlogbeat-add_sysmon-v11_support branch May 14, 2020 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Winlogbeat] Update Sysmon module for v11.0
3 participants