Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat][New Module] Adding support for Microsoft Defender ATP #19197

Merged
merged 22 commits into from
Jul 14, 2020

Conversation

P1llus
Copy link
Member

@P1llus P1llus commented Jun 15, 2020

What does this PR do?

This PR adds the initial beta support for Microsoft Defender ATP.

Why is it important?

Adds new products to the supported list for filebeat

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Need to expand ECS event tagging
  • Need to add more testdata
  • Need to create some dashboards

Related issues

Reviewers:

The documentation for the incoming JSON fields is available here:
https://docs.microsoft.com/en-us/windows/security/threat-protection/microsoft-defender-atp/alerts

@P1llus P1llus added enhancement in progress Pull request is currently in progress. Filebeat Filebeat labels Jun 15, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 15, 2020
@P1llus P1llus added the Team:Integrations Label for the Integrations team label Jun 15, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 15, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 15, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19197 updated]

  • Start Time: 2020-07-14T12:19:12.312+0000

  • Duration: 37 min 0 sec

Test stats 🧪

Test Results
Failed 1
Passed 3162
Skipped 674
Total 3837

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat Mac OS X / test_inode_marker_based_identity_tracking – test_input.Test

    • Age: 1
    • Duration: 10.326
    • Error Details: Timeout waiting for 'cond' to be true. Waited 10 seconds.

Steps errors

Expand to view the steps failures

  • Name: Make -C filebeat testsuite

    • Description: make -C filebeat testsuite

    • Duration: 7 min 54 sec

    • Start Time: 2020-07-14T12:45:19.481+0000

    • log

  • Name: Mage update build test

    • Description: mage update build test

    • Duration: 5 min 49 sec

    • Start Time: 2020-07-14T12:47:32.384+0000

    • log

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Duration: 7 min 59 sec

    • Start Time: 2020-07-14T12:46:01.266+0000

    • log

  • Name: Recursively delete the current directory from the workspace

    • Description: script returned exit code 1

    • Duration: 0 min 11 sec

    • Start Time: 2020-07-14T12:55:39.910+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-14T12:54:47.187Z] [success] 0.28% test_harvester.Test.test_boms_2_utf_16le_bom: 0.4978s
[2020-07-14T12:54:47.187Z] [success] 0.28% test_container.Test.test_container_input_cri: 0.4976s
[2020-07-14T12:54:47.187Z] [success] 0.28% test_harvester.Test.test_boms_1_utf_16be_bom: 0.4975s
[2020-07-14T12:54:47.187Z] [success] 0.28% test_fields.Test.test_agent_name_custom: 0.4962s
[2020-07-14T12:54:47.187Z] [success] 0.28% test_harvester.Test.test_decode_error: 0.4949s
[2020-07-14T12:54:47.187Z] [success] 0.28% test_harvester.Test.test_close_removed: 0.4929s
[2020-07-14T12:54:47.187Z] [success] 0.27% test_crawler.Test.test_include_exclude_lines: 0.4915s
[2020-07-14T12:54:47.187Z] [success] 0.27% test_harvester.Test.test_close_eof: 0.4915s
[2020-07-14T12:54:47.187Z] [success] 0.27% test_harvester.Test.test_symlink_and_file: 0.4909s
[2020-07-14T12:54:47.187Z] [success] 0.27% test_deprecated.Test.test_input_type_deprecated: 0.4895s
[2020-07-14T12:54:47.187Z] [success] 0.27% test_fields.Test.test_custom_fields: 0.4877s
[2020-07-14T12:54:47.187Z] [success] 0.27% test_harvester.Test.test_exceed_buffer: 0.4871s
[2020-07-14T12:54:47.187Z] [success] 0.27% test_stdin.Test.test_stdin_eof: 0.4850s
[2020-07-14T12:54:47.187Z] [success] 0.27% test_fields.Test.test_custom_fields_under_root: 0.4845s
[2020-07-14T12:54:47.187Z] [success] 0.27% test_harvester.Test.test_symlinks_enabled: 0.4802s
[2020-07-14T12:54:47.187Z] [success] 0.26% test_input.Test.test_disable_recursive_glob: 0.4628s
[2020-07-14T12:54:47.187Z] [success] 0.25% test_tcp_tls.Test.test_tcp_tls_with_a_plain_text_socket: 0.4408s
[2020-07-14T12:54:47.187Z] [success] 0.24% test_generate.Test.test_generate_fileset: 0.4219s
[2020-07-14T12:54:47.187Z] [success] 0.23% test_crawler.Test.test_fetched_lines: 0.4196s
[2020-07-14T12:54:47.187Z] [success] 0.23% test_index_pattern.Test.test_export_index_pattern_migration: 0.4099s
[2020-07-14T12:54:47.187Z] [success] 0.23% test_crawler.Test.test_exclude_lines: 0.4033s
[2020-07-14T12:54:47.187Z] [success] 0.22% test_index_pattern.Test.test_export_index_pattern: 0.3969s
[2020-07-14T12:54:47.187Z] [success] 0.22% test_harvester.Test.test_boms_0_utf_8: 0.3900s
[2020-07-14T12:54:47.187Z] [success] 0.21% test_crawler.Test.test_include_lines: 0.3840s
[2020-07-14T12:54:47.187Z] [success] 0.21% test_harvester.Test.test_empty_lines_only: 0.3803s
[2020-07-14T12:54:47.187Z] [success] 0.20% test_shutdown.Test.test_once: 0.3622s
[2020-07-14T12:54:47.187Z] [success] 0.16% test_harvester.Test.test_debug_reader: 0.2904s
[2020-07-14T12:54:47.187Z] [success] 0.16% test_harvester.Test.test_ignore_symlink: 0.2880s
[2020-07-14T12:54:47.187Z] [success] 0.15% test_tcp_tls.Test.test_tcp_over_tls_mutual_auth_fails: 0.2768s
[2020-07-14T12:54:47.187Z] [success] 0.15% test_tcp_tls.Test.test_tcp_over_tls_and_verify_invalid_server_without_mutual_auth: 0.2713s
[2020-07-14T12:54:47.187Z] [success] 0.14% test_input.Test.test_shutdown_no_inputs: 0.2520s
[2020-07-14T12:54:47.187Z] [success] 0.14% test_multiline.Test.test_invalid_config: 0.2518s
[2020-07-14T12:54:47.187Z] [success] 0.14% test_input.Test.test_no_paths_defined: 0.2517s
[2020-07-14T12:54:47.187Z] [success] 0.12% test_reload_modules.Test.test_wrong_module_no_reload: 0.2136s
[2020-07-14T12:54:47.187Z] [success] 0.12% test_generate.Test.test_generate_module: 0.2122s
[2020-07-14T12:54:47.187Z] [success] 0.10% test_json.Test.test_config_no_msg_key_multiline: 0.1831s
[2020-07-14T12:54:47.187Z] [success] 0.10% test_keystore.TestKeystore.test_keystore_with_key_not_present: 0.1809s
[2020-07-14T12:54:47.187Z] [success] 0.10% test_json.Test.test_config_no_msg_key_filtering: 0.1800s
[2020-07-14T12:54:47.187Z] [success] 0.10% test_stdin.Test.test_stdin_is_exclusive: 0.1721s
[2020-07-14T12:54:47.187Z] [success] 0.08% test_deprecated.Test.test_invalid_config_with_removed_settings: 0.1469s
[2020-07-14T12:54:47.187Z] [success] 0.01% test_modules.load_fileset_test_cases: 0.0139s
[2020-07-14T12:54:47.187Z] ----------------------------------------------------------------------
[2020-07-14T12:54:47.187Z] Ran 316 tests in 179.060s
[2020-07-14T12:54:47.187Z] 
[2020-07-14T12:54:47.187Z] OK (SKIP=155)
[2020-07-14T12:54:47.446Z] >> python test: Unit Testing Complete
[2020-07-14T12:54:47.510Z] Recording test results
[2020-07-14T12:54:51.264Z] Stashed 2 file(s)
[2020-07-14T12:54:51.274Z] Archiving artifacts
[2020-07-14T12:55:50.834Z] Failed in branch Filebeat Mac OS X
[2020-07-14T12:55:50.961Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19197/src/github.com/elastic/beats
[2020-07-14T12:55:51.276Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-14T12:55:51.288Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19197/src/github.com/elastic/beats/Lint
[2020-07-14T12:55:51.370Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19197/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-14T12:55:51.445Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19197/src/github.com/elastic/beats/Filebeat-oss
[2020-07-14T12:55:51.531Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19197/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-14T12:55:51.615Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19197/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-07-14T12:55:51.709Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19197/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-14T12:55:51.804Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19197/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-14T12:55:52.169Z] + cat
[2020-07-14T12:55:52.169Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-14T12:55:52.169Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-14T12:55:58.822Z] runbld>>> runbld started
[2020-07-14T12:55:58.822Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-14T12:56:00.200Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-19197' in order of occurrence in the config (last value wins).
[2020-07-14T12:56:01.136Z] runbld>>> Debug logging enabled.
[2020-07-14T12:56:01.136Z] runbld>>> Storing result
[2020-07-14T12:56:01.395Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-14T12:56:01.396Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200714125601-C68CF278
[2020-07-14T12:56:01.396Z] runbld>>> Adding system facts.
[2020-07-14T12:56:02.333Z] runbld>>> Adding vcs info for the latest commit:  30c673106923f7568a148386810d41370e759a45
[2020-07-14T12:56:02.333Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-14T12:56:02.333Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-14T12:56:02.333Z] Processing JUnit reports with runbld...
[2020-07-14T12:56:02.333Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-14T12:56:02.592Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-14T12:56:02.592Z] runbld>>> DURATION: 17ms
[2020-07-14T12:56:02.592Z] runbld>>> STDOUT: 40 bytes
[2020-07-14T12:56:02.592Z] runbld>>> STDERR: 49 bytes
[2020-07-14T12:56:02.592Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-14T12:56:02.592Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-19197/src/github.com/elastic/beats
[2020-07-14T12:56:03.534Z] runbld>>> Storing build metadata: 
[2020-07-14T12:56:03.534Z] runbld>>> Adding test report.
[2020-07-14T12:56:03.534Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-19197/src/github.com/elastic/beats
[2020-07-14T12:56:04.472Z] runbld>>> Found 10 test output files
[2020-07-14T12:56:05.415Z] runbld>>> Test output logs contained: Errors: 1 Failures: 0 Tests: 3837 Skipped: 651
[2020-07-14T12:56:05.415Z] runbld>>> Storing result
[2020-07-14T12:56:05.415Z] runbld>>> FAILURES: 1
[2020-07-14T12:56:05.674Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-14T12:56:05.674Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200714125601-C68CF278
[2020-07-14T12:56:05.934Z] runbld>>> Email notification disabled by environment variable.
[2020-07-14T12:56:05.934Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-14T12:56:11.789Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-19197
[2020-07-14T12:56:12.004Z] [INFO] getVaultSecret: Getting secrets
[2020-07-14T12:56:12.077Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-14T12:56:12.796Z] + chmod 755 generate-build-data.sh
[2020-07-14T12:56:12.796Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19197/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19197/runs/9 FAILURE 2220224
[2020-07-14T12:56:12.796Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19197/runs/9/steps/?limit=10000 -o steps-info.json
[2020-07-14T12:56:13.722Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19197/runs/9/tests/?status=FAILED -o tests-errors.json
[2020-07-14T12:56:13.972Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19197/runs/9/log/ -o pipeline-log.txt

@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@P1llus P1llus added review and removed in progress Pull request is currently in progress. labels Jul 6, 2020
Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couple little changes.

I'll look at the golden files see if I can get those generate.

field: json.description
target_field: rule.description
ignore_missing: true
if: (ctx.json?.description).length() < 1020
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional. I'm wondering if we should copy the first 1019 chars, not skip if over 1019.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the sentence would just break off at that point, it would be half a sentence. I think that its just to ensure it never hits the limit as it was the only field that can in very niche usecases maybe include too much info. It wouldn't stop any of the events I have seen, and its more to drop it on ingest rather than creating an error in elasticsearch due to the field size limit (1024?)

Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@leehinman
Copy link
Contributor

run tests

@P1llus
Copy link
Member Author

P1llus commented Jul 10, 2020

jenkins run tests

This is a list of Defender ATP fields that are mapped to ECS.

[options="header"]
|======================================================================|
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great info to have in the docs.

@P1llus
Copy link
Member Author

P1llus commented Jul 13, 2020

jenkins run tests

@P1llus
Copy link
Member Author

P1llus commented Jul 14, 2020

jenkins run tests

@andrewkroh
Copy link
Member

andrewkroh commented Jul 14, 2020

This has a docs build issue that needs fixed.

03:53:26 INFO:build_docs:asciidoctor: WARNING: include/configuring-intro.asciidoc: line 3: id assigned to block already in use: configuring-microsoft-module
03:53:26 INFO:build_docs:asciidoctor: WARNING: include/config-option-intro.asciidoc: line 3: id assigned to block already in use: microsoft-settings

@adriansr adriansr merged commit fead071 into elastic:master Jul 14, 2020
marc-gr pushed a commit to marc-gr/beats that referenced this pull request Jul 14, 2020
…stic#19197)

What does this PR do?

This PR adds the initial beta support for Microsoft Defender ATP.
Why is it important?

Adds new products to the supported list for filebeat

Co-authored-by: Lee E. Hinman <lee.e.hinman@elastic.co>
(cherry picked from commit fead071)
@marc-gr marc-gr added the v7.9.0 label Jul 14, 2020
andrewkroh pushed a commit that referenced this pull request Jul 14, 2020
…Microsoft Defender ATP (#19907)

* [Filebeat][New Module] Adding support for Microsoft Defender ATP (#19197)

What does this PR do?

This PR adds the initial beta support for Microsoft Defender ATP.
Why is it important?

Adds new products to the supported list for filebeat

Co-authored-by: Lee E. Hinman <lee.e.hinman@elastic.co>
Co-authored-by: Marius Iversen <pillus@chasenet.org>

(cherry picked from commit fead071)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
…stic#19197)

What does this PR do?

This PR adds the initial beta support for Microsoft Defender ATP.
Why is it important?

Adds new products to the supported list for filebeat

Co-authored-by: Lee E. Hinman <lee.e.hinman@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants