[ML] Adds Metadata and Discovery Analysis Jobs to Security Integration #76023

blaklaybul · 2020-08-26T19:30:01Z

Summary

This PR contains a suite of new ML job for the Security solution's anomaly detection integration that offers new analyses to enable threat detection on metadata services, system and network discovery processes, compiler events, and processes invoked by sudo.

In total, there are 12 new jobs:

siem_auditbeat/ml/linux_rare_kernel_module_arguments
siem_auditbeat/ml/linux_rare_metadata_process
siem_auditbeat/ml/linux_rare_metadata_user
siem_auditbeat/ml/linux_rare_user_compiler
siem_auditbeat/ml/linux_rare_sudo_user
siem_auditbeat/ml/linux_network_connection_discovery
siem_auditbeat/ml/linux_network_configuration_discovery
siem_auditbeat/ml/linux_system_information_discovery
siem_auditbeat/ml/linux_system_process_discovery
siem_auditbeat/ml/linux_system_user_discovery
siem_winlogbeat/ml/windows_rare_metadata_process
siem_winlogbeat/ml/windows_rare_metadata_user

modules: siem_auditbeat, siem_winlogbeat

elasticmachine · 2020-08-26T19:30:04Z

Pinging @elastic/ml-ui (:ml)

pheyos · 2020-08-27T11:05:44Z

@blaklaybul in order to make the api integration tests pass, you have to add the new modules to the list of expected modules.

blaklaybul · 2020-08-27T12:41:52Z

@pheyos there are no new modules in this PR, it only updates to existing modules. Not sure why I'm getting this error.

blaklaybul · 2020-08-27T12:42:06Z

@elasticmachine merge upstream

x-pack/plugins/ml/server/models/data_recognizer/modules/siem_winlogbeat/manifest.json

…nto ml-security-metadata-analysis

peteharverson · 2020-09-01T11:16:35Z

x-pack/plugins/ml/server/models/data_recognizer/modules/siem_auditbeat/manifest.json

@@ -40,6 +40,46 @@
    {
      "id": "linux_anomalous_user_name_ecs",
      "file": "linux_anomalous_user_name_ecs.json"
+    },


Is the description for this module still OK, or does it need amending to reflect the new mix of jobs in this module?

@randomuserid how would you like this to be updated?

I don't see why that description cannot apply to the new Linux jobs

great - leaving it as is

x-pack/plugins/ml/server/models/data_recognizer/modules/siem_auditbeat/manifest.json

peteharverson · 2020-09-01T11:20:56Z

x-pack/plugins/ml/server/models/data_recognizer/modules/siem_winlogbeat/manifest.json

@@ -48,6 +48,14 @@
    {
      "id": "windows_rare_user_runas_event",
      "file": "windows_rare_user_runas_event.json"
+    },


As above, is the description for the winlogbeat module still ok?

we could remove the words "process and network" because we have branched out to additional event types

@randomuserid do you want to update this? This is what a user sees when interacting with the module via the ML ui.

Why don't we leave it alone for now. When we have a decision on multi-index Windows events we can rewrite them then.

.../models/data_recognizer/modules/siem_auditbeat/ml/linux_network_configuration_discovery.json

peteharverson

Added extra comments around the custom URLs.

.../server/models/data_recognizer/modules/siem_winlogbeat/ml/windows_rare_metadata_process.json

.../ml/server/models/data_recognizer/modules/siem_winlogbeat/ml/windows_rare_metadata_user.json

...ver/models/data_recognizer/modules/siem_auditbeat/ml/linux_network_connection_discovery.json

...ver/models/data_recognizer/modules/siem_auditbeat/ml/linux_rare_kernel_module_arguments.json

.../ml/server/models/data_recognizer/modules/siem_auditbeat/ml/linux_rare_metadata_process.json

...plugins/ml/server/models/data_recognizer/modules/siem_auditbeat/ml/linux_rare_sudo_user.json

...ins/ml/server/models/data_recognizer/modules/siem_auditbeat/ml/linux_rare_user_compiler.json

...ver/models/data_recognizer/modules/siem_auditbeat/ml/linux_system_information_discovery.json

.../server/models/data_recognizer/modules/siem_auditbeat/ml/linux_system_process_discovery.json

.../ml/server/models/data_recognizer/modules/siem_auditbeat/ml/linux_system_user_discovery.json

.../server/models/data_recognizer/modules/siem_winlogbeat/ml/windows_rare_metadata_process.json

blaklaybul · 2020-09-09T19:36:52Z

@elasticmachine merge upstream

peteharverson

Tested latest changes and all looks good. Just added one comment about the influencers for one of the jobs.

peteharverson · 2020-09-10T11:54:12Z

...ver/models/data_recognizer/modules/siem_auditbeat/ml/linux_rare_kernel_module_arguments.json

+      ],
+      "influencers": [
+        "process.name",
+        "process.working_directory",


If we want to limit the job to 4 influencers for this linux_rare_kernel_module_arguments job, is it better to use process.title in place of process.working_directory? The Anomaly Explorer won't list process.title as a swim lane 'view by' option or in the left hand Top Influencers list otherwise.

It's my understanding that for this analysis, it's valuable to know the location (i.e. process.working_directory) from which the process is being executed. @randomuserid, if we had to choose between swapping either process.name or process.working_directory for process.title as an influencer, which do you think should go? Here's some example data:

"process" : { "title" : "/bin/sh -e /usr/share/initramfs-tools/hooks/kmod prereqs", "name" : "kmod", "working_directory" : "/" }

Note that if we choose to swap out process.name then the two custom URLs that use process.name should be deleted (unless they also work with process.title?).

Process.name is probably the best one to cut because it will tend to show up in the process.title field & the data feed is limited to a few kernel module related processes by name so process name will always be one of those.

spoke with @randomuserid. process.name is limited to just 4 processes because of the datafeed query, so let's swap it out for process.title in the influencer list. As @peteharverson mentioned, this means we'll need to remove the custom urls that utilize process.name. I'll make the change.

…/kibana into ml-security-metadata-analysis

peteharverson

Latest changes LGTM!

kibanamachine · 2020-09-10T17:09:00Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 734da20

Build metrics

distributable file count

id	value	diff	baseline
default	45501	+24	45477

History

💚 Build #73514 succeeded 1c2f034
💚 Build #73418 succeeded c5c7cb4
💚 Build #71900 succeeded 2963e7a
💚 Build #70904 succeeded 1043b4f
💔 Build #70827 failed 2ac8f58

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2020-09-14T17:18:44Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 76023 or prevent reminders by adding the backport:skip label.

elastic#76023) * adds enhanced winlogbeat module * adds enhanced auditbeat module * splits discovery jobs * fixes winlogbeat manifest * adds process group * adds custom urls * adds by field as influencer * use process.title as influencer * updates custom url Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

kibanamachine · 2020-09-15T17:28:44Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

#76023) (#77383) * adds enhanced winlogbeat module * adds enhanced auditbeat module * splits discovery jobs * fixes winlogbeat manifest * adds process group * adds custom urls * adds by field as influencer * use process.title as influencer * updates custom url Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

blaklaybul added 3 commits August 26, 2020 15:08

adds enhanced winlogbeat module

a295517

adds enhanced auditbeat module

0f0ca94

splits discovery jobs

39c2bf3

blaklaybul added release_note:enhancement :ml v7.10.0 labels Aug 26, 2020

blaklaybul requested a review from peteharverson August 26, 2020 19:30

blaklaybul requested a review from a team as a code owner August 26, 2020 19:30

blaklaybul self-assigned this Aug 26, 2020

peteharverson added the Feature:Anomaly Detection ML anomaly detection label Aug 27, 2020

Merge branch 'master' into ml-security-metadata-analysis

2ac8f58

peteharverson reviewed Aug 27, 2020

View reviewed changes

x-pack/plugins/ml/server/models/data_recognizer/modules/siem_winlogbeat/manifest.json Outdated Show resolved Hide resolved

blaklaybul added 2 commits August 27, 2020 11:28

fixes winlogbeat manifest

171eb59

Merge remote-tracking branch 'origin/ml-security-metadata-analysis' i…

1043b4f

…nto ml-security-metadata-analysis

peteharverson reviewed Sep 1, 2020

View reviewed changes

adds process group

2963e7a

peteharverson reviewed Sep 4, 2020

View reviewed changes

.../models/data_recognizer/modules/siem_auditbeat/ml/linux_network_configuration_discovery.json Outdated Show resolved Hide resolved

peteharverson reviewed Sep 8, 2020

View reviewed changes

adds custom urls

c5c7cb4

peteharverson reviewed Sep 9, 2020

View reviewed changes

.../server/models/data_recognizer/modules/siem_winlogbeat/ml/windows_rare_metadata_process.json Show resolved Hide resolved

adds by field as influencer

b9cb89f

Merge branch 'master' into ml-security-metadata-analysis

1c2f034

peteharverson reviewed Sep 10, 2020

View reviewed changes

blaklaybul added 2 commits September 10, 2020 11:17

use process.title as influencer

d5ccd80

Merge branch 'ml-security-metadata-analysis' of github.com:blaklaybul…

1aed23f

…/kibana into ml-security-metadata-analysis

updates custom url

734da20

peteharverson approved these changes Sep 10, 2020

View reviewed changes

blaklaybul merged commit a55edc9 into elastic:master Sep 10, 2020

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Sep 14, 2020

blaklaybul mentioned this pull request Sep 14, 2020

[7.x] [ML] Adds Metadata and Discovery Analysis Jobs to Security Integration (#76023) #77383

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Sep 15, 2020

This was referenced Sep 16, 2020

[ML] Fixes typo in data recognizer text #77691

Merged

[DOCS] Add metadata and discovery analysis jobs for security integration elastic/stack-docs#1363

Merged

lcawl mentioned this pull request Oct 15, 2020

[DOCS] Add machine learning jobs to Whats New elastic/security-docs#319

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Adds Metadata and Discovery Analysis Jobs to Security Integration #76023

[ML] Adds Metadata and Discovery Analysis Jobs to Security Integration #76023

blaklaybul commented Aug 26, 2020

elasticmachine commented Aug 26, 2020

pheyos commented Aug 27, 2020

blaklaybul commented Aug 27, 2020

blaklaybul commented Aug 27, 2020

peteharverson Sep 1, 2020

blaklaybul Sep 1, 2020

randomuserid Sep 1, 2020

blaklaybul Sep 2, 2020

peteharverson Sep 1, 2020

randomuserid Sep 1, 2020

blaklaybul Sep 1, 2020

randomuserid Sep 1, 2020

peteharverson left a comment

blaklaybul commented Sep 9, 2020

peteharverson left a comment

peteharverson Sep 10, 2020

blaklaybul Sep 10, 2020 •

edited

Loading

peteharverson Sep 10, 2020

randomuserid Sep 10, 2020

blaklaybul Sep 10, 2020

peteharverson left a comment

kibanamachine commented Sep 10, 2020

kibanamachine commented Sep 14, 2020

kibanamachine commented Sep 15, 2020

[ML] Adds Metadata and Discovery Analysis Jobs to Security Integration #76023

[ML] Adds Metadata and Discovery Analysis Jobs to Security Integration #76023

Conversation

blaklaybul commented Aug 26, 2020

Summary

elasticmachine commented Aug 26, 2020

pheyos commented Aug 27, 2020

blaklaybul commented Aug 27, 2020

blaklaybul commented Aug 27, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

blaklaybul commented Sep 9, 2020

peteharverson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blaklaybul Sep 10, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

kibanamachine commented Sep 10, 2020

💚 Build Succeeded

Build metrics

distributable file count

History

kibanamachine commented Sep 14, 2020

kibanamachine commented Sep 15, 2020

blaklaybul Sep 10, 2020 •

edited

Loading