Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security_solution][Detections] Refactor signal ancestry to allow multiple parents #76531

Merged
merged 12 commits into from
Sep 9, 2020
Merged

[Security_solution][Detections] Refactor signal ancestry to allow multiple parents #76531

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
39a5f49
Refactors signal ancestry to allow multiple parents
marshallmain Sep 2, 2020
de9debb
Fix depth calculation for 7.10+ signals on pre-7.10 signals
marshallmain Sep 2, 2020
a3e1559
Comment build_signal functions
marshallmain Sep 2, 2020
d818f21
Merge branch 'master' into refactor-ancestry
elasticmachine Sep 2, 2020
2c9d038
Rename buildAncestorsSignal to buildAncestors
marshallmain Sep 3, 2020
93094ab
Update detection engine depth test scripts and docs
marshallmain Sep 4, 2020
08a2a4d
Update halting test readme
marshallmain Sep 4, 2020
9143432
Match up rule ids in readme
marshallmain Sep 4, 2020
f3b294e
Continue populating signal.parent along with signal.parents
marshallmain Sep 4, 2020
166f491
Merge branch 'master' into refactor-ancestry
elasticmachine Sep 4, 2020
0a8caa5
pr comments
marshallmain Sep 8, 2020
203c363
Merge branch 'master' into refactor-ancestry
elasticmachine Sep 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions ...k/plugins/security_solution/server/lib/detection_engine/routes/index/signals_mapping.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,33 @@
}
}
},
"parents": {
"properties": {
"rule": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the rule.id, not rule.rule_id right? Curious if there's any upside to using one over the other. Like if users often import/export rules, would we lose tracing by using the id?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, this is rule.id. I think for the parents/ancestry we want to know which specific instance of a rule generated the alert so id is a good choice.

We could consider making signal.parent.rule an object where we could have id, rule_id, and any other rule fields that could be useful in signal.parent.rule.*, but that would be a breaking change with the existing signal.parent and signal.ancestors so I think we'd need a migration.

"type": "keyword"
},
"index": {
"type": "keyword"
},
"id": {
"type": "keyword"
},
"type": {
"type": "keyword"
},
"depth": {
"type": "long"
}
}
},
"ancestors": {
"properties": {
"rule": {
"type": "keyword"
},
"index": {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

index was in parent but not here

"type": "keyword"
},
"id": {
"type": "keyword"
},
Expand Down Expand Up @@ -299,6 +321,9 @@
},
"threshold_count": {
"type": "float"
},
"depth": {
"type": "integer"
}
}
}
Expand Down
10 changes: 6 additions & 4 deletions x-pack/plugins/security_solution/server/lib/detection_engine/signals/__mocks__/es_results.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,21 +149,23 @@ export const sampleDocWithAncestors = (): SignalSearchResponse => {
delete sampleDoc._source.source;
sampleDoc._source.signal = {
parent: {
rule: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
id: 'd5e8eb51-a6a0-456d-8a15-4b79bfec3d71',
type: 'event',
index: 'myFakeSignalIndex',
depth: 1,
depth: 0,
},
ancestors: [
{
rule: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
id: 'd5e8eb51-a6a0-456d-8a15-4b79bfec3d71',
type: 'event',
index: 'myFakeSignalIndex',
depth: 1,
depth: 0,
},
],
rule: {
id: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
},
depth: 1,
};

return {
Expand Down
76 changes: 40 additions & 36 deletions x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_bulk_body.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,20 +46,20 @@ describe('buildBulkBody', () => {
kind: 'signal',
},
signal: {
parent: {
rule: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 1,
},
parents: [
{
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 0,
},
],
ancestors: [
{
rule: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 1,
depth: 0,
},
],
original_time: '2020-04-20T21:27:45+0000',
Expand Down Expand Up @@ -101,6 +101,7 @@ describe('buildBulkBody', () => {
updated_at: fakeSignalSourceHit.signal.rule?.updated_at,
exceptions_list: getListArrayMock(),
},
depth: 1,
},
};
expect(fakeSignalSourceHit).toEqual(expected);
Expand Down Expand Up @@ -148,20 +149,20 @@ describe('buildBulkBody', () => {
kind: 'event',
module: 'system',
},
parent: {
rule: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 1,
},
parents: [
{
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 0,
},
],
ancestors: [
{
rule: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 1,
depth: 0,
},
],
original_time: '2020-04-20T21:27:45+0000',
Expand Down Expand Up @@ -203,6 +204,7 @@ describe('buildBulkBody', () => {
threat: [],
exceptions_list: getListArrayMock(),
},
depth: 1,
},
};
expect(fakeSignalSourceHit).toEqual(expected);
Expand Down Expand Up @@ -248,20 +250,20 @@ describe('buildBulkBody', () => {
dataset: 'socket',
module: 'system',
},
parent: {
rule: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 1,
},
parents: [
{
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 0,
},
],
ancestors: [
{
rule: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 1,
depth: 0,
},
],
original_time: '2020-04-20T21:27:45+0000',
Expand Down Expand Up @@ -303,6 +305,7 @@ describe('buildBulkBody', () => {
throttle: 'no_actions',
exceptions_list: getListArrayMock(),
},
depth: 1,
},
};
expect(fakeSignalSourceHit).toEqual(expected);
Expand Down Expand Up @@ -341,20 +344,20 @@ describe('buildBulkBody', () => {
original_event: {
kind: 'event',
},
parent: {
rule: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 1,
},
parents: [
{
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 0,
},
],
ancestors: [
{
rule: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
id: sampleIdGuid,
type: 'event',
index: 'myFakeSignalIndex',
depth: 1,
depth: 0,
},
],
original_time: '2020-04-20T21:27:45+0000',
Expand Down Expand Up @@ -396,6 +399,7 @@ describe('buildBulkBody', () => {
throttle: 'no_actions',
exceptions_list: getListArrayMock(),
},
depth: 1,
},
};
expect(fakeSignalSourceHit).toEqual(expected);
Expand Down
9 changes: 6 additions & 3 deletions x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_bulk_body.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
* you may not use this file except in compliance with the Elastic License.
*/

import { SignalSourceHit, SignalHit } from './types';
import { SignalSourceHit, SignalHit, Signal } from './types';
import { buildRule } from './build_rule';
import { buildSignal } from './build_signal';
import { additionalSignalFields, buildSignal } from './build_signal';
import { buildEventTypeSignal } from './build_event_type_signal';
import { RuleAlertAction } from '../../../../common/detection_engine/types';
import { RuleTypeParams } from '../types';
Expand Down Expand Up @@ -58,7 +58,10 @@ export const buildBulkBody = ({
tags,
throttle,
});
const signal = buildSignal(doc, rule);
const signal: Signal = {
...buildSignal([doc], rule),
Copy link
Contributor Author

@marshallmain marshallmain Sep 2, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

buildSignal now expects an array of documents and handles building the rule, parents, and ancestors. additionalSignalFields takes a single document and creates the fields for the special case where we only have 1 event triggering the signal (so we can copy the fields from the event into the signal). When we add signals based on EQL sequences we won't be able to copy every event from the sequence into the signal, instead relying on signal.parents to contain references to the sub-events.

...additionalSignalFields(doc),
};
const event = buildEventTypeSignal(doc);
const signalHit: SignalHit = {
...doc._source,
Expand Down
102 changes: 101 additions & 1 deletion x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_rule.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@
* you may not use this file except in compliance with the Elastic License.
*/

import { buildRule } from './build_rule';
import { buildRule, removeInternalTagsFromRule } from './build_rule';
import { sampleDocNoSortId, sampleRuleAlertParams, sampleRuleGuid } from './__mocks__/es_results';
import { RulesSchema } from '../../../../common/detection_engine/schemas/response/rules_schema';
import { getListArrayMock } from '../../../../common/detection_engine/schemas/types/lists.mock';
import { INTERNAL_RULE_ID_KEY, INTERNAL_IMMUTABLE_KEY } from '../../../../common/constants';
import { getPartialRulesSchemaMock } from '../../../../common/detection_engine/schemas/response/rules_schema.mocks';

describe('buildRule', () => {
beforeEach(() => {
Expand Down Expand Up @@ -208,4 +210,102 @@ describe('buildRule', () => {
};
expect(rule).toEqual(expected);
});

test('it builds a rule and removes internal tags', () => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removeInternalTagsFromRule now happens inside buildRule so 1 additional test here.

const ruleParams = sampleRuleAlertParams();
const rule = buildRule({
actions: [],
doc: sampleDocNoSortId(),
ruleParams,
name: 'some-name',
id: sampleRuleGuid,
enabled: false,
createdAt: '2020-01-28T15:58:34.810Z',
updatedAt: '2020-01-28T15:59:14.004Z',
createdBy: 'elastic',
updatedBy: 'elastic',
interval: 'some interval',
tags: [
'some fake tag 1',
'some fake tag 2',
`${INTERNAL_RULE_ID_KEY}:rule-1`,
`${INTERNAL_IMMUTABLE_KEY}:true`,
],
throttle: 'no_actions',
});
const expected: Partial<RulesSchema> = {
actions: [],
author: ['Elastic'],
building_block_type: 'default',
created_by: 'elastic',
description: 'Detecting root and admin users',
enabled: false,
false_positives: [],
from: 'now-6m',
id: '04128c15-0d1b-4716-a4c5-46997ac7f3bd',
immutable: false,
index: ['auditbeat-*', 'filebeat-*', 'packetbeat-*', 'winlogbeat-*'],
interval: 'some interval',
language: 'kuery',
license: 'Elastic License',
max_signals: 10000,
name: 'some-name',
output_index: '.siem-signals',
query: 'user.name: root or user.name: admin',
references: ['http://google.com'],
risk_score: 50,
risk_score_mapping: [],
rule_id: 'rule-1',
severity: 'high',
severity_mapping: [],
tags: ['some fake tag 1', 'some fake tag 2'],
threat: [],
to: 'now',
type: 'query',
note: '',
updated_by: 'elastic',
updated_at: rule.updated_at,
created_at: rule.created_at,
throttle: 'no_actions',
exceptions_list: getListArrayMock(),
version: 1,
};
expect(rule).toEqual(expected);
});

Copy link
Contributor Author

@marshallmain marshallmain Sep 2, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Next 4 tests are simply moved from build_signal.test.ts since removeInternalTagsFromRule was moved from build_signal.ts to build_rule.ts

test('it removes internal tags from a typical rule', () => {
const rule = getPartialRulesSchemaMock();
rule.tags = [
'some fake tag 1',
'some fake tag 2',
`${INTERNAL_RULE_ID_KEY}:rule-1`,
`${INTERNAL_IMMUTABLE_KEY}:true`,
];
const noInternals = removeInternalTagsFromRule(rule);
expect(noInternals).toEqual(getPartialRulesSchemaMock());
});

test('it works with an empty array', () => {
const rule = getPartialRulesSchemaMock();
rule.tags = [];
const noInternals = removeInternalTagsFromRule(rule);
const expected = getPartialRulesSchemaMock();
expected.tags = [];
expect(noInternals).toEqual(expected);
});

test('it works if tags does not exist', () => {
const rule = getPartialRulesSchemaMock();
delete rule.tags;
const noInternals = removeInternalTagsFromRule(rule);
const expected = getPartialRulesSchemaMock();
delete expected.tags;
expect(noInternals).toEqual(expected);
});

test('it works if tags contains normal values and no internal values', () => {
const rule = getPartialRulesSchemaMock();
const noInternals = removeInternalTagsFromRule(rule);
expect(noInternals).toEqual(rule);
});
});
16 changes: 15 additions & 1 deletion x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_rule.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import { buildRiskScoreFromMapping } from './mappings/build_risk_score_from_mapp
import { SignalSourceHit } from './types';
import { buildSeverityFromMapping } from './mappings/build_severity_from_mapping';
import { buildRuleNameFromMapping } from './mappings/build_rule_name_from_mapping';
import { INTERNAL_IDENTIFIER } from '../../../../common/constants';

interface BuildRuleParams {
ruleParams: RuleTypeParams;
Expand Down Expand Up @@ -64,7 +65,7 @@ export const buildRule = ({

const meta = { ...ruleParams.meta, ...riskScoreMeta, ...severityMeta, ...ruleNameMeta };

return pickBy<RulesSchema>((value: unknown) => value != null, {
const rule = pickBy<RulesSchema>((value: unknown) => value != null, {
id,
rule_id: ruleParams.ruleId ?? '(unknown rule_id)',
actions,
Expand Down Expand Up @@ -111,4 +112,17 @@ export const buildRule = ({
anomaly_threshold: ruleParams.anomalyThreshold,
threshold: ruleParams.threshold,
});
return removeInternalTagsFromRule(rule);
};

export const removeInternalTagsFromRule = (rule: Partial<RulesSchema>): Partial<RulesSchema> => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cut/paste from build_signal.ts since this is rule-related code.

if (rule.tags == null) {
return rule;
} else {
const ruleWithoutInternalTags: Partial<RulesSchema> = {
...rule,
tags: rule.tags.filter((tag) => !tag.startsWith(INTERNAL_IDENTIFIER)),
};
return ruleWithoutInternalTags;
}
};
Loading