-
Notifications
You must be signed in to change notification settings - Fork 8.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution] Siem signals -> alerts as data field and index aliases #106049
Changes from 5 commits
010f093
c4edc7d
f943313
5f11d31
979ad9a
3148569
3e08e73
58f7464
1788952
9377e0c
df9b010
bad2321
b837f27
94f9dee
28722b1
185f4ba
bf89a05
30ac118
3c6f7b7
3c26be4
f1f5ddc
58d0e01
e8f464c
3054481
d7bc0da
0910131
3827188
0cd9b83
5973dde
ded440e
99d83ee
8e7e00e
c033517
fc475fd
5828090
4275da7
0214b61
498f9c4
6bcb6ae
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -96,7 +96,8 @@ export class RuleDataClient implements IRuleDataClient { | |
if (response.body.errors) { | ||
if ( | ||
response.body.items.length > 0 && | ||
response.body.items?.[0]?.index?.error?.type === 'index_not_found_exception' | ||
(response.body.items?.[0]?.index?.error?.type === 'index_not_found_exception' || | ||
response.body.items?.[0]?.index?.error?.type === 'illegal_argument_exception') | ||
) { | ||
return this.createWriteTargetIfNeeded({ namespace }).then(() => { | ||
return clusterClient.bulk(requestWithDefaultParameters); | ||
|
@@ -116,29 +117,22 @@ export class RuleDataClient implements IRuleDataClient { | |
|
||
const clusterClient = await this.getClusterClient(); | ||
|
||
const { body: aliasExists } = await clusterClient.indices.existsAlias({ | ||
name: alias, | ||
}); | ||
|
||
const concreteIndexName = `${alias}-000001`; | ||
|
||
if (!aliasExists) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I removed this check so that we can add the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This check has now been replaced with a more specific check - querying to find concrete indices rather than the alias. |
||
try { | ||
await clusterClient.indices.create({ | ||
index: concreteIndexName, | ||
body: { | ||
aliases: { | ||
[alias]: { | ||
is_write_index: true, | ||
}, | ||
try { | ||
await clusterClient.indices.create({ | ||
index: concreteIndexName, | ||
body: { | ||
aliases: { | ||
[alias]: { | ||
is_write_index: true, | ||
}, | ||
}, | ||
}); | ||
} catch (err) { | ||
// something might have created the index already, that sounds OK | ||
if (err?.meta?.body?.error?.type !== 'resource_already_exists_exception') { | ||
throw err; | ||
} | ||
}, | ||
}); | ||
} catch (err) { | ||
// something might have created the index already, that sounds OK | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What if the index already exists, but isn't set as the write index for the alias? Wouldn't we have to add the alias here? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it possible to get into a scenario where the index exists but the alias has no write target, other than by an admin manually creating the index but creating it incorrectly? My assumption was that this function is the only place where the Perhaps if we get a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I can't think of a way this code would produce inconsistent alias settings, but I've seen many situations in which backup scripts have not correctly restored all aliases. We might not be able to automatically fix all such situations, but we should try to to make the error message as useful as possible IMHO. What if we split setting the write index for an existing index into a separate function and call it or There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For both Going back to the original question
I think not - if the index exists but isn't set as the write index, something has gone wrong but I don't think we have enough information to safely attempt to automatically fix it. If this happens due to a scenario like backup scripts not restoring everything correctly, then (with the latest changes I pushed up) the initial bulk index would fail, we'd call Re: automatically fixing things, I'm imagining a case where a user restores There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. good points 👍 showing a relatively accurate error to the user should be enough for now |
||
if (err?.meta?.body?.error?.type !== 'resource_already_exists_exception') { | ||
throw err; | ||
} | ||
} | ||
} | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
{ | ||
"signal.ancestors.depth": "kibana.alert.ancestors.depth", | ||
"signal.ancestors.id": "kibana.alert.ancestors.id", | ||
"signal.ancestors.index": "kibana.alert.ancestors.index", | ||
"signal.ancestors.type": "kibana.alert.ancestors.type", | ||
"signal.depth": "kibana.alert.depth", | ||
"signal.original_event.action": "kibana.alert.original_event.action", | ||
"signal.original_event.category": "kibana.alert.original_event.category", | ||
"signal.original_event.code": "kibana.alert.original_event.code", | ||
"signal.original_event.created": "kibana.alert.original_event.created", | ||
"signal.original_event.dataset": "kibana.alert.original_event.dataset", | ||
"signal.original_event.duration": "kibana.alert.original_event.duration", | ||
"signal.original_event.end": "kibana.alert.original_event.end", | ||
"signal.original_event.hash": "kibana.alert.original_event.hash", | ||
"signal.original_event.id": "kibana.alert.original_event.id", | ||
"signal.original_event.kind": "kibana.alert.original_event.kind", | ||
"signal.original_event.module": "kibana.alert.original_event.module", | ||
"signal.original_event.outcome": "kibana.alert.original_event.outcome", | ||
"signal.original_event.provider": "kibana.alert.original_event.provider", | ||
"signal.original_event.risk_score": "kibana.alert.original_event.risk_score", | ||
"signal.original_event.risk_score_norm": "kibana.alert.original_event.risk_score_norm", | ||
"signal.original_event.sequence": "kibana.alert.original_event.sequence", | ||
"signal.original_event.severity": "kibana.alert.original_event.severity", | ||
"signal.original_event.start": "kibana.alert.original_event.start", | ||
"signal.original_event.timezone": "kibana.alert.original_event.timezone", | ||
"signal.original_event.type": "kibana.alert.original_event.type", | ||
"signal.original_time": "kibana.alert.original_time", | ||
"signal.rule.author": "kibana.alert.rule.author", | ||
"signal.rule.building_block_type": "kibana.alert.rule.building_block_type", | ||
"signal.rule.created_at": "kibana.alert.rule.created_at", | ||
"signal.rule.created_by": "kibana.alert.rule.created_by", | ||
"signal.rule.description": "kibana.alert.rule.description", | ||
"signal.rule.enabled": "kibana.alert.rule.enabled", | ||
"signal.rule.false_positives": "kibana.alert.rule.false_positives", | ||
"signal.rule.from": "kibana.alert.rule.from", | ||
"signal.rule.id": "kibana.alert.rule.id", | ||
"signal.rule.immutable": "kibana.alert.rule.immutable", | ||
"signal.rule.index": "kibana.alert.rule.index", | ||
"signal.rule.interval": "kibana.alert.rule.interval", | ||
"signal.rule.language": "kibana.alert.rule.language", | ||
"signal.rule.license": "kibana.alert.rule.license", | ||
"signal.rule.max_signals": "kibana.alert.rule.max_signals", | ||
"signal.rule.name": "kibana.alert.rule.name", | ||
"signal.rule.note": "kibana.alert.rule.note", | ||
"signal.rule.query": "kibana.alert.rule.query", | ||
"signal.rule.references": "kibana.alert.rule.references", | ||
"signal.rule.risk_score": "kibana.alert.risk_score", | ||
"signal.rule.risk_score_mapping.field": "kibana.alert.rule.risk_score_mapping.field", | ||
"signal.rule.risk_score_mapping.operator": "kibana.alert.rule.risk_score_mapping.operator", | ||
"signal.rule.risk_score_mapping.value": "kibana.alert.rule.risk_score_mapping.value", | ||
"signal.rule.rule_id": "kibana.alert.rule.rule_id", | ||
"signal.rule.rule_name_override": "kibana.alert.rule.rule_name_override", | ||
"signal.rule.saved_id": "kibana.alert.rule.saved_id", | ||
"signal.rule.severity": "kibana.alert.severity", | ||
"signal.rule.severity_mapping.field": "kibana.alert.rule.severity_mapping.field", | ||
"signal.rule.severity_mapping.operator": "kibana.alert.rule.severity_mapping.operator", | ||
"signal.rule.severity_mapping.value": "kibana.alert.rule.severity_mapping.value", | ||
"signal.rule.severity_mapping.severity": "kibana.alert.rule.severity_mapping.severity", | ||
"signal.rule.tags": "kibana.alert.rule.tags", | ||
"signal.rule.threat.framework": "kibana.alert.rule.threat.framework", | ||
"signal.rule.threat.tactic.id": "kibana.alert.rule.threat.tactic.id", | ||
"signal.rule.threat.tactic.name": "kibana.alert.rule.threat.tactic.name", | ||
"signal.rule.threat.tactic.reference": "kibana.alert.rule.threat.tactic.reference", | ||
"signal.rule.threat.technique.id": "kibana.alert.rule.threat.technique.id", | ||
"signal.rule.threat.technique.name": "kibana.alert.rule.threat.technique.name", | ||
"signal.rule.threat.technique.reference": "kibana.alert.rule.threat.technique.reference", | ||
"signal.rule.threat.technique.subtechnique.id": "kibana.alert.rule.threat.technique.subtechnique.id", | ||
"signal.rule.threat.technique.subtechnique.name": "kibana.alert.rule.threat.technique.subtechnique.name", | ||
"signal.rule.threat.technique.subtechnique.reference": "kibana.alert.rule.threat.technique.subtechnique.reference", | ||
"signal.rule.threat_index": "kibana.alert.rule.threat_index", | ||
"signal.rule.threat_indicator_path": "kibana.alert.rule.threat_indicator_path", | ||
"signal.rule.threat_language": "kibana.alert.rule.threat_language", | ||
"signal.rule.threat_mapping.entries.field": "kibana.alert.rule.threat_mapping.entries.field", | ||
"signal.rule.threat_mapping.entries.value": "kibana.alert.rule.threat_mapping.entries.value", | ||
"signal.rule.threat_mapping.entries.type": "kibana.alert.rule.threat_mapping.entries.type", | ||
"signal.rule.threat_query": "kibana.alert.rule.threat_query", | ||
"signal.rule.threshold.field": "kibana.alert.rule.threshold.field", | ||
"signal.rule.threshold.value": "kibana.alert.rule.threshold.value", | ||
"signal.rule.timeline_id": "kibana.alert.rule.timeline_id", | ||
"signal.rule.timeline_title": "kibana.alert.rule.timeline_title", | ||
"signal.rule.to": "kibana.alert.rule.to", | ||
"signal.rule.type": "kibana.alert.rule.type", | ||
"signal.rule.updated_at": "kibana.alert.rule.updated_at", | ||
"signal.rule.updated_by": "kibana.alert.rule.updated_by", | ||
"signal.rule.version": "kibana.alert.rule.version", | ||
"signal.status": "kibana.alert.workflow_status", | ||
"signal.threshold_result.from": "kibana.alert.threshold_result.from", | ||
"signal.threshold_result.terms.field": "kibana.alert.threshold_result.terms.field", | ||
"signal.threshold_result.terms.value": "kibana.alert.threshold_result.terms.value", | ||
"signal.threshold_result.cardinality.field": "kibana.alert.threshold_result.cardinality.field", | ||
"signal.threshold_result.cardinality.value": "kibana.alert.threshold_result.cardinality.value", | ||
"signal.threshold_result.count": "kibana.alert.threshold_result.count" | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,195 @@ | ||
{ | ||
"signal": { | ||
"type": "object", | ||
"properties": { | ||
"_meta": { | ||
"type": "object", | ||
"properties": { | ||
"version": { | ||
"type": "long" | ||
} | ||
} | ||
}, | ||
"ancestors": { | ||
"properties": { | ||
"rule": { | ||
"type": "keyword" | ||
}, | ||
"index": { | ||
"type": "keyword" | ||
}, | ||
"id": { | ||
"type": "keyword" | ||
}, | ||
"type": { | ||
"type": "keyword" | ||
}, | ||
"depth": { | ||
"type": "long" | ||
} | ||
} | ||
}, | ||
"depth": { | ||
"type": "integer" | ||
}, | ||
"group": { | ||
"type": "object", | ||
"properties": { | ||
"id": { | ||
"type": "keyword" | ||
}, | ||
"index": { | ||
"type": "integer" | ||
} | ||
} | ||
}, | ||
"rule": { | ||
"type": "object", | ||
"properties": { | ||
"author": { | ||
"type": "keyword" | ||
}, | ||
"building_block_type": { | ||
"type": "keyword" | ||
}, | ||
"license": { | ||
"type": "keyword" | ||
}, | ||
"note": { | ||
"type": "text" | ||
}, | ||
"risk_score_mapping": { | ||
"type": "object", | ||
"properties": { | ||
"field": { | ||
"type": "keyword" | ||
}, | ||
"operator": { | ||
"type": "keyword" | ||
}, | ||
"value": { | ||
"type": "keyword" | ||
} | ||
} | ||
}, | ||
"rule_name_override": { | ||
"type": "keyword" | ||
}, | ||
"severity_mapping": { | ||
"type": "object", | ||
"properties": { | ||
"field": { | ||
"type": "keyword" | ||
}, | ||
"operator": { | ||
"type": "keyword" | ||
}, | ||
"value": { | ||
"type": "keyword" | ||
}, | ||
"severity": { | ||
"type": "keyword" | ||
} | ||
} | ||
}, | ||
"threat": { | ||
"type": "object", | ||
"properties": { | ||
"technique": { | ||
"type": "object", | ||
"properties": { | ||
"subtechnique": { | ||
"type": "object", | ||
"properties": { | ||
"id": { | ||
"type": "keyword" | ||
}, | ||
"name": { | ||
"type": "keyword" | ||
}, | ||
"reference": { | ||
"type": "keyword" | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} | ||
}, | ||
"threat_index": { | ||
"type": "keyword" | ||
}, | ||
"threat_indicator_path": { | ||
"type": "keyword" | ||
}, | ||
"threat_language": { | ||
"type": "keyword" | ||
}, | ||
"threat_mapping": { | ||
"type": "object", | ||
"properties": { | ||
"entries": { | ||
"type": "object", | ||
"properties": { | ||
"field": { | ||
"type": "keyword" | ||
}, | ||
"value": { | ||
"type": "keyword" | ||
}, | ||
"type": { | ||
"type": "keyword" | ||
} | ||
} | ||
} | ||
} | ||
}, | ||
"threat_query": { | ||
"type": "keyword" | ||
}, | ||
"threshold": { | ||
"type": "object", | ||
"properties": { | ||
"field": { | ||
"type": "keyword" | ||
}, | ||
"value": { | ||
"type": "float" | ||
} | ||
} | ||
} | ||
} | ||
}, | ||
"threshold_result": { | ||
"properties": { | ||
"from": { | ||
"type": "date" | ||
}, | ||
"terms": { | ||
"properties": { | ||
"field": { | ||
"type": "keyword" | ||
}, | ||
"value": { | ||
"type": "keyword" | ||
} | ||
} | ||
}, | ||
"cardinality": { | ||
"properties": { | ||
"field": { | ||
"type": "keyword" | ||
}, | ||
"value": { | ||
"type": "long" | ||
} | ||
} | ||
}, | ||
"count": { | ||
"type": "long" | ||
} | ||
} | ||
} | ||
} | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
illegal_argument_exception
is returned if the alias exists but has no write_index.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately
illegal_argument_exception
can be returned for many reasons other than the lack of a write index. I can imagine two ways to reduce the risk of misinterpretation:is_write_alias
.Do you think it's safe enough to just assume the reason?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a good point, I'll add logic similar to the old "alias exists" check in
createWriteTargetIfNeeded
but instead checking to ensure the alias has no write target before attempting to create a write target for it. You're right that otherwise we could end up withillegal_argument_exceptions
that cause us to try creating a new write target and that would cause problems if the alias already has a write target.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I replaced the
aliasExists
check with a more general "index exists" check on the AAD indices. This makes thecreateWriteTargetIfNeeded
function safer to call optimistically when we encounterindex_not_found
orillegal_argument
exceptions while still allowing the security solution to add AAD aliases to our.siem-signals
indices. Now, if we do get anillegal_argument_exception
for a reason other than "no concrete AAD indices exist", thencreateWriteTargetIfNeeded
will return without making new indices and potentially making things worse.