Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Request][8.14] improved ES|QL investigation (highlighted) fields #5182

Merged
merged 30 commits into from
May 20, 2024
Merged
Changes from 25 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
eb9de95
First draft
nastasha-solomon May 8, 2024
739359b
Additional information
nastasha-solomon May 9, 2024
b0da54d
Small edits
nastasha-solomon May 9, 2024
a4eded2
Merge branch 'main' into issue-5054-improved-esql-investigation
nastasha-solomon May 9, 2024
1ee4a17
Merge branch 'main' into issue-5054-improved-esql-investigation
nastasha-solomon May 9, 2024
ab386a5
Part of Vitalii's feedback
nastasha-solomon May 11, 2024
c0e0a4a
Merge branch 'main' into issue-5054-improved-esql-investigation
nastasha-solomon May 11, 2024
4743784
Re-orged info
nastasha-solomon May 12, 2024
69e7f00
Re-adding into sen
nastasha-solomon May 12, 2024
75b4818
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 12, 2024
26de3a4
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 13, 2024
c723497
Merge branch 'main' into issue-5054-improved-esql-investigation
nastasha-solomon May 14, 2024
c64371a
Draft of Vitalii's input
nastasha-solomon May 14, 2024
df9eabe
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 14, 2024
5846790
Expanding definition for new fields
nastasha-solomon May 14, 2024
d325351
Merge branch 'issue-5054-improved-esql-investigation' of github.com:e…
nastasha-solomon May 14, 2024
664f620
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 14, 2024
0fadccc
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 14, 2024
08631d6
Merge branch 'main' into issue-5054-improved-esql-investigation
nastasha-solomon May 14, 2024
23d529a
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 14, 2024
47fb98e
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 14, 2024
b52728c
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 14, 2024
fbdc6dc
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 14, 2024
2d1339e
Merge branch 'main' into issue-5054-improved-esql-investigation
nastasha-solomon May 16, 2024
218e0d3
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 17, 2024
d24c3d2
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 17, 2024
814166b
Update docs/detections/rules-ui-create.asciidoc
nastasha-solomon May 17, 2024
6e76bfe
Merge branch 'main' into issue-5054-improved-esql-investigation
nastasha-solomon May 17, 2024
68b306e
Merge branch 'main' into issue-5054-improved-esql-investigation
nastasha-solomon May 19, 2024
1fdd04b
Merge branch 'main' into issue-5054-improved-esql-investigation
nastasha-solomon May 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions docs/detections/rules-ui-create.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,9 @@ TIP: Click the help icon (image:images/esql-ref-button.png[Click the ES|QL help
[[esql-agg-query]]
===== Aggregating query

Aggregating queries use {ref}/esql-functions-operators.html#esql-agg-functions[`STATS...BY`] functions to aggregate source event data. Alerts generated by an {esql} rule with an aggregating query only contain the fields returned by the query.
Aggregating queries use {ref}/esql-functions-operators.html#esql-agg-functions[`STATS...BY`] functions to aggregate source event data. Alerts generated by a rule with an aggregating query only contain the fields that the {esql} query returns and any new fields that the query creates.

NOTE: A _new field_ is a field that doesn't exist in the query's source index and is instead created when the rule runs. You can access new fields in the details of any alerts that are generated by the rule. For example, if you use the {ref}/esql-commands.html#esql-stats-by[`STATS...BY`] function to create a column with aggregated values, the column is created when the rule runs, and is added as a new field to any alerts that are generated by the rule.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

Here is an example aggregating query:

Expand All @@ -311,7 +313,9 @@ NOTE: Rules that use aggregating queries might create duplicate alerts. This can
[float]
[[esql-non-agg-query]]
===== Non-aggregating query
Non-aggregating queries don't use `STATS...BY` functions and don't aggregate source event data. Alerts generated by an {esql} rule with a non-aggregating query only contain the fields returned by the query.
Non-aggregating queries don't use `STATS...BY` functions and don't aggregate source event data. Alerts generated by a non-aggregating query contain source event fields that the query returns, new fields the query creates, and all other fields in the source event document.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

NOTE: New fields don't exist in the source index that the query is searching. Instead, they're created when the query runs, and you can access them in the details of any alerts that are generated. For example, if you use the {ref}/esql-commands.html#esql-eval[`EVAL`] command to append new columns with calculated values, the columns are created when the query runs, and are added as new fields to any alerts generated by the rule.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

Here is an example non-aggregating query:
[source,esql]
Expand Down Expand Up @@ -378,12 +382,15 @@ NOTE: The `max_signals` default value is 100. You can modify it using the <<rule

[float]
[[esql-rule-limitations]]
==== {esql} rule limitations
==== {esql} rule limitations

The {esql} rule has the following limitations:
If your {esql} query creates new fields that aren’t part of the ECS schema, they aren't mapped to the alerts index so you can't search for or filter them in the Alerts table. As a workaround, create <<runtime-fields,runtime fields>>.

[float]
[[custom-highlighted-esql-fields]]
==== Highlight fields returned by the {esql} rule query

- If your {esql} query creates new fields that aren’t part of the ECS schema, they won’t be mapped to the alerts index, and you can't search or filter for them from the Alerts table. As a workaround, create <<runtime-fields,runtime fields>>.
- If your {esql} query creates new fields that aren’t in the query’s source index, they can’t be added to the rule’s <<rule-ui-advanced-params,custom highlighted fields>>.
When configuring an {esql} rule's **<<rule-ui-advanced-params,Custom highlighted fields>>**, you can specify any fields that the rule's aggregating or non-aggregating query return. This can help ensure that returned fields are visible in the alert details flyout while you're investigating alerts.

[float]
[[rule-ui-basic-params]]
Expand Down