Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RAM] Add query conditional action filter to Security Solution UI #154680

Merged
merged 22 commits into from
Apr 21, 2023

Conversation

Zacqary
Copy link
Contributor

@Zacqary Zacqary commented Apr 10, 2023

Summary

Closes #152611
Fixes #155146 too

Screenshot 2023-04-10 at 2 09 29 PM

Checklist

Delete any items that are not applicable to this PR.

@Zacqary Zacqary added Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) release_note:feature Makes this part of the condensed release notes Feature:Alerting/RulesManagement Issues related to the Rules Management UX v8.8.0 labels Apr 10, 2023
@apmmachine
Copy link
Contributor

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@Zacqary Zacqary marked this pull request as ready for review April 13, 2023 19:42
@Zacqary Zacqary requested review from a team as code owners April 13, 2023 19:42
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@Zacqary Zacqary enabled auto-merge (squash) April 17, 2023 15:40
Copy link
Contributor

@CoenWarmer CoenWarmer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AO Changes LGTM

Copy link
Contributor

@JiaweiWu JiaweiWu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good, I saw a few inconsistencies between this query filter and the 1 in discovery:

When trying to use DSLs:
Discovery:
image

New alert filter query:
image

The new alert filter query also seems to lack spacing, and the left button seems to be smaller, just small UI things.

I will test the filter with Xavier after he shows me how to.

Copy link
Contributor

@JiaweiWu JiaweiWu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, aside from the comment I left earlier 🙂

@Zacqary
Copy link
Contributor Author

Zacqary commented Apr 18, 2023

The Discover plugin seems to be processing the query DSL and adding it to the filter's meta.value prop. Unclear how or where it's doing this.

It's also unclear where the Discover plugin is actually implementing the <SearchBar> component so I'm not able to find what this plugin is doing differently.

@elastic/kibana-data-discovery please advise? What do we need to be passing into the <SearchBar> to get DSL filter badges to display correctly?

@yctercero
Copy link
Contributor

yctercero commented Apr 18, 2023

Hey! This is a really cool feature! Tested it out and am seeing some strange behaviors. Let me know if you need more details from me but I noticed the following:

  • First time going through I created a rule with an action, the query text did not end up saving (didn't get on video)
  • When I edit the query, it sometimes saves, sometimes doesn't (see in video, the edit did not take)
  • When I select to edit my filter, I don't see my values, but if I view it as DSL, I see them (see in video)
  • For one of my filter values it just says "Warning", not sure what to do with that (see in video)
154680.mov

I did see some React errors like Can't perform a React state update on an unmounted component. for QueryStringInputUI and QueryBarMenuComponent not sure if that's related to what I'm seeing in the video.

@XavierM
Copy link
Contributor

XavierM commented Apr 19, 2023

@yctercero, I think everything will start to work when @e40pud merged his PR.

@ersin-erdal
Copy link
Contributor

ersin-erdal commented Apr 21, 2023

Hey! This is a really cool feature! Tested it out and am seeing some strange behaviors. Let me know if you need more details from me but I noticed the following:

  • First time going through I created a rule with an action, the query text did not end up saving (didn't get on video)
  • When I edit the query, it sometimes saves, sometimes doesn't (see in video, the edit did not take)
  • When I select to edit my filter, I don't see my values, but if I view it as DSL, I see them (see in video)
  • For one of my filter values it just says "Warning", not sure what to do with that (see in video)

154680.mov
I did see some React errors like Can't perform a React state update on an unmounted component. for QueryStringInputUI and QueryBarMenuComponent not sure if that's related to what I'm seeing in the video.

@yctercero
Fixed

@ersin-erdal
Copy link
Contributor

Testing locally I'm not seeing the alert_query functionality working at the moment. I have the main rule query as host.name:*, and the alert_query as a filter w/ host.name: exists (tried host.name:* in the query as well), however I'm not seeing my action fired. Note: alerts are being generated, and they do have host.name set. Once I disable the alert_query I do see the action firing again.

edit: Also not seeing the within the selected time frame option working either. I have it configured for all day Thursday (today) local time, but am only seeing alerts created, not my slack action firing:

Hi @spong
Must be fixed now, but i've just realised that Security Rules always trigger just one action no matter how many alerts there are.
Therefore, in order to skip an action, your query must filter out all of the alerts.

@spong
Copy link
Member

spong commented Apr 21, 2023

Hi @spong
Must be fixed now, but i've just realised that Security Rules always trigger just one action no matter how many alerts there are.
Therefore, in order to skip an action, your query must filter out all of the alerts.

Thanks @ersin-erdal! I'll test again later today. That said, since the rule query was the same as the alert query (host.name:*) all generated alerts should be matched.

If this is still an issue when I test later I'll provide detailed reproduction steps 👍

@ersin-erdal
Copy link
Contributor

ersin-erdal commented Apr 21, 2023

If saving after first enabling within selected time frame I'm seeing an error that says only 00:00-24:00 is supported, however re-selecting 00:00 as the end time seems to get it to pass validation.

Note: a little interesting here that it says 24:00 is a valid value but it isn't listed in the combobox, and when manually entered it will auto-convert back to 00:00.

It looks like EuiDatePicker doesn't allow us to use 24:00, replaces it with 00:00.
Even though the default time range is 00:00 -> 24:00 , the component shows it as 00:00 -> 00:00 but posts as 00:00 -> 24:00.

I changed default end time to 23:59 to avoid confusion.
And updated code to cover the full 24h when end and start times are the same eg. 00:00 -> 00:00

Copy link
Contributor

@stratoula stratoula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unified search changes LGTM, code review only

Copy link
Member

@markov00 markov00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, code review only. The added optional prop doesn't change the current behaviour of the unified search so it is safe on our end.

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
triggersActionsUi 554 556 +2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 9.1MB 9.1MB +339.0B
triggersActionsUi 1.4MB 1.4MB -1.3KB
unifiedSearch 268.8KB 268.8KB +65.0B
total -879.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
triggersActionsUi 84.3KB 84.4KB +74.0B
Unknown metric groups

ESLint disabled line counts

id before after diff
securitySolution 394 397 +3

Total ESLint disabled count

id before after diff
securitySolution 474 477 +3

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ersin-erdal

Copy link
Member

@spong spong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked out, tested locally, and code reviewed relevant detection-rules area files -- LGTM! 👍 🎉

All previous issues I saw appear to be resolved @ersin-erdal -- I was able to receive notifications with each new setting, both configured together and independently. 🚀

Only remaining nits I see are some styling issues with the KQL bar -- filter options button seems squished compared to the add filter button (see over diff in image), and as I think someone else mentioned the filter pills need some margin and are being stacked vertically instead of horizontally added:

Main KQL bar from Definition step:

Lastly, I have one question about the 23:59 fix w/ the ending time window:

Does this leave a 1min gap (23:59:00 -> 00:00:00) that an action wouldn't fire, or do you round up somewhere downstream to cover the full 24hrs?

@Zacqary Zacqary merged commit 350bd3e into elastic:main Apr 21, 2023
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Apr 21, 2023
@spong spong added needs_docs Feature:Alerting/RuleActions Issues related to the Actions attached to Rules on the Alerting Framework Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Apr 24, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@ersin-erdal
Copy link
Contributor

ersin-erdal commented Apr 24, 2023

Checked out, tested locally, and code reviewed relevant detection-rules area files -- LGTM! 👍 🎉

All previous issues I saw appear to be resolved @ersin-erdal -- I was able to receive notifications with each new setting, both configured together and independently. 🚀

Only remaining nits I see are some styling issues with the KQL bar -- filter options button seems squished compared to the add filter button (see over diff in image), and as I think someone else mentioned the filter pills need some margin and are being stacked vertically instead of horizontally added:

Main KQL bar from Definition step:

Lastly, I have one question about the 23:59 fix w/ the ending time window:

Does this leave a 1min gap (23:59:00 -> 00:00:00) that an action wouldn't fire, or do you round up somewhere downstream to cover the full 24hrs?

@spong

There will be another PR for Conditonal Actions for For each alert type, i can fix that UI issue (margin around the filter items) in that PR.

For 00:00 -> 24:00 issue i changed it to 23:59 to avoid 00:00 -> 00:00 (which means 00:00 -> 24:00) confusion.
If a user wants full day (24h) coverage they need to use a time range that has the same start and end times such as 00:00 -> 00:00. or something like 15:00 -> 15:00. If the end time is equal or lower than the start time, the framework considers the end time in the next day. But if you think that using 00:00 -> 00:00 is better we can simply switch to that.

@spong
Copy link
Member

spong commented Apr 24, 2023

There will be another PR for Conditonal Actions for For each alert type, i can fix that UI issue (margin around the filter items) in that PR.

Sounds good 👍

For 00:00 -> 24:00 issue i changed it to 23:59 to avoid 00:00 -> 00:00 (which means 00:00 -> 24:00) confusion.
If a user wants full day (24h) coverage they need to use a time range that has the same start and end times such as 00:00 -> 00:00. or something like 15:00 -> 15:00. If the end time is equal or lower than the start time, the framework considers the end time in the next day. But if you think that using 00:00 -> 00:00 is better we can simply switch to that.

Thank you for clarifying the behavior here. 🙂 From a UX perspective, I think the UI should default the time-range to the full day, and users can modify from there. So if it's 00:00 -> 00:00 or 00:00 -> 24:00 I don't think it matters -- both read fine to me, but sounds like the former was causing that validation issue.

nikitaindik pushed a commit to nikitaindik/kibana that referenced this pull request Apr 25, 2023
…astic#154680)

## Summary

Closes elastic#152611 

<img width="788" alt="Screenshot 2023-04-10 at 2 09 29 PM"
src="https://user-images.githubusercontent.com/1445834/230977273-9d00aa3d-af2c-4dff-80ef-3034213df90d.png">


### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
@CoenWarmer
Copy link
Contributor

This caused #155655, but we have a fix with #155796

ersin-erdal added a commit that referenced this pull request May 9, 2023
Fixes: #156878

As discussed in the issue just turning `if alert is generated within
timeframe` on should not change the filter scope.
And the UI should not confuse the users by setting some days as default.

And as discussed in #154680, default hours filter should cover the whole
day.

Therefore, this PR sets default alerts filter options as:
```
  {
    days: [],
    hours: {
      start: '00:00',
      end: '00:00',
    },
  };
```

empty days array maps to all weekdays  `[1,2,3,4,5,6,7]`
and hours `00:00 -> 00:00` maps to `00:00 -> 24:00`
ersin-erdal added a commit to ersin-erdal/kibana that referenced this pull request May 9, 2023
…#156913)

Fixes: elastic#156878

As discussed in the issue just turning `if alert is generated within
timeframe` on should not change the filter scope.
And the UI should not confuse the users by setting some days as default.

And as discussed in elastic#154680, default hours filter should cover the whole
day.

Therefore, this PR sets default alerts filter options as:
```
  {
    days: [],
    hours: {
      start: '00:00',
      end: '00:00',
    },
  };
```

empty days array maps to all weekdays  `[1,2,3,4,5,6,7]`
and hours `00:00 -> 00:00` maps to `00:00 -> 24:00`

(cherry picked from commit d157389)

# Conflicts:
#	x-pack/plugins/rule_registry/server/utils/create_get_summarized_alerts_fn.ts
ersin-erdal added a commit that referenced this pull request May 15, 2023
…156913) (#157211)

# Backport

This will backport the following commits from `main` to `8.8`:
- [Not selecting any days in alerts filter maps to all weekdays
(#156913)](#156913)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ersin
Erdal","email":"92688503+ersin-erdal@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-05-09T15:29:14Z","message":"Not
selecting any days in alerts filter maps to all weekdays
(#156913)\n\nFixes: #156878\r\n\r\nAs discussed in the issue just
turning `if alert is generated within\r\ntimeframe` on should not change
the filter scope.\r\nAnd the UI should not confuse the users by setting
some days as default.\r\n\r\nAnd as discussed in #154680, default hours
filter should cover the whole\r\nday.\r\n\r\nTherefore, this PR sets
default alerts filter options as:\r\n```\r\n {\r\n days: [],\r\n hours:
{\r\n start: '00:00',\r\n end: '00:00',\r\n },\r\n
};\r\n```\r\n\r\nempty days array maps to all weekdays
`[1,2,3,4,5,6,7]`\r\nand hours `00:00 -> 00:00` maps to `00:00 ->
24:00`","sha":"d15738989a6544c9b939cde5afc3533c9850d7d4","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:ResponseOps","v8.8.0","v8.9.0"],"number":156913,"url":"#156913
selecting any days in alerts filter maps to all weekdays
(#156913)\n\nFixes: #156878\r\n\r\nAs discussed in the issue just
turning `if alert is generated within\r\ntimeframe` on should not change
the filter scope.\r\nAnd the UI should not confuse the users by setting
some days as default.\r\n\r\nAnd as discussed in #154680, default hours
filter should cover the whole\r\nday.\r\n\r\nTherefore, this PR sets
default alerts filter options as:\r\n```\r\n {\r\n days: [],\r\n hours:
{\r\n start: '00:00',\r\n end: '00:00',\r\n },\r\n
};\r\n```\r\n\r\nempty days array maps to all weekdays
`[1,2,3,4,5,6,7]`\r\nand hours `00:00 -> 00:00` maps to `00:00 ->
24:00`","sha":"d15738989a6544c9b939cde5afc3533c9850d7d4"}},"sourceBranch":"main","suggestedTargetBranches":["8.8"],"targetPullRequestStates":[{"branch":"8.8","label":"v8.8.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"#156913
selecting any days in alerts filter maps to all weekdays
(#156913)\n\nFixes: #156878\r\n\r\nAs discussed in the issue just
turning `if alert is generated within\r\ntimeframe` on should not change
the filter scope.\r\nAnd the UI should not confuse the users by setting
some days as default.\r\n\r\nAnd as discussed in #154680, default hours
filter should cover the whole\r\nday.\r\n\r\nTherefore, this PR sets
default alerts filter options as:\r\n```\r\n {\r\n days: [],\r\n hours:
{\r\n start: '00:00',\r\n end: '00:00',\r\n },\r\n
};\r\n```\r\n\r\nempty days array maps to all weekdays
`[1,2,3,4,5,6,7]`\r\nand hours `00:00 -> 00:00` maps to `00:00 ->
24:00`","sha":"d15738989a6544c9b939cde5afc3533c9850d7d4"}}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting Feature:Alerting/RuleActions Issues related to the Actions attached to Rules on the Alerting Framework Feature:Alerting/RulesManagement Issues related to the Rules Management UX needs_docs release_note:feature Makes this part of the condensed release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.8.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Security Solution] Detection rules with actions cannot be created [RAM] Build conditional action UI