-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution][Detections] Implement in-memory rule management table #89877
[Security Solution][Detections] Implement in-memory rule management table #89877
Conversation
Pinging @elastic/security-solution (Team: SecuritySolution) |
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
export interface RulesTableFacade { | ||
setRules(newRules: Rule[], newPagination: Partial<PaginationOptions>): void; | ||
updateRules(rules: Rule[]): void; | ||
updateOptions(filter: Partial<FilterOptions>, pagination: Partial<PaginationOptions>): void; | ||
actionStarted(actionType: LoadingRuleAction, ruleIds: string[]): void; | ||
actionStopped(): void; | ||
setShowIdleModal(show: boolean): void; | ||
setLastRefreshDate(): void; | ||
setAutoRefreshOn(on: boolean): void; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Naming as "facade" and even introduction of this guy is a temporary measure. I'd revisit this later when I start refactoring rules management logic. If you have a better naming suggestion for short term, I'd be glad to rename though.
if ( | ||
tableRef != null && | ||
tableRef.current != null && | ||
tableRef.current.changeSelection != null | ||
) { | ||
// for future devs: eui basic table is not giving us a prop to set the value, so | ||
// we are using the ref in setTimeout to reset on the next loop so that we | ||
// do not get a warning telling us we are trying to update during a render | ||
window.setTimeout(() => tableRef?.current?.changeSelection([]), 0); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wasn't sure if this is necessary, so decided to leave as is at this point.
// Why these values? | ||
// If we request more rules than 10000, we'll get an error from Elasticsearch (most likely). | ||
// https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html | ||
// https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#dynamic-index-settings | ||
// see `index.max_result_window`, default value is 10000. | ||
// Example: | ||
// In requests like `GET my-index/_search?from=40&size=20` it's not about items per page, | ||
// but overall from+size count (requested page + all previous pages). | ||
// If max_result_window=10000 this will fail: `GET my-index/_search?from=9990&size=20`. | ||
// We load "all" rules in memory at once. So, because we always request page=1: | ||
const MAX_RULES_TO_LOAD = 10000; // must not be increased | ||
const NUM_RULES_TO_LOAD = MAX_RULES_TO_LOAD / 2; // must be < MAX_RULES_TO_LOAD |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
5000 is an arguable value, please make suggestions if you will.
Unfortunately, no specific value can give us full confidence that all the rules will be loaded in all cases for all Stack deployments and users. For the time being, this should work, but we definitely need to switch back to server-side filtering later.
I should also note that loading our current 461 rules takes quite some time, and I'm not sure if loading 5000 real rules would be practical prom the perf/UX standpoint.
const projections = useProjectedState(state); | ||
|
||
return { | ||
state: { ...state, ...projections }, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here we decorate state with projected values (derived state) which allows us to leave most of the code in components untouched.
8b9d4f1
to
13df11c
Compare
f0fcdde
to
638a192
Compare
1244a12
to
e087f56
Compare
e087f56
to
bc862cf
Compare
In the team, we decided to not go this way of building the in-memory table as an interim solution. Alerting team found a promising approach to supporting filtering, searching and sorting for rule parameters in Alerting APIs (see #50213 (comment)), so we're going to invest our time in other work, e.g. in optimizing our rules-related endpoints, getting rid of "sidecar" saved objects, etc. Closing this one. |
💚 Build SucceededMetrics [docs]Module Count
Async chunks
History
To update your PR or re-run it, just comment with: |
…nt table (#91302) **Related to:** #89877 ## Summary This is based on #89877 and the kind of pre-refactoring that has been done there. Mainly this: - consolidates application logic in a single place (moves the reducer and the side effects close to each other, etc) - removes some of the redundant state, leverages the reducer as the source of truth for state - makes it easier to dispatch events, removes some of the noise While this refactoring is a totally unfinished work, and might look not good enough (or at all), still I'd like to merge it because of the logic consolidation. I'm going to finalize the refactoring later when I start implementing new filters and other table UX improvements. So the code is going to become better and maybe even quite different from what it is right now. (Btw because of that, I'm not adding or removing any tests here because this is an intermediate kind of state of the code). ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
…nt table (elastic#91302) **Related to:** elastic#89877 ## Summary This is based on elastic#89877 and the kind of pre-refactoring that has been done there. Mainly this: - consolidates application logic in a single place (moves the reducer and the side effects close to each other, etc) - removes some of the redundant state, leverages the reducer as the source of truth for state - makes it easier to dispatch events, removes some of the noise While this refactoring is a totally unfinished work, and might look not good enough (or at all), still I'd like to merge it because of the logic consolidation. I'm going to finalize the refactoring later when I start implementing new filters and other table UX improvements. So the code is going to become better and maybe even quite different from what it is right now. (Btw because of that, I'm not adding or removing any tests here because this is an intermediate kind of state of the code). ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
…nt table (#91302) (#91777) **Related to:** #89877 ## Summary This is based on #89877 and the kind of pre-refactoring that has been done there. Mainly this: - consolidates application logic in a single place (moves the reducer and the side effects close to each other, etc) - removes some of the redundant state, leverages the reducer as the source of truth for state - makes it easier to dispatch events, removes some of the noise While this refactoring is a totally unfinished work, and might look not good enough (or at all), still I'd like to merge it because of the logic consolidation. I'm going to finalize the refactoring later when I start implementing new filters and other table UX improvements. So the code is going to become better and maybe even quite different from what it is right now. (Btw because of that, I'm not adding or removing any tests here because this is an intermediate kind of state of the code). ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
…nt table (elastic#91302) **Related to:** elastic#89877 ## Summary This is based on elastic#89877 and the kind of pre-refactoring that has been done there. Mainly this: - consolidates application logic in a single place (moves the reducer and the side effects close to each other, etc) - removes some of the redundant state, leverages the reducer as the source of truth for state - makes it easier to dispatch events, removes some of the noise While this refactoring is a totally unfinished work, and might look not good enough (or at all), still I'd like to merge it because of the logic consolidation. I'm going to finalize the refactoring later when I start implementing new filters and other table UX improvements. So the code is going to become better and maybe even quite different from what it is right now. (Btw because of that, I'm not adding or removing any tests here because this is an intermediate kind of state of the code). ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios # Conflicts: # x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/all/rules_tables.tsx
Related to: #50213
Summary
While support for filtering, sorting and searching is in progress on the alerting side (see #50213 (comment)), @peluja1012 and I decided to proceed with in-memory table approach for the time being. This should allow us to build basic filters and ship some UX improvements to our users sooner.
"In-memory table approach" simply means that we load all the detection rules on the client - all at once - and implement filtering, sorting and searching in our custom code.
Scope and plan
The goal of this PR is to implement in-memory table in a sort of hack-ish way as quickly as I can. So I tried to avoid any changes unnecessary for that. If you see a lot of code to improve in this PR - I totally agree with you!
I'm planning to address other topics in follow-up PRs (rough plan):
EuiInMemoryTable
.Checklist