-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution][Detections] Refactor ML calls for newest ML permissions #74582
Conversation
This is just some cleanup, this should be functionally equivalent. This does use the new toast service, however.
This is incredibly similar to the existing useSiemJobs hook, with the following exceptions: * Does not fetch data if the user is not an ML admin * Returns additional booleans: `isMlAdmin` and `isLicensed` * uses new toasts service
Our jobs summary call returns all installed jobs regardless of group; passing groups as jobIds does not perform group filtering. This adds a helper predicate function on which to filter these results, and updates tests accordingly.
This allows us to use this predicate on both JobSummary and MlSummaryJob until we can consolidate to the former.
* Replaces existing API call with pure version * Adds hook via useAsync * Moves types to the more general ml/ folder out of ml_popover/ The JobSummary type can be retrieved from the ml plugin eventually, but this is an intermediate step until I get there.
Unlike useSecurityJobs, which additionally: * fetches uninstalled jobs * requires ML Admin permissions useInstalledSecurityJobs: * fetches only installed jobs * requires ml User permissions This hook is a lightweight replacement for the more restrictive useSiemJobs in places like the alerts table and the ML Jobs dropdown.
This was mostly a dropin replacement for the remaining use cases; there were a few now-unnecessary `isInstalled` filters that were removed.
These are helpers for our new hook, not the old one.
This is no longer specific to the ml popover, nor has it been for quite a while.
We should not be maintaining our own type here. There were two small fixes needed to make this work: * export the anomaly job types from ml/public * change the auditMessage level value from a number to a string The latter seems to have been an error, as `auditMessage.level` comes back as a string, seemingly an enum. We treat it as an enum in our own plugin, but I'll leave it as a string for now within ml.
Conflicts: x-pack/plugins/security_solution/common/machine_learning/is_security_job.ts
a8d6865
to
62b479f
Compare
* moves corresponding API mocks to new .mock format * the automock folder was actually getting in my way * Adds new mocks for our useAppToasts hook
We have this info here, best to leverage it.
This abstracts away any dealing with our raw capabilities response.
There's already a translation for this, no need for extra work here.
For now, we don't want to filter our dropdown to just installed jobs; that's a broader product decision that we'll circle back to.
0459dd6
to
be0a06c
Compare
@elasticmachine merge upstream |
💚 Build SucceededBuild metrics@kbn/optimizer bundle module count
async chunks size
page load bundle size
History
To update your PR or re-run it, just comment with: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ML changes LGTM
@@ -104,13 +100,13 @@ export const useAnomaliesTableData = ({ | |||
} | |||
} catch (error) { | |||
if (isSubscribed) { | |||
errorToToaster({ title: i18n.SIEM_TABLE_FETCH_FAILURE, error, dispatchToaster }); | |||
addError(error, { title: i18n.SIEM_TABLE_FETCH_FAILURE }); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
++ for refactoring to leverage useAppToasts
, thanks!
const _getJobsSummary = withOptionalSignal(getJobsSummary); | ||
|
||
export const useGetJobsSummary = () => useAsync(_getJobsSummary); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Really liking how the new composable hooks are turning out. Keeps the actual API request logic small and easy to grok, while making creating specific hooks like above super simple. Autocomplete is all 👍 too, so that's nice as well. Thanks for all the effort here @rylnd 🙂
it('renders a toast error if the ML call fails', async () => { | ||
(getJobsSummary as jest.Mock).mockRejectedValue('whoops'); | ||
const { waitForNextUpdate } = renderHook(() => useInstalledSecurityJobs()); | ||
await waitForNextUpdate(); | ||
|
||
expect(appToastsMock.addError).toHaveBeenCalledWith('whoops', { | ||
title: 'Security job fetch failure', | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We've had a gap in our coverage around the ML hooks for a long time, so awesome to see these tests! 🚀 🎉
@@ -31,7 +31,7 @@ export const mockGroupsResponse: Group[] = [ | |||
{ id: 'suricata', jobIds: ['suricata_alert_rate'], calendarIds: [] }, | |||
]; | |||
|
|||
export const mockOpenedJob: JobSummary = { | |||
export const mockOpenedJob: MlSummaryJob = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
++ for moving to use the ML types. More safety here now 🙂
* ml_module (whether installed or not). Use the corresponding helper functions to filter the job | ||
* list as necessary. E.g. installed jobs, running jobs, etc. | ||
* | ||
* NOTE: If the user is not an ml admin, jobs will be empty and isMlAdmin will be false. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: May be worth referencing useInstalledSecurityJobs
as an alternate here for when the user is not an ml admin
and we still need to fetch relevant jobs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's an excellent point, will do.
const [jobs, setJobs] = useState<SecurityJob[]>([]); | ||
const [loading, setLoading] = useState(true); | ||
const mlCapabilities = useMlCapabilities(); | ||
const [siemDefaultIndex] = useUiSetting$<string[]>(DEFAULT_INDEX_KEY); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const [siemDefaultIndex] = useUiSetting$<string[]>(DEFAULT_INDEX_KEY); | |
const [securitySolutionDefaultIndex] = useUiSetting$<string[]>(DEFAULT_INDEX_KEY); |
And other references below.... 👋 SIEM
@@ -133,18 +111,18 @@ export interface CustomURL { | |||
} | |||
|
|||
/** | |||
* Representation of an ML Job as used by the SIEM App -- a composition of ModuleJob and JobSummary | |||
* Representation of an ML Job as used by the SIEM App -- a composition of ModuleJob and MlSummaryJob |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
* Representation of an ML Job as used by the SIEM App -- a composition of ModuleJob and MlSummaryJob | |
* Representation of an ML Job as used by the Security Solution App -- a composition of ModuleJob and MlSummaryJob |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Performed code review, and checked out locally and tested with multiple different user permissions from read_only
, machine_learning_user
and machine_learning_admin
and all behavior has been as expected.
Left a couple nits for cleaning up some lingering siem
references, but everything else looks great. Super clean code here, and love the new tests for the ML hooks -- really appreciate the attention to detail here @rylnd! LGTM! 👍 🎉
Note: In testing, @rylnd and I found an extra call to the getJobsSummary
API on the Hosts page and it looks like there's a MatrixHistogram
being rendered in the DOM above the Anomalies table
, but it's not configured to the ML indices so it returns nothing and is not shown. Just some tech debt for us to clean up here it looks like.
…ssions (elastic#74582) ## Summary Addresses elastic#73567. ML Users (role: `machine_learning_user`) were previously able to invoke the ML Recognizer API, which we use to get not-yet-installed ML Jobs relevant to our index patterns. As of elastic#64662 this is not true, and so we receive errors from components using the underlying hook, `useSiemJobs`. To solve this I've created two separate hooks to replace `useSiemJobs`: * `useSecurityJobs` * used on ML Popover * includes uninstalled ML Jobs * checks (and returns) `isMlAdmin` before fetching data * `useInstalledSecurityJobs` * used on ML Jobs Dropdown and Anomalies Table * includes only installed ML Jobs * checks (and returns) `isMlUser` before fetching data Note that we while we now receive the knowledge to do so, we do not always inform the user in the case of invalid permissions, and instead have the following behaviors: #### User has insufficient license * ML Popover: shows an upgrade CTA * Anomalies Tables: show no data * Rule Creation: ML Rule option is disabled, shows upgrade CTA * Rule Details: ML Job Id is displayed as text #### User is ML User * ML Popover: not shown * Anomalies Tables: show no data * Rule Creation: ML Rule option is disabled * Rule Details: ML Job Id is displayed as text #### User is ML Admin * ML Popover: shown * Anomalies Tables: show data __for installed ML Jobs__ * This is the same as previous logic, but worth calling out that you can't view historical anomalies * Rule Creation: ML Rule option is enabled, all ML Jobs available * Rule Details: ML Job Id is displayed as hyperlink, job status badge shown ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
…ssions (#74582) (#74919) ## Summary Addresses #73567. ML Users (role: `machine_learning_user`) were previously able to invoke the ML Recognizer API, which we use to get not-yet-installed ML Jobs relevant to our index patterns. As of #64662 this is not true, and so we receive errors from components using the underlying hook, `useSiemJobs`. To solve this I've created two separate hooks to replace `useSiemJobs`: * `useSecurityJobs` * used on ML Popover * includes uninstalled ML Jobs * checks (and returns) `isMlAdmin` before fetching data * `useInstalledSecurityJobs` * used on ML Jobs Dropdown and Anomalies Table * includes only installed ML Jobs * checks (and returns) `isMlUser` before fetching data Note that we while we now receive the knowledge to do so, we do not always inform the user in the case of invalid permissions, and instead have the following behaviors: #### User has insufficient license * ML Popover: shows an upgrade CTA * Anomalies Tables: show no data * Rule Creation: ML Rule option is disabled, shows upgrade CTA * Rule Details: ML Job Id is displayed as text #### User is ML User * ML Popover: not shown * Anomalies Tables: show no data * Rule Creation: ML Rule option is disabled * Rule Details: ML Job Id is displayed as text #### User is ML Admin * ML Popover: shown * Anomalies Tables: show data __for installed ML Jobs__ * This is the same as previous logic, but worth calling out that you can't view historical anomalies * Rule Creation: ML Rule option is enabled, all ML Jobs available * Rule Details: ML Job Id is displayed as hyperlink, job status badge shown ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process) Co-authored-by: Ryland Herrick <ryalnd@gmail.com>
* master: (28 commits) [Task manager] Prevents edge case where already running tasks are reschedule every polling interval (elastic#74606) [Security Solution] Fix the status of timelines' bulk actions (elastic#74560) Data plugin: Suggested enhance pattern (elastic#74505) Use jest.useFakeTimers instead of hard coded timeout for tooltip tests. (elastic#74642) [Security Solution][lists] Adds tests for exception lists and items part 2 (elastic#74815) [Security Solution][Resolver] fix presentation role on edgeline (elastic#74869) [Security Solution][Detections] Refactor ML calls for newest ML permissions (elastic#74582) [bin/kibana-plugin] support KP plugins instead (elastic#74604) Reduce number of indexed fields in index pattern saved object (elastic#74817) [reporting] Pass along generic parameters in high-order route handler (elastic#74892) Migrated last pieces of legacy fixture code (elastic#74470) Empty index patterns page re-design (elastic#68819) [babel] coalese some versions to prevent breaking yarn install (elastic#74864) [Dashboard First] Decouple Attribute Service and By Value Embeddables (elastic#74302) Revert "[reporting] Pass along generic parameters in high-order route handler" (elastic#74891) [reporting] Pass along generic parameters in high-order route handler (elastic#74879) [src/dev/build] implement a getBuildNumber() mock (elastic#74881) [Enterprise Search] Add solution-level side navigation (elastic#74705) [DOCS] Canvas docs 7.9 refresh (elastic#74000) [Security Solution][Resolver]Enzyme test related events closing (elastic#74811) ...
…le-buffer-with-update-of-same-id * upstream/master: (37 commits) [Task manager] Prevents edge case where already running tasks are reschedule every polling interval (elastic#74606) [Security Solution] Fix the status of timelines' bulk actions (elastic#74560) Data plugin: Suggested enhance pattern (elastic#74505) Use jest.useFakeTimers instead of hard coded timeout for tooltip tests. (elastic#74642) [Security Solution][lists] Adds tests for exception lists and items part 2 (elastic#74815) [Security Solution][Resolver] fix presentation role on edgeline (elastic#74869) [Security Solution][Detections] Refactor ML calls for newest ML permissions (elastic#74582) [bin/kibana-plugin] support KP plugins instead (elastic#74604) Reduce number of indexed fields in index pattern saved object (elastic#74817) [reporting] Pass along generic parameters in high-order route handler (elastic#74892) Migrated last pieces of legacy fixture code (elastic#74470) Empty index patterns page re-design (elastic#68819) [babel] coalese some versions to prevent breaking yarn install (elastic#74864) [Dashboard First] Decouple Attribute Service and By Value Embeddables (elastic#74302) Revert "[reporting] Pass along generic parameters in high-order route handler" (elastic#74891) [reporting] Pass along generic parameters in high-order route handler (elastic#74879) [src/dev/build] implement a getBuildNumber() mock (elastic#74881) [Enterprise Search] Add solution-level side navigation (elastic#74705) [DOCS] Canvas docs 7.9 refresh (elastic#74000) [Security Solution][Resolver]Enzyme test related events closing (elastic#74811) ...
* upstream/master: (45 commits) [Metrics UI] Fix inventory footer misalignment (elastic#74707) Remove legacy optimizer (elastic#73154) Update design-specific GH code-owners (elastic#74877) skip test Reporting paginates content elastic#74922 [Metrics UI] Add Jest tests for alert previews (elastic#74890) Fixed tooltip (elastic#74074) [Ingest Pipelines] Processor forms for processors A-D (elastic#72849) [Observability] change ingest manager link (elastic#74928) [Task manager] Prevents edge case where already running tasks are reschedule every polling interval (elastic#74606) [Security Solution] Fix the status of timelines' bulk actions (elastic#74560) Data plugin: Suggested enhance pattern (elastic#74505) Use jest.useFakeTimers instead of hard coded timeout for tooltip tests. (elastic#74642) [Security Solution][lists] Adds tests for exception lists and items part 2 (elastic#74815) [Security Solution][Resolver] fix presentation role on edgeline (elastic#74869) [Security Solution][Detections] Refactor ML calls for newest ML permissions (elastic#74582) [bin/kibana-plugin] support KP plugins instead (elastic#74604) Reduce number of indexed fields in index pattern saved object (elastic#74817) [reporting] Pass along generic parameters in high-order route handler (elastic#74892) Migrated last pieces of legacy fixture code (elastic#74470) Empty index patterns page re-design (elastic#68819) ...
…ssions (elastic#74582) ## Summary Addresses elastic#73567. ML Users (role: `machine_learning_user`) were previously able to invoke the ML Recognizer API, which we use to get not-yet-installed ML Jobs relevant to our index patterns. As of elastic#64662 this is not true, and so we receive errors from components using the underlying hook, `useSiemJobs`. To solve this I've created two separate hooks to replace `useSiemJobs`: * `useSecurityJobs` * used on ML Popover * includes uninstalled ML Jobs * checks (and returns) `isMlAdmin` before fetching data * `useInstalledSecurityJobs` * used on ML Jobs Dropdown and Anomalies Table * includes only installed ML Jobs * checks (and returns) `isMlUser` before fetching data Note that we while we now receive the knowledge to do so, we do not always inform the user in the case of invalid permissions, and instead have the following behaviors: #### User has insufficient license * ML Popover: shows an upgrade CTA * Anomalies Tables: show no data * Rule Creation: ML Rule option is disabled, shows upgrade CTA * Rule Details: ML Job Id is displayed as text #### User is ML User * ML Popover: not shown * Anomalies Tables: show no data * Rule Creation: ML Rule option is disabled * Rule Details: ML Job Id is displayed as text #### User is ML Admin * ML Popover: shown * Anomalies Tables: show data __for installed ML Jobs__ * This is the same as previous logic, but worth calling out that you can't view historical anomalies * Rule Creation: ML Rule option is enabled, all ML Jobs available * Rule Details: ML Job Id is displayed as hyperlink, job status badge shown ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
…ssions (#74582) (#75287) ## Summary Addresses #73567. ML Users (role: `machine_learning_user`) were previously able to invoke the ML Recognizer API, which we use to get not-yet-installed ML Jobs relevant to our index patterns. As of #64662 this is not true, and so we receive errors from components using the underlying hook, `useSiemJobs`. To solve this I've created two separate hooks to replace `useSiemJobs`: * `useSecurityJobs` * used on ML Popover * includes uninstalled ML Jobs * checks (and returns) `isMlAdmin` before fetching data * `useInstalledSecurityJobs` * used on ML Jobs Dropdown and Anomalies Table * includes only installed ML Jobs * checks (and returns) `isMlUser` before fetching data Note that we while we now receive the knowledge to do so, we do not always inform the user in the case of invalid permissions, and instead have the following behaviors: #### User has insufficient license * ML Popover: shows an upgrade CTA * Anomalies Tables: show no data * Rule Creation: ML Rule option is disabled, shows upgrade CTA * Rule Details: ML Job Id is displayed as text #### User is ML User * ML Popover: not shown * Anomalies Tables: show no data * Rule Creation: ML Rule option is disabled * Rule Details: ML Job Id is displayed as text #### User is ML Admin * ML Popover: shown * Anomalies Tables: show data __for installed ML Jobs__ * This is the same as previous logic, but worth calling out that you can't view historical anomalies * Rule Creation: ML Rule option is enabled, all ML Jobs available * Rule Details: ML Job Id is displayed as hyperlink, job status badge shown ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process) Co-authored-by: Ryland Herrick <ryalnd@gmail.com>
Pinging @elastic/security-solution (Team: SecuritySolution) |
Summary
Addresses #73567.
ML Users (role:
machine_learning_user
) were previously able to invoke the ML Recognizer API, which we use to get not-yet-installed ML Jobs relevant to our index patterns. As of #64662 this is not true, and so we receive errors from components using the underlying hook,useSiemJobs
.To solve this I've created two separate hooks to replace
useSiemJobs
:useSecurityJobs
isMlAdmin
before fetching datauseInstalledSecurityJobs
isMlUser
before fetching dataNote that we while we now receive the knowledge to do so, we do not always inform the user in the case of invalid permissions, and instead have the following behaviors:
User has insufficient license
User is ML User
User is ML Admin
Checklist
For maintainers