[Security Solution] [Platform] Adds support for data views and runtime field mappings in rule creation, exceptions, and during execution

[dhurley14]

Summary

Ref: #103587 and #97778

Overview

Users will now have the option of selecting a data view or an index pattern when creating rules. Data views will be available in the EQL, Query, Threat Indicator** , and Threshold rule types.

If a rule is configured to query a data view, any runtime field mappings defined on that data view will also be available as fields to execute during the query and for any applicable override fields such as the severity override, risk score override, rule name override, and timestamp override.

** for querying source documents only. There is not support for threat indicator data views, only index patterns for now.

Affected flows

Rule Creation

• User can select to use index pattern, or data view
• Only selection of a single data view, not multiple, is supported
• Runtime fields associated with data view are viewable in preview
• Options to overwrite fields will also display runtime fields if relevant (timestamp override, risk score...)

Rule creation

Rule Preview

Rule Edit

• User can select to use index pattern, or data view
• Only selection of a single data view, not multiple, is supported
• Runtime fields associated with data view are viewable in preview
• 👀 If the selected data view no longer exists (like it was deleted), user is prompted with a warning, letting them know
  • To test create a rule with a data view, go to Stack Management --> Data Views --> delete the data view you selected, go back to the rule and click to edit

Rule Edit when error present

Rule Details

• User can see what data view they've selected
• 👀 If the selected data view no longer exists (like it was deleted), user is prompted with an error, letting them know that rule execution failed
  • To test create a rule with a data view, go to Stack Management --> Data Views --> delete the data view you selected, go back to the rule

Rule details when error present

Rules Management

• User can select to bulk edit rule index patterns
  • If they select to apply changes to rules with data views, those rules will be updated as well
  • If they do not select to apply changes to rules with data views, those rules are not updated, but the rest are
• To test:
  • Create a rule with a data view and a rule with index patterns. Go to rule management and select both rules. Select 'Bulk actions' --> Add index patterns --> go through flow to add index pattern (don't select any boxes). The rule using index patterns should be updated, the rule using data views should not be.
  • Create a rule with a data view and a rule with index patterns. Go to rule management and select both rules. Select 'Bulk actions' --> Add index patterns --> go through flow to add index pattern --> check the box Apply changes to rules configured with data views. The rule using index patterns should be updated, the rule using data views should not be.

Bulk action flyout

Bulk action flyout success toast

Add rule exception modal

• If rule list is assigned to is using runtime fields, runtime fields show as a field option to add an exception for

Example

A user has a runtime field mapping they developed on the logs-* data view in Discover

Add runtime field in Discover

And the user would now like to use that runtime field mapping as part of a query for an EQL rule (or any of the rules listed above)

Rule creation

Rule Preview

Runtime field is populated in the preview alert

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[madirey]

Looks like all the comments were addressed, thank you!

[yctercero]

LGTM - thanks so much for all your work on this feature. Have played around with it and it's looking awesome. So many edge cases you were able to cover here.

Just left some nits around dead code left in some files.

🎉 🎉 🎉

[kqualters-elastic]

think it would be a good idea to do the data fetching and munging of the result outside of useEffects, perhaps use react query or even better in a redux middleware, but outside of that the code owners changes lgtm minus some small things already commented on 👍

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: d2da1da
• Storybooks Preview
• Cloud Deployment
• Webpack Bundle Analyzer

Failed CI Steps

• FTR Configs #25

Test Failures

• [job] [logs] FTR Configs #25 / apis Machine Learning data frame analytics POST data_frame/_evaluate "before all" hook in "POST data_frame/_evaluate"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	3077	3079	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	5.2MB	5.3MB	+8.0KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	250.6KB	250.7KB	+140.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	444	443	-1

Total ESLint disabled count

id	before	after	diff
securitySolution	519	518	-1

History

• 💚 Build #51251 succeeded 56bed04
• 💚 Build #51234 succeeded 594b1c0
• 💔 Build #50920 failed 6637e05
• 💚 Build #50491 succeeded 6ef4f2a
• 💚 Build #49672 succeeded 3b39205
• 💚 Build #49452 succeeded ef10521

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @dhurley14