[Security Solution] Users can Customize Prebuilt Detection Rules: Milestone 3

[banderror]

Epic: https://github.com/elastic/security-team/issues/1974 (internal)
Milestones: << • >>

Status: Done.

Summary

Milestone 3: Add support for customizing prebuilt rules. Extend the rule upgrade workflow with full support for 3-way diffs and conflict resolution. Allow users to:

• Edit and customize prebuilt rules
• Export and import prebuilt rules, including customized ones
• Upgrade prebuilt rules while keeping the user customizations whenever possible

User stories

Prebuilt rule customization workflow:

• User can edit a single prebuilt rule
  • User can click "edit" button for prebuilt rules and customize (almost) any field on the Rule Editing page, just like it's possible to do with custom rules
  • User can't edit the Author and License fields
• User can bulk edit multiple prebuilt rules via bulk actions
• User can see if the rule is customized on the Rule Details page
• User can see which rules are customized on the Rule Management page

Prebuilt rule upgrade workflow:

• User can upgrade a single prebuilt rule to its latest version with previewing the incoming updates
  • User can preview updates from Elastic, for each rule field that has an update from Elastic
  • User can preview their customizations, for each rule field that was customized
  • User can compare their customizations with updates from Elastic and see if there are any conflicts between them, per each rule field
  • User can manually resolve conflicts between their customizations and updates from Elastic, per each rule field
  • User can edit the final field values before submitting the update
  • User can upgrade a rule if its type has been changed by Elastic in the latest version, but can only accept the incoming changes
• User can upgrade a single prebuilt rule to its latest version without previewing the incoming updates
• User can bulk upgrade multiple prebuilt rules to their latest versions

Prebuilt rule export/import workflow:

• User can export a single prebuilt rule
  • Pages: Rule Details, Rule Management
  • It can be a prebuilt non-customized or prebuilt customized rule
• User can bulk export multiple prebuilt rules via bulk actions
  • Pages: Rule Management
  • We support exporting prebuilt non-customized, prebuilt customized, and custom rules in any combination
• User can bulk import multiple prebuilt rules
  • Pages: Rule Management
  • We support importing prebuilt non-customized, prebuilt customized, and custom rules - in any combination

Useful info

• RFC: Prebuilt Rules Customization - a new technical design document describing in detail how prebuilt rule customization should work and affect other existing features.
• Prebuilt Rules Customization Technical Design - an older document from Milestone 1.

Design

Technical design

• [Security Solution] Write an RFC for customizing prebuilt rules #171309
• [Security Solution] Discuss rule fields in the context of the prebuilt rule upgrade workflow #147239
• [Security Solution] Discuss rule field diffs and conflicts to handle in the UI #148913

UI/UX design

• [Security Solution] Discuss UI/UX designs for customizing prebuilt rules #178211
• Design rule management changes (Rule Management and Details pages)
• Design rule customization changes (Rule Management and Editing pages)
• Design rule installation and upgrade changes (Add Rules page, Rule updates table, rule upgrade details flyout)
• Design rule upgrade UI for rule type changes (implementation ticket)

Preparatory changes

Preparatory changes is something we can work on before starting to hide functionality behind a feature flag. This will reduce the overall complexity introduced by feature toggling.

Missing UI for editing certain rule fields

• [Security Solution] Allow users to edit setup field for custom rules #173626
• [Security Solution] Allow users to edit max_signals field for custom rules #173593
• [Security Solution] Allow users to edit related_integrations field for custom rules #173595
• [Security Solution] Allow users to edit required_fields field for custom rules #173594

Missing UI for editing certain rule fields (docs)

• Edit setup guide field for custom rules in UI and API security-docs#4917
• Edit max_signals field for custom rules in UI and API security-docs#5029
• Edit related_integrations field for custom rules in UI and API security-docs#5099
• Edit required_fields field for custom rules in UI and API security-docs#5131

Schema-related changes

• [Security Solution] Add rule_source to the API schema #180122
• [Security Solution] Make changes in the internal rule schema #180124
• [Security Solution] Replaced the incorrect runtime type used for ruleSource #184004
• [Security Solution] Write the rule_source field together with immutable #180141
• [Security Solution] RulesManagementClient refactoring. Part 1 #180128
• [Security Solution] DetectionRulesClient refactoring. Part 2 #184364
• [Security Solution] Implement normalization on read for rule_source and immutable fields #180140

Rule customization, API changes

• [Security Solution] Calculate and save ruleSource.isCustomized in API endpoint handlers #180145
• [Security Solution] Calculate and save ruleSource.isCustomized in bulk edit API #187706
• [Security Solution] Make rule_source field required in RuleResponse #180270
• [Security Solution] Prevent from updating "non-customizable" fields in prebuilt rules #180273

Rule upgrade, API changes

• [Security Solution] Return user-customized fields from the POST /prebuilt_rules/upgrade/_review API endpoint even if they haven't been updated by Elastic in the target version #180154
• [Security Solution] Extend the POST /upgrade/_review API endpoint's contract and functionality #180153
• [Security Solution] Handle specific fields in /upgrade/_review upgrade workflow #180393
• [Security Solution] Substitute MissingVersion symbol in the ThreeWayDiff object with a boolean #188277
• [Security Solution] Extend the POST /upgrade/_perform API endpoint's contract and functionality #166376
• [Security Solution] Handle specific fields in /upgrade/_perform endpoint upgrade workflow #186544
• [Security Solution] Add InvestigationFields and AlertSuppression fields to the upgrade workflow #190597
• [Security Solution] Remove exceptions_list, author and license from Diffable Rule #196213

Rule upgrade, diff algorithms

• [Security Solution] Implement number diff algorithm #180160
• [Security Solution] Implement single-line string diff algorithm #180158
• [Security Solution] Implement multi-line string diff algorithm #180159
• [Security Solution] Implement array of scalar values diff algorithm #180162
• [Security Solution] Implement data source fields diff algorithm #187659
• [Security Solution] Implement query fields diff algorithms #187658
• [Security Solution] Implement concurrent_searches and items_per_search fields diff algorithms #188061
• [Security Solution] Implement rule type diff algorithm #190482

Fleet package with prebuilt rules

• [Security Solution] OOMs during prebuilt rules package installation #187969
• [Fleet] All package SOs are being read into memory during package upgrade or force installation #187975
• [Fleet] Split bulk delete into chunks to avoid high memory consumption #188208
• [Fleet] Split asset installation into chunks to avoid high memory pressure #189043
• [Security Solution] API endpoint for the package with prebuilt rules #187647
• [Fleet] Stream-based programmatic API for installing packages #187646
• [Security Solution] Stream-based installation of the package with prebuilt rules #192350
• [Security Solution] Smart limits for the package with prebuilt rules #187645
• Include all historical rule versions in the prebuilt rules package detection-rules#4311
• Ensure that all historical rule versions are included in the prebuilt rules package detection-rules#4312
• [Investigation] Smart Limits for Detection Rules detection-rules#4150
• [FR] [Implementation] Smart Limits for Detection Rules detection-rules#4388
• https://github.com/elastic/ia-trade-team/issues/552
• Fleet package is being released with historical rule versions (many, but not all of them)

Changes hidden behind the feature flag

These are changes that will need to be hidden behind the prebuiltRulesCustomizationEnabled feature flag.

Rule customization, UI changes

• [Security Solution] Add a new feature flag prebuiltRulesCustomizationEnabled #180130
• [Security Solution] Show if a rule is customized on the Rule Details page #180170
• [Security Solution] Support filtering by "Modified rules" and "Unmodified rules"  #180169
• [Security Solution] Add support for editing prebuilt rules to the Rule Management and Rule Details pages #180171
• [Security Solution] Add support for editing prebuilt rules to the Rule Editing page #180172

Rule upgrade, UI changes

• [Security Solution] Handle conflicting rules in the Rule Upgrade table #180589
• [Security Solution] Rework Update flyout to display all field updates and build Three-Way-Diff field component #171520
• [Security Solution] Handle specific fields in the upgrade workflow's UI #188065
• [Security Solution] Implement UI for updating prebuilt rule to a new rule type (MVP) #180395
• [Security Solution] Show Modified badge for fields customized by users #203718

Rule export and import, API and UI changes

• [Security Solution] Allow exporting prebuilt rules at the API level #180167
• [Security Solution] Allow importing prebuilt rules at the API level #180168
• [Security Solution] Support exporting prebuilt rules from the Rule Management page #180173
• [Security Solution] Support exporting prebuilt rules from the Rule Details page #180176
• [Security Solution] Benchmark performance of importing a large number of prebuilt rules #195632

Licensing

• https://github.com/elastic/security-team/issues/10410
• https://github.com/elastic/security-team/issues/11504
• https://github.com/elastic/security-team/issues/11502
• Add documentation

Before release

Bugs

• [Security Solution] Users can Customize Prebuilt Detection Rules: Milestone 3 bugs #201502

Testing

• [Security Solution] Tests for prebuilt rule customization workflow #202068
• [Security Solution] Tests for prebuilt rule upgrade workflow #202078
• [Security Solution] Tests for prebuilt rule import/export workflow #202079
• [Security Solution] Exploratory testing of prebuilt rule customization workflows #180398
• https://github.com/elastic/security-team/issues/11572

Documentation

• [Request] Prebuilt rule customization, upgrade, and export/import workflows security-docs#5061
• [Request] Prebuilt rule customization, upgrade, and export/import workflows - UI copy review security-docs#6238
• [Request] Document new rule_source property for rules in the API schema (DRAFT) security-docs#5063

Release

• [Security Solution] Add a banner to promote prebuilt rule customization in Serverless #209000
• [Security Solution] Add a banner to promote prebuilt rule customization in ESS #205594
• [Security Solution] Release prebuilt rule customization feature #180267
• [Security Solution] Remove the prebuiltRulesCustomizationEnabled feature flag #180272

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)