[Request] [Detection Engine] add EQL alert suppression docs for non-sequence queries

[WafaaNasr]

Description

Implementing alert suppression for EQL non-sequence-based alerts while disabling it for sequence alerts involves two main scenarios:

Scenario 1:
When a user inputs a sequence query, suppression fields are automatically disabled. A tooltip appears when hovering over any suppression field, indicating "Suppression is not enabled for EQL sequence queries."

Scenario 2:
If suppression fields are configured prior, and the user edits an EQL rule that wasn't originally a sequence query but is changed to one, a validation error occurs below the suppression group by fields. The error message states: "Suppression is not enabled for EQL sequence queries. Please reset the suppression fields." Once the user removes the suppression fields, they are disabled, following the behavior outlined in Scenario 1.

Explaining the second Scenario:
https://github.com/elastic/security-docs/assets/12671903/9eafe84e-c8dd-4c39-9c90-5c76ea2b8e9a

Details

• When users create an Eql rule, they can specify the fields they want to be grouped for suppression. For EQL rules, the minimum number of field names that can be specified is 1. The max is 3.

• The maximum number of alerts that can be suppressed is tied to the max_signals setting.

• If enabling suppression causes the EQL rule to timeout during the rule preview or when the rule is enabled and running, there are two possible ways to resolve this:

  • Turn off suppression on the rule.
  • Shorten the rule's look-back time to reduce the number of documents the rule analyzes.

• Fields with an array of values are treated as a single group and suppressed together (i.e., they're treated as a single suppression value). When you check the kibana.alert.suppression.terms field, you'll see an array of values for suppressed fields.

  1. Imagine we have the below document:

  {
            id,
            '@timestamp': timestamp,
            host: { name: ['host-a', 'host-b'] },
          };
  

  2. We indexed this document 3 times with different timestamps
  3. We have the below threat index

  {
              host: {
                    name: 'host-a',
                },
   };
  

  4. We enabled suppression for the IM rule
  5. we will end up having 1 alert generated only and the suppression count will be 2 and the suppression.terms will be an array, not a single value as in the Query rule case

   kibana.alert.suppression.terms: [
              {
                field: 'host.name',
                value: ['host-a', 'host-b'],
              },
            ],
  

• The new alert_supression param for eql rules has the same functionality as the query rule.

• The default maximum number of alerts that can be suppressed for each rule type:

  • Custom query rule: indefinite number of alerts
  • Threshold: The max_signals value (100 by default)
  • Indicator match rule: Five times the max_signals value (500 by default)
  • Eql non-sequence rule: The max_signals value (100 by default)

Background & resources

• PRs: [Detection Engine][Rule Suppression] Add Suppression to EQL Non-sequence based queries kibana#176422
• Issues/metas: https://github.com/elastic/security-team/issues/7773 https://github.com/elastic/security-team/issues/8432
• Point of contact:
• Test environments: ESS and serverless

Which documentation set does this change impact?

ESS and serverless

ESS release

8.14

Serverless release

Around 8.14 release

Feature differences

ESS: Alert suppression is available for Platinum license
Serverless: available for Essentials and Complete tiers

API docs impact

Adds alert_suppression field to Eql rule create/update/patch rule APIs

Prerequisites, privileges, feature flags

ESS: requires Platinum license
Feature Flag: alertSuppressionForNonSequenceEqlRuleEnabled

[nastasha-solomon]

@rylnd if you still need it, here are some options for the scenarios listed above.

Scenario 1

Tooltip that lets users know that alert suppression isn't supported for sequence queries:

• Original text: Suppression is not enabled for EQL sequence queries.
• Option 1: Suppression is not supported for EQL sequence queries.
• Option 2: Suppression isn't available for EQL sequence queries.
• Option 3: Suppression can't be configured for EQL sequence queries.

Scenario 2

Validation message that displays when users change a non-sequence query to a sequence query:

• Original text: Suppression is not enabled for EQL sequence queries. Please reset the suppression fields.
• Option 1: Suppression isn't supported for EQL sequence queries. Change the EQL query to a non-sequence query, or remove the suppression fields.
• Option 2: Suppression isn't supported for EQL sequence queries. You must remove all suppression fields before you can save this rule.
• Option 3: Suppression isn't supported for EQL sequence queries. To fix this error, remove the suppression fields.

Let me know if you have questions, concerns, or suggestions for any of these.

[nastasha-solomon]

Action items from meeting with @rylnd today:

• Follow-up on how fields with multiple values are suppressed. Are they suppressed like custom query and threshold alerts (alert grouping is created for each value), or are they handled like IM alerts (fields that contain an array of values are grouped and suppressed)? Or, are they handled in a completely different way?
• Figure out the target Serverless release date. This will determine when docs need to be done and when I should publish them to the Serverless docset.
• Determine whether this feature will be, or needs to be, merged behind a feature flag for Serverless. Might be helpful to discuss this more with @yctercero.
• (Optional) Review validation message that appears if users enable alert suppression on an EQL rule that uses a sequence query.

[rylnd]

Recording here for posterity: @banderror made the point that we may want to reference the current "execute but with a warning, and ignoring suppression" behavior for EQL sequence rules. We do refer to "non-sequence EQL rules" throughout this PR, but perhaps we want to reference this behavior for explicitness' sake.

[nastasha-solomon]

Thanks @banderror and @rylnd for the feedback. I think the warnings that display on the Rules page, the rule details page, and the rule execution log do a great job of communicating this temporary limitation.

Also AFAIK, the UI doesn't offer a way to configure suppression on an EQL rule that's using a sequence query. Users would have to use the create/update rule APIs if they wanted to do this, and suppression still wouldn't work after they enabled the rule. At that point, the warnings that @rylnd added the UI would pop up and inform users that their suppression configuration isn't supported.

[nastasha-solomon]

ESS and Serverless docs merged!