[Security Solution][Detections] Server-side filtering and sorting POC

[banderror]

Epic: https://github.com/elastic/security-team/issues/1972

Summary

Create a POC to verify that we will be able to implement new filters and sorting in the Rules table in the server-side mode. No changes in the UI expected (like adding new filters). We just need some test code on the backend + some test Postman requests to the API to make sure it all works.

Fields we are interested in:

• severity
• risk score
• index patterns
• MITRE ATT&CK model

Todo

• Make sure that filtering and sorting by severity and risk score work as expected (implemented in [ResponseOps] Mapped/searchable params #126531)
• Implement a POC for [ResponseOps] Support aggregations in RulesClient #125659
• Test the following fields and requests to ES for them:
  • alert.params.severity: filtering by a single and multiple values, sorting should sort by its severity level and not alphabetically
  • alert.params.riskScore: range filter, sorting
  • alert.params.index: filtering by a single and multiple values, aggregations (need to answer the questions "what index patterns do our rules have in the current space" and "how many rules have each given index pattern")
  • alert.params.threat: filtering by its properties

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[xcrzx]

✅ Checked rules filtering and sorting by severity; works as expected. Sample queries:

curl --location --request GET 'http://localhost:5601/kbn/api/detection_engine/rules/_find?filter=alert.attributes.params.severity: medium or alert.attributes.params.severity: high' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json'

curl --location --request GET 'http://localhost:5601/kbn/api/detection_engine/rules/_find?sort_field=severity&sort_order=desc' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json'

[xcrzx]

✅ Checked rules filtering and sorting by risk_score; works as expected. Sample queries:

curl --location --request GET 'http://localhost:5601/kbn/api/detection_engine/rules/_find?filter=alert.attributes.params.risk_score > 40 and alert.attributes.params.risk_score < 50' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' 

curl --location --request GET 'http://localhost:5601/kbn/api/detection_engine/rules/_find?sort_field=risk_score&sort_order=desc' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json'

[xcrzx]

With regards to filtering by threat, index, or timeline. After #126531 was merged, filtering doesn't seem to be working anymore.

curl --location --request GET 'http://localhost:5601/kbn/api/detection_engine/rules/_find?filter=alert.attributes.params.threat.tactic.name: Exfiltration' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json'

Gives the following response:

{
    "message": "This key 'alert.attributes.mapped_params.threat.tactic.name' does NOT exist in alert saved object index patterns: Bad Request",
    "status_code": 400
}

I checked the above request on a commit before the mapped_params changes were introduced, it works correctly. @JiaweiWu, was it regression or an intentional change?

cc @XavierM

Confirmed with @XavierM that it's a bug and converted to an issue: #127876

[XavierM]

With regards to filtering by threat. After #126531 was merged, filtering by threat doesn't seem to be working anymore.

curl --location --request GET 'http://localhost:5601/kbn/api/detection_engine/rules/_find?filter=alert.attributes.params.threat.tactic.name: Exfiltration' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json'

Gives the following response:

{
    "message": "This key 'alert.attributes.mapped_params.threat.tactic.name' does NOT exist in alert saved object index patterns: Bad Request",
    "status_code": 400
}

I checked the above request on a commit before the mapped_params changes were introduced, it works correctly. @JiaweiWu, was it regression or an intentional change?

cc @XavierM

UPD: getting a similar error when filtering by alert.attributes.params.index or alert.attributes.params.timeline_title.

It should be fix!!!

[xcrzx]

Checked RulesClient.aggregate method with minimal adjustments:

diff --git a/x-pack/plugins/alerting/server/rules_client/rules_client.ts b/x-pack/plugins/alerting/server/rules_client/rules_client.ts
index 1512959384a..22249dafbc8 100644
--- a/x-pack/plugins/alerting/server/rules_client/rules_client.ts
+++ b/x-pack/plugins/alerting/server/rules_client/rules_client.ts
@@ -783,6 +783,9 @@ export class RulesClient {
         muted: {
           terms: { field: 'alert.attributes.muteAll' },
         },
+        indices: {
+          terms: { field: 'alert.attributes.params.index' },
+        },
       },
     });
 
@@ -839,6 +842,11 @@ export class RulesClient {
       unmuted: mutedBuckets.find((bucket) => bucket.key === 0)?.doc_count ?? 0,
     };
 
+    ret.ruleIndices = resp.aggregations.indices.buckets.map((bucket) => ({
+      index: bucket.key,
+      rulesCount: bucket.doc_count,
+    }));
+
     return ret;
   }

It allows us to retrieve a list of all index patterns used in detection rules:

{
    "ruleIndices": [
        {
            "index": "winlogbeat-*",
            "rulesCount": 249
        },
        {
            "index": "logs-windows.*",
            "rulesCount": 239
        },
        {
            "index": "logs-endpoint.events.*",
            "rulesCount": 208
        },
        {
            "index": "logs-system.*",
            "rulesCount": 12
        },
        {
            "index": "auditbeat-*",
            "rulesCount": 7
        },
        {
            "index": "filebeat-*",
            "rulesCount": 4
        },
        {
            "index": "logs-*",
            "rulesCount": 4
        },
        {
            "index": "packetbeat-*",
            "rulesCount": 4
        },
        {
            "index": "endgame-*",
            "rulesCount": 3
        },
        {
            "index": "apm-*-transaction*",
            "rulesCount": 2
        }
    ]
}

[xcrzx]

✅ Filtering by index or threat works as expected.

curl 'http://localhost:5601/kbn/api/detection_engine/rules/_find?filter=alert.attributes.params.index: winlogbeat-*' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json'

curl 'http://localhost:5601/kbn/api/detection_engine/rules/_find?filter=alert.attributes.params.threat.tactic.name: Exfiltration' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json'