[SecuritySolution][SIEM migrations] Add macros and lookups support in the API

[semd]

Summary

Part of: https://github.com/elastic/security-team/issues/10653

Implements the support for resources (macros and lookup lists) for SIEM rule migrations, including the API, the persistence layer and the retrieval for the LLM agent.

Note

This feature needs siemMigrationsEnabled experimental flag enabled to work. Otherwise, no code related to SIEM migrations is executed.

Schema

The resource object schema is:

RuleMigrationResource:
  _id:
    type: string
    description: The resource id, is generated by hashing the migration_id, type, and name.
  migration_id:
    type: string
    description: The migration id
  type:      
    type: string
    description: The type of the rule migration resource.
    enum:
      - macro
      - list
  name:
    type: string
    description: The resource name identifier.
  content:
    type: string
    description: The resource content value.
  metadata:
    type: object
    description: The resource arbitrary metadata.
  updated_at:
    type: string
    description: The moment of the last update
  updated_by:
    type: string
    description: The user who last updated the resource

2 new routes are now exposed:

• POST /internal/siem_migrations/rules/{migration_id}/resources -> Creates the resources, the ones that already exist are updated.
• GET /internal/siem_migrations/rules/{migration_id}/resources -> Retrieves all the stored resources for a given migration

Resources index

A new index is created when the resources need to be stored: .kibana-siem-rule-migrations-resources-[spaceId]
The mapping is the same as the schema.

The RuleMigrationsDataClient has been extended to handle two different kinds of objects now: rules and resources.

Resource identifier

The resourceIdentifier module has been implemented (x-pack/plugins/security_solution/common/siem_migrations/rules/resources/splunk_identifier.ts) to extract the resource (macros or lists) names from the queries. There will be a different implementation for each vendor/query_language.

Resource retriever

The resourceRetriever (x-pack/plugins/security_solution/server/lib/siem_migrations/rules/task/util/rule_resource_retriever.ts) has been implemented to retrieve the resources content taking only an original (splunk) query as input, combining the resourceIdentifier and the resources content stored in the index.
It is used by the LLM agent to obtain the resources content when executing the query translation to ES|QL.

The resourceRetriever implementation is recursive, so we can extract all the nested resources, since macros may contain other resources inside (lists or other macros).

LLM Agent

A new agent call has been added to the translation node. The LLM is asked to replace all the resources in the original query with their content, so we have the query with no macro call or lookup list, everything is inline.
With the replaced query the ES|QL translation is executed as usual.

Example:

Original query:

`linux_auditd` `linux_auditd_normalized_proctitle_process`
| rename host as dest 
| where LIKE (process_exec, "%chown %root%") 
| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest 
| `security_content_ctime(firstTime)` 
| `security_content_ctime(lastTime)`
| `linux_auditd_change_file_owner_to_root_filter`

Resources:

`security_content_ctime(1)` -> convert timeformat="%Y-%m-%dT%H:%M:%S" ctime($field$)
`linux_auditd` -> sourcetype="linux:audit"
`linux_auditd_change_file_owner_to_root_filter` -> search *

Inline query:

sourcetype="linux:audit" `linux_auditd_normalized_proctitle_process`
| rename host as dest
| where LIKE (process_exec, "%chown %root%")
| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest
| convert timeformat="%Y-%m-%dT%H:%M:%S" ctime(firstTime)
| convert timeformat="%Y-%m-%dT%H:%M:%S" ctime(lastTime)
| search *

ES|QL translated query:

FROM logs-*
| WHERE process_exec LIKE "*chown *root*"
| STATS count = COUNT(*), firstTime = MIN(@timestamp), lastTime = MAX(@timestamp) BY process_exec, proctitle, normalized_proctitle_delimiter, host.name
| EVAL firstTime = DATE_FORMAT(firstTime, \"yyyy-MM-dd'T'HH:mm:ss\"), lastTime = DATE_FORMAT(lastTime, \"yyyy-MM-dd'T'HH:mm:ss\")

Considerations:

• logs-* index pattern is used as a temporary workaround while integrations RAG is being implemented.
• ECS field conversation has not been implemented yet

[michaelolo24]

side note: it would be great to track telemetry on the UI on our success and failure rates and retry count as well

[michaelolo24]

👏🏾

[michaelolo24]

👌🏾 Thanks for adding these!

[michaelolo24]

TIL 💡

[michaelolo24]

👌🏾 nice

[michaelolo24]

Thank you for addressing the comments! Excited to see the progress on this work!

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 61aae02

Failed CI Steps

• FTR Configs #29

Test Failures

• [job] [logs] FTR Configs #29 / aiops change point detection attaches change point charts to a case

Metrics [docs]

✅ unchanged

History

• 💔 Build #251962 failed 186699b
• 💛 Build #251385 was flaky cb37dfc
• 💔 Build #251154 failed 4c0dcb6
• 💔 Build #251013 failed dd1f7a1
• 💔 Build #250974 failed 553ffa3

cc @semd

[kibanamachine]

Starting backport for target branches: 8.18, 8.x

https://github.com/elastic/kibana/actions/runs/11899927531

[kibanamachine]

💔 Some backports could not be created

Status	Branch	Result
❌	8.18	The branch "8.18" does not exist
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 199370

Questions ?

Please refer to the Backport tool documentation