Implement EQL Correlations

[Mat0vu]

Hi everyone,

because of the issues with the case sensitiveness of ESQL (#107) and because the backend for EQL is not yet supporting correlations, although the language itself supports this kind of queries (https://www.elastic.co/guide/en/elasticsearch/reference/8.16/eql-syntax.html), I´ve started implementing this.
By the way, EQL is also supporting case-insensitive searches :D

This is still WIP as not all use cases are implemented yet and there need to be more tests. The code also depends on a newer pysigma release, but you are welcome to comment on the current status and of course help.

While working on the templates for the correlations, I´ve ran into an issue which makes a small adaption in the convert_correlation_rule_from_template method in the TextqueryBackend in the main repo necessary:

The template for correlations in EQL looks like this

default_correlation_query: ClassVar[str] = {
        "sequence": "sequence with maxspan={timespan} \n [{search}] {aggregate} {condition}",
        "sample": "sample {condition} \n [{search}] {aggregate}",
    }

The issue is the {timespan} parameter. In the current implementation, the value is inserted into the {aggregate} string, but for EQL it needs to be inserted before the {search} part, so it needs to be handled outside of the {aggregate} expression parsing.

To fix this, I´ve just added the marked line into the convert_correlation_rule_from_template method.

        return [
            template[method].format(
                search=search,
                typing=self.convert_correlation_typing(rule),
                timespan=self.convert_timespan(rule.timespan, method), # this is new
                aggregate=self.convert_correlation_aggregation_from_template(
                    rule, correlation_type, method, search
                ),
                condition=self.convert_correlation_condition_from_template(
                    rule.condition, rule.rules, correlation_type, method
                ),
            )
        ]

I think there is no other way to achieve the necessary format. But this change should not lead to undesired behavior because it won´t change the output unless there is a {timespan} in the correlation_query variable specified.

[thomaspatzke]

Nice one! Please submit the change to convert_correlation_rule_from_template as PR to pySigma!