Support for event context correlation rules.

[lclin56]

Motivation

Falco currently seems to lack support for context correlation and event counting, which limits its utility in certain scenarios. For instance, in remote download and execution scenarios, I require at least two events to trigger a rule. Similarly, for IP scanning, when a program attempts to connect to a large number of IPs, I want to be able to trigger the IP scanning rule. The absence of these functionalities makes Falco less effective in handling complex security threats.

Feature

I would like Falco to add support for context correlation and event counting. Specifically, for remote download and execution scenarios, Falco should be able to trigger an alert based on at least two related events (e.g., file download and execution). For IP scanning, the ability to trigger a rule when consecutive attempts to connect to multiple IPs are detected is desired. This capability would enable Falco to more effectively identify and respond to complex security threats.

Alternatives

The current alternatives include using additional tools to supplement Falco's capabilities, but this adds to the system's complexity and maintenance difficulty. Having these features built into Falco would greatly simplify the setup and maintenance of security monitoring.

Additional context

Incorporating these features would make Falco a more comprehensive security monitoring tool, better suited to adapt to the evolving landscape of security threats.

[incertum]

Hi @lclin56 would you want to check out the anomaly detection proposal falcosecurity/falco#2655?
It aims to feature "probabilistic counting" on the host itself.

However,

when a program attempts to connect to a large number of IPs

this is an aggregation query, e.g. "distinct count" in SQL language and has to be performed over (micro) batches and therefore is not yet part of the planned "probabilistic counting" approach.

Re correlating rules, I agree, I was already thinking about how to effectively do this for Falco upstream. Likely it should be part of the planned anomaly detection module, but it's a bit of a separate use case than the counting. This may be more feasible near-term.

In summary, I agree with all you are saying, it's just very challenging to do this all on the host given the resource utilization constraints. Looking forward to hearing more thoughts you may want to share.

---

Sometimes more powerful kernel signals can solve the problem without needing stats or event correlations.

Falco should be able to trigger an alert based on at least two related events (e.g., file download and execution)

For containers you may be interested in exploring the proc.is_exe_upper_layer field. It is addressing exactly the TTP you mentioned. The stable rule Drop and execute new binary in container may serve as example rule. Lastly see also the Falco release 0.34 post.

[lclin56]

I'm glad you're considering my proposal. The anomaly detection module you mentioned may indeed meet my needs, and I hope to see more information about this part. Regarding the resource issues you mentioned, I wonder if Falco's rule engine should focus more on the completeness of the function rather than performance? Because in some scenarios, we might use Falco as an engine for analyzing Sysdig dump event files, rather than a runtime detection scenario on the host. Thanks again for your reply.

[incertum]

I'm glad you're considering my proposal. The anomaly detection module you mentioned may indeed meet my needs, and I hope to see more information about this part.

One more note: If it would be ok to not do micro / time bound batch analyses for the "program attempts to connect to a large number of IPs" use case and instead also treat it as a real-time stream problem, the HyperLogLog algorithm could also be featured in addition to the Count Min Sketch. But some details and challenges need to be discussed as the "distinct counts" would need to be partitioned by a key (in your example the process).

Regarding the resource issues you mentioned, I wonder if Falco's rule engine should focus more on the completeness of the function rather than performance? Because in some scenarios, we might use Falco as an engine for analyzing Sysdig dump event files, rather than a runtime detection scenario on the host. Thanks again for your reply.

For the runtime detection features we don't have much of a choice other than carefully considering a balance of capabilities / feature completeness and performance. This is because security monitoring tools never have unlimited budgeting. And coming back to the completeness aspect -- if you are not careful on the "hot path" / meaning the kernel event stream code flow, you can add back pressure and drop events in the kernel driver which is undesirable as well.

we might use Falco as an engine for analyzing Sysdig dump event files, rather than a runtime detection scenario on the host

Perhaps this could be tracked as a separate issue? This feature request would equate to transforming all data into a queryable table and full SQL like support. It could be done. For such major feature requests we would ask you to open a formal proposal.

I know you want to avoid "using additional tools to supplement Falco's capabilities". For now I would recommend still checking out some more generic data analytics frameworks. It's the current common approach: Piping Falco rules outputs into data lake systems for more detailed analyses off the host.

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

[poiana]

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

[poiana]

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.