EQL - query language for event data in Elasticsearch #59686
Description
Meta issue consolidating the EQL functionality released in Elasticsearch 7.9 as experimental.
EQL or Event Query Language is a declarative language dedicated for identifying patterns and relationships between events.
Consider using EQL if you:
- Use Elasticsearch for threat hunting or other security use cases
- Search time-series data or logs, such as network or system logs
- Want an easy way to explore relationships between events
A good intro on EQL and its purpose is available here. The language reference can be found at this address while EQL on Elasticsearch is explained at length through a dedicated chapter.
This release includes the following features:
- event queries
- sequences
- pipes
An in-depth discussion of EQL in ES scope can be found at #49581.
Full history available here.
High-level tasks
- Create EQL plugin Create EQL Plugin projects #49583
- Extract reusable components from SQL into common project Extract common/reusable components from SQL for EQL #49773
- Synchronous EQL REST API for ad-hoc querying Create Synchronous EQL querying REST API #49634
- Build query parser and plan on reusable SQL components
- EQL transpiler for stateless EQL expressions to ES Search DSL Transpile EQL stateless expressions into ES Search DSL #49589
- Response format enhancements EQL: Response format enhancements #52845
- Support existing EQL functions EQL: Add support for existing functions #51556
- Map EQL sequence parts to ES requests Map EQL sequence/join parts to ES requests #49590
- Implement pipes logic in the plugin (not using aggregations) Implement EQL Pipes in EQL Plugin #49627
- Head and Tail pipe EQL: Add Head/Tail pipe support #58536
- Sequence implementation at EQL: Sequence/Join parsing and model #54227, EQL: Introduce support for sequences #56300, EQL: Sequence improvements #56768, EQL: Introduce support for sequence maxspan #58635, EQL: Introduce until functionality #59292
- High Level REST client support for EQL API EQL: Add High Level Rest Client #51961
- Cancelling a task's grandchildren when the task is cancelled Cancelling a task's grandchildren when the task is cancelled #50990
- Convert EQL REST API to be async (similar to async search) Make EQL REST Querying API async #49638
- Create documentation for using EQL [DOCS] Document EQL support in Elasticsearch #51057
- Telemetry for EQL usage EQL usage telemetry #49630
- Escape non-alphanumeric fields for EQL Escape non-alphanumeric fields for EQL #51443
- Remove feature flags and prepare for release Prepare EQL for release #51613