Add EQL rules and schema validation

[rw-access]

Issues

Closes #285

Summary

Added EQL to the schema for 7.10.
Also added EQL syntax/schema validation.

@rylnd I added you to this PR as a reviewer. Can you double check that everything looks good on the schema side?

[rylnd]

@rw-access I hope that I understood your request; let me know if that's not true!

[rylnd]

Should this be get_schema_from_eql?

[rylnd]

@rw-access I believe this is what you wanted 👀 on, schema-wise? I can confirm that these are the EQL-specific fields and their correct types 👍

[rw-access]

yep, thanks!

[brokensound77]

This is great, thanks! Can you create a rule with create-rule too, just to make sure everything seamlessly applied.

Also, (not specifically related to EQL) one thing to note about module-specific validation is rules using/targeting the integrations on the elastic-agent with logs-* indexes. Those would not be recognized as beats and so wouldn't know to associate the schemas. We are ok today since all those rules overlap with a beat index anyway. Retrieving these schemas may differ too.

[brokensound77]

can you change the parsed_kql property to

    def parsed_query(self):
        language = self.contents.get('language')
        if self.query and language in ('kuery', 'eql'):
            return kql.parse(self.query) if language == 'kuery' else eql.parse_query(self.query)

Then you can use that here (and more consistently as needed)

[rw-access]

i can't use it here because this is a static method and needs to be for caching to work.

but i did update the method regardless, even though it's never used

[rw-access]

@brokensound77

$  python -m detection_rules create-rule x.toml -t eql

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

rule_id [fa8d139a-4075-4ae9-a198-305fd0a88450] ("n/a" to leave blank)  (required): 
actions (multi, comma separated): 
description (required): Test rule description.
enabled [false] ("n/a" to leave blank): 
from [now-6m] ("n/a" to leave blank): 
false_positives (multi, comma separated): 
filters (multi, comma separated): 
interval [5m] ("n/a" to leave blank): 
max_signals [100] ("n/a" to leave blank): 
meta: 
name (required): Test EQL rule
note: 
references (multi, comma separated): 
risk_score [21] ("n/a" to leave blank)  (required): 
severity [low] ("n/a" to leave blank)  (required): 
tags (multi, comma separated): 
throttle: 
timeline_id: 
timeline_title: 
to [now] ("n/a" to leave blank): 
add mitre tactic? [y/N]: 
index (multi, comma separated): 
language [eql] ("n/a" to leave blank)  (required): 
query (required): process where true
author (required)  (multi, comma separated): Elastic
building_block_type: 
exceptions_list (multi, comma separated): 
license [Elastic License] ("n/a" to leave blank)  (required): 
risk_score_mapping (multi, comma separated): 
rule_name_override: 
severity_mapping (multi, comma separated): 
timestamp_override: 
ecs_version (multi, comma separated): 
Rule Test EQL rule saved to x.toml
Did not set the following values because they are un-required when set to the default value
 - from
 - interval
 - max_signals
 - to