[FR] Update schemas to support runtime fields

[Mikaayenson]

Summary

elastic/kibana#130929

Tasks

PR Checklist

Preview Give feedback

1. Link to the relevant Kibana PR or issue provided
2. Exported detection rule(s) from Kibana to showcase the feature(s)
3. Converted the exported ndjson file(s) to toml in the detection-rules repo
4. Re-exported the toml rule(s) to ndjson and re-imported into Kibana
5. Updated necessary unit tests to accommodate the feature
6. Applied min_compat restrictions to limit the feature to a specified minimum stack version
7. Executed all unit tests locally with a test toml rule to confirm passing
8. Included Kibana PR implementer as an optional reviewer for insights on the feature
9. Implemented requisite downgrade functionality
10. Cross-referenced the feature with product documentation for consistency
11. Incorporated a comprehensive test rule in unit tests for full schema coverage
12. Conducted system testing, including fleet, import, and create APIs

Dependencies and Constraints

...

[Mikaayenson]

Update Aug 15

Now that we support custom schemas as a beta feature, we can probably close this out. We just need to build the runtime field in a stack and try to use DAC features with it to double check.

1. Following the Example in the summary description of [Security Solution] [Platform] Adds support for data views and runtime field mappings in rule creation, exceptions, and during execution kibana#130929 add a runtime field and then create a test rule
2. Setup a custom DAC setup to test

python -m detection_rules custom-rules setup-config custom_rules

then add this config

bbr_rules_dirs:
- rules_building_block
directories:
  action_connector_dir: action_connectors
  action_dir: actions
  exception_dir: exceptions
files:
  deprecated_rules: etc/deprecated_rules.json
  packages: etc/packages.yaml
  stack_schema_map: etc/stack-schema-map.yaml
  version_lock: etc/version.lock.json
rule_dirs:
- rules
testing:
  config: etc/test_config.yaml
bypass_version_lock: True
normalize_kql_keywords: True
auto_gen_schema_file: "etc/schemas/auto_gen.json"
bypass_optional_elastic_validation: True

3. Use the import / export commands to try importing and exporting the new rule created.

python -m detection_rules export-rules-from-repo  : TOML --> NDJSON
python -m detection_rules import-rules-to-repo  : NDJSON --> TOML
python -m detection_rules kibana export-rules  : Pull directly from Elastic Security --> TOML
python -m detection_rules kibana import-rules : Push from local TOML --> Elastic Security

[shashank-elastic]

• Created Custom Run Time Filed

• Sample Test Rule Created

• Setup a custom DAC

Details

detection-rules on  issue-3509 [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ python -m detection_rules custom-rules setup-config custom_rules

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Created directory: custom_rules/actions
Created directory: custom_rules/action_connectors
Created directory: custom_rules/exceptions
Created directory: custom_rules/rules
Created directory: custom_rules/rules_building_block
Created directory: custom_rules/etc
Created file with default content: custom_rules/etc/deprecated_rules.json
Created file with default content: custom_rules/etc/version.lock.json
Created file with default content: custom_rules/etc/packages.yaml
Created file with default content: custom_rules/etc/stack-schema-map.yaml
Created file with default content: custom_rules/etc/test_config.yaml
Created file with default content: custom_rules/_config.yaml

# For details on how to configure the _config.yaml file,
# consult: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/_config.yaml
# or the docs: /Users/shashankks/elastic_workspace/detection-rules/docs/custom-rules.md
(.venv) 
detection-rules on  issue-3509 [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ 

Updated Config

• The Import of the Sample Rule Failed

Details

python -m detection_rules import-rules-to-repo /Users/shashankks/Downloads/test_run_time_filed.ndjson

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building rule for /Users/shashankks/elastic_workspace/detection-rules/rules/run_time_filed_mapping_to_data_views.toml
actions (multi, comma separated):
alert_suppression:
building_block_type:
event_category_override:
exceptions_list (multi, comma separated):
false_positives (multi, comma separated):
filters (multi, comma separated):
index (multi, comma separated):
investigation_fields:
license:
note:
references (multi, comma separated):
related_integrations (multi, comma separated):
required_fields (multi, comma separated):
risk_score_mapping (multi, comma separated):
rule_name_override:
setup:
severity_mapping (multi, comma separated):
tags (multi, comma separated):
add mitre tactic? [y/N]: N
throttle:
tiebreaker_field:
timeline_id:
timeline_title:
timestamp_field:
timestamp_override:
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/main.py", line 35, in
main()
File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/main.py", line 32, in main
root(prog_name="detection_rules")
File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/main.py", line 177, in import_rules_into_repo
output = rule_prompt(
^^^^^^^^^^^^
File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/cli_utils.py", line 237, in rule_prompt
raise e
File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/cli_utils.py", line 206, in rule_prompt
rule = TOMLRule(path=Path(path), contents=TOMLRuleContents.from_dict({'rule': contents, 'metadata': meta}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/mixins.py", line 142, in from_dict
return schema.load(obj)
^^^^^^^^^^^^^^^^
File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/marshmallow_dataclass/init.py", line 910, in load
all_loaded = super().load(data, many=many, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/marshmallow/schema.py", line 722, in load
return self._do_load(
^^^^^^^^^^^^^^
File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/marshmallow/schema.py", line 884, in _do_load
self._invoke_schema_validators(
File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/marshmallow/schema.py", line 1185, in _invoke_schema_validators
self._run_validator(
File "/Users/shashankks/elastic_workspace/detection-rules/.venv/lib/python3.12/site-packages/marshmallow/schema.py", line 774, in _run_validator
validator_func(output, partial=partial, many=many)
File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule.py", line 1355, in post_conversion_validation
data.validate_query(metadata)
File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule.py", line 735, in validate_query
return validator.validate(self, meta)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_validators.py", line 363, in validate
raise validation_checks["stack"]
eql.errors.EqlSchemaError: Error at line:1,column:11
Field not recognized
any where logs-data-view-files == "logs data view filed"
^^^^^^^^^^^^^^^^^^^^^^
stack: 8.16.0, beats: 8.15.0,ecs: 8.11.0, endgame: 8.4.0
(.venv)

Sample ndjson of the exported rule is like below

{"id":"c234a270-5e5d-41c9-9237-f716c36ac34c","updated_at":"2024-09-03T17:07:07.929Z","updated_by":"841510929","created_at":"2024-09-03T17:07:07.929Z","created_by":"841510929","name":"Run Time Filed Mapping to Data Views","tags":[],"interval":"5m","enabled":true,"revision":0,"description":"Test Run Time Filed Mapping to Data Views","risk_score":21,"severity":"low","license":"","output_index":"","meta":{"from":"1m","kibana_siem_app_url":"https://e2erelease.kb.us-west2.gcp.elastic-cloud.com:9243/app/security"},"author":[],"false_positives":[],"from":"now-360s","rule_id":"a97cf517-bb1c-4d46-9522-f449fd3b0873","max_signals":100,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":1,"exceptions_list":[],"immutable":false,"rule_source":{"type":"internal"},"related_integrations":[],"required_fields":[],"setup":"","type":"eql","language":"eql","data_view_id":"logs-*","query":"any where `logs-data-view-files` == \"logs data view filed\"","filters":[],"actions":[]}
{"exported_count":1,"exported_rules_count":1,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":0,"exported_exception_list_item_count":0,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0,"exported_action_connector_count":0,"missing_action_connection_count":0,"missing_action_connections":[],"excluded_action_connection_count":0,"excluded_action_connections":[]}

I have double checked twice my steps, hopefully i am not missing anything here! @Mikaayenson

[Mikaayenson]

The only thing im not seeing is if you exported the CUSTOM_RULES_DIR environment variable. Also can you share the contents of auto_gen.json

[shashank-elastic]

The only thing im not seeing is if you exported the CUSTOM_RULES_DIR environment variable. Also can you share the contents of auto_gen.json

Yes the environment variable export was missing from the steps, and that fixed the problem

Successful Import of Rule with Run Time Field

❯ python -m detection_rules import-rules-to-repo /Users/shashankks/Downloads/test_run_time_filed.ndjson --required-only 

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building rule for /Users/shashankks/elastic_workspace/detection-rules/custom_rules/rules/run_time_filed_mapping_to_data_views.toml
1 results exported
1 rules converted
0 exceptions exported
0 actions connectors exported
(.venv) 
detection-rules on  issue-3509 [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 2s

AutoGen Schema populated successfully

Successful Export of Rule with Run Time Field

❯ python -m detection_rules export-rules-from-repo -id "a97cf517-bb1c-4d46-9522-f449fd3b0873"
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Exported 1 rules into /Users/shashankks/elastic_workspace/detection-rules/exports/20240903T234353L.ndjson
(.venv) 
detection-rules on  issue-3509 [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️  shashank.suryanarayana@elastic.co 
❯ 

Successful Export of Rule with Run Time Field From Kibana

❯ python -m detection_rules kibana export-rules -r "a97cf517-bb1c-4d46-9522-f449fd3b0873" -d /Users/shashankks/elastic_workspace/detection-rules/custom_rules
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

1 results exported
1 rules converted
0 exceptions exported
0 action connectors exported
1 rules saved to /Users/shashankks/elastic_workspace/detection-rules/custom_rules
0 exception lists saved to /Users/shashankks/elastic_workspace/detection-rules/custom_rules/exceptions
0 action connectors saved to /Users/shashankks/elastic_workspace/detection-rules/custom_rules/action_connectors
(.venv) 
detection-rules on  issue-3509 [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 3s 

Successful Import of Rule with Run Time Field To Kibana

Expected failure as the rule already exists

❯ python -m detection_rules kibana import-rules -id "a97cf517-bb1c-4d46-9522-f449fd3b0873" 
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

1 rule(s) failed to import!
 - a97cf517-bb1c-4d46-9522-f449fd3b0873: (409) rule_id: "a97cf517-bb1c-4d46-9522-f449fd3b0873" already exists
(.venv) 
detection-rules on  issue-3509 [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 3s 
❯ 

Changed rule id and name just to test

❯ python -m detection_rules kibana import-rules -id "b97cf517-bb1c-4d46-9522-f449fd3b0873" 
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

1 rule(s) successfully imported
 - b97cf517-bb1c-4d46-9522-f449fd3b0873
(.venv) 
detection-rules on  issue-3509 [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 4s 
❯ 

@Mikaayenson With DAC the feature is working as expected. This should be good to close

[shashank-elastic]

All verification steps completed