[FR] Add support for dataviews in the rule schema

[Mikaayenson]

Issues

Resolves #2096

Summary

Add's support for Data Views within the rule schema data class. We're adding the field so that users can manage their custom rules with this field as an alternative to using the index field.

Note: We do not validate the data_view_id as the inputs are arbitrary and configured in Elastic Security.

Additional Context:

• I didn't update unit tests to accommodate the feature as the validation occurs within the data classes.
• There were not min_compat restrictions to limit the feature to a specified minimum stack version since it came in 8.4
• I executed all unit tests locally with a test toml rule to confirm passing.
• I double checked the documentation and test stack to confirm index and data_view_id could not be supplied at the same time.
• Covered edge cases when the index field is not supplied.
• Included @dhurley14 (Kibana PR implementer) as an optional reviewer for insights on the feature.

Testing

1. Exported detection rule from Kibana with the data view added.

Details

{"id":"fad62ac0-7358-11ee-b1a2-d102d5e21efd","updated_at":"2024-03-13T14:01:33.351Z","updated_by":"513814043","created_at":"2023-10-25T17:07:29.786Z","created_by":"513814043","name":"test","tags":[],"interval":"5m","enabled":false,"revision":4,"description":"ere add aims aewerdas","risk_score":21,"severity":"low","license":"","output_index":"","meta":{"from":"1m","kibana_siem_app_url":"https://stryker8-10.kb.europe-west1.gcp.cloud.es.io:9243/app/security"},"author":[],"false_positives":[],"from":"now-360s","rule_id":"8f6eb3b6-e9f2-4c10-a72b-cf48b4e90c2d","max_signals":100,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":1,"exceptions_list":[{"id":"731032b5-429f-4ed7-a305-47399e9e6c6f","list_id":"119388ba-a5c6-4acf-a069-80aaf400e80a","type":"rule_default","namespace_type":"single"}],"immutable":false,"related_integrations":[],"required_fields":[],"setup":"","type":"saved_query","language":"kuery","saved_id":"c4078ed0-7358-11ee-b1a2-d102d5e21efd","data_view_id":"logs-*","response_actions":[{"params":{"command":"isolate","comment":"test"},"action_type_id":".endpoint"}],"actions":[]}
{"_version":"WzE1NDQ0NSwyNV0=","created_at":"2024-02-16T02:25:26.850Z","created_by":"513814043","description":"Exception list containing exceptions for rule with id: fad62ac0-7358-11ee-b1a2-d102d5e21efd","id":"731032b5-429f-4ed7-a305-47399e9e6c6f","immutable":false,"list_id":"119388ba-a5c6-4acf-a069-80aaf400e80a","name":"Exceptions for rule - test","namespace_type":"single","os_types":[],"tags":["default_rule_exception_list"],"tie_breaker_id":"8fc87b5d-7830-4948-b9d3-0dd4b7a0f33e","type":"rule_default","updated_at":"2024-02-16T02:25:26.850Z","updated_by":"513814043","version":1}
{"_version":"WzE1NDQ0NiwyNV0=","comments":[],"created_at":"2024-02-16T02:25:28.718Z","created_by":"513814043","description":"Exception list item","entries":[{"field":"Persistence.name","operator":"included","type":"match","value":"com.jamf.management.agent"}],"id":"397326c4-7278-4122-92c8-873bd31537bc","item_id":"a7b385ca-9957-4c6c-9f1f-8eaf751bf3c8","list_id":"119388ba-a5c6-4acf-a069-80aaf400e80a","name":"mytest","namespace_type":"single","os_types":[],"tags":[],"tie_breaker_id":"72869632-da73-4b9c-bd09-cc98169f7689","type":"simple","updated_at":"2024-02-16T02:25:28.719Z","updated_by":"513814043"}
{"exported_count":3,"exported_rules_count":1,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":1,"exported_exception_list_item_count":1,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0,"exported_action_connector_count":0,"missing_action_connection_count":0,"missing_action_connections":[],"excluded_action_connection_count":0,"excluded_action_connections":[]}

2. Converted the exported ndjson file(s) to toml in the detection-rules repo

Details

[metadata]
creation_date = "2024/03/13"
maturity = "development"
updated_date = "2024/03/13"

[rule]
author = ["me"]
data_view_id = "logs-*"
description = "ere add aims aewerdas"
language = "eql"
name = "test"
risk_score = 21
rule_id = "8f6eb3b6-e9f2-4c10-a72b-cf48b4e90c2d"
severity = "low"
type = "eql"

query = '''
process where true
'''

3. Re-exported the toml rule(s) to ndjson and re-imported into Kibana

Details

{"author": ["me"], "data_view_id": "logs-*", "description": "ere add aims aewerdas", "language": "eql", "name": "test", "query": "process where true\n", "risk_score": 21, "rule_id": "8f6eb3b6-e9f2-4c10-a72b-cf48b4e90c2d", "severity": "low", "type": "eql", "version": 1}

[brokensound77]

🚀

[dhurley14]

Hi @Mikaayenson 👋

Thanks for the tag. Currently there are no limitations on the rule creation api in kibana for index XOR data_view_id. If a data_view_id field is present on a rule, regardless of whether an index property is defined on that rule, the default action for that rule will be to query the data view associated with that id. You can test this for yourself via the following script and sample rule (pulled + modified from here):

Script to post a rule to api

$ cat post_rule.sh 
#!/bin/sh

#
# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
# or more contributor license agreements. Licensed under the Elastic License
# 2.0; you may not use this file except in compliance with the Elastic License
# 2.0.
#

set -e
./check_env_variables.sh

# Uses a default if no argument is specified
RULES=(${@:-./rules/queries/query_with_rule_id.json})

# Example: ./post_rule.sh
# Example: ./post_rule.sh ./rules/queries/query_with_rule_id.json
# Example glob: ./post_rule.sh ./rules/queries/*
for RULE in "${RULES[@]}"
do {
  [ -e "$RULE" ] || continue
  curl -s -k \
  -H 'Content-Type: application/json' \
  -H 'kbn-xsrf: 123' \
  -H 'elastic-api-version: 2023-10-31' \
  -u ${ELASTICSEARCH_USERNAME}:${ELASTICSEARCH_PASSWORD} \
  -X POST ${KIBANA_URL}${SPACE_URL}/api/detection_engine/rules \
   -d @${RULE} \
  | jq -S .;
} &
done

wait

Sample rule

$ cat rules/queries/simplest_query.json 
{
  "name": "Simplest Query",
  "description": "Simplest query with the least amount of fields required",
  "risk_score": 1,
  "severity": "high",
  "type": "query",
  "query": "user.name: root or user.name: admin",
  "index": ["myfakeindex*"],
  "data_view_id": "myfakedataviewid"
}

Happy to approve the PR since this is probably something you want to ensure doesn't happen for the rules developed here but just wanted to clarify the exact logic from the API.

[Mikaayenson]

Hi @Mikaayenson 👋

Thanks for the tag. Currently there are no limitations on the rule creation api in kibana for index XOR data_view_id. If a data_view_id field is present on a rule, regardless of whether an index property is defined on that rule, the default action for that rule will be to query the data view associated with that id. You can test this for yourself via the following script and sample rule (pulled + modified from here):

Script to post a rule to api
Sample rule
Happy to approve the PR since this is probably something you want to ensure doesn't happen for the rules developed here but just wanted to clarify the exact logic from the API.

Thanks! I updated to remove the explicit check for both fields.