[RAC] - Update field names #107857

yctercero · 2021-08-08T08:32:03Z

Summary

Trying to align on field naming. In the code there seems to be the following that are all referencing the context in which a rule instance was created (rules created in stack have consumer of alerts):

rule.consumers
kibana.alert.consumer
kibana.alert.consumers
kibana.alert.rule.consumers
kibana.alert.owner

In the code there are also the following all referencing a rule's alert type (ex: apm.error_rate):

rule.id
kibana.alert.rule.rule_type_id

It's my understanding that these are all meant to refer to the same thing - the single feature which created the rule instance. So if a rule was created in security solution, the consumer is siem, if created in apm it's apm. Observability and the rule registry were using kibana.alert.owner though it'd been on the list to update to kibana.alert.consumer and security solution was using kibana.alert.owners. The other ones were there as fields but not being used.

I think there is some confusion as to what consumer is meant to denote. It should denote the same thing it does for rules rbac. If there's other context we want to save on an alert, that should be considered a different field, the consumer field was meant to be specific to role based access logic shared with alerting framework.

Fields used moving forward

kibana.alert.rule.consumer will refer to the context in which a rule instance is created. Rules created in:

stack --> alerts
security solution --> siem
apm --> apm

kibana.alert.rule.producer will refer to the plugin that registered a rule type. Rules registered in:

stack --> alerts
security solution --> siem
apm --> apm

So an apm.error_rate rule created in stack will have:

consumer: alerts and producer: apm
An apm.error_rate rule created in apm will have:
consumer: apm and producer: apm

kibana.alert.rule.rule_type_id will refer to a rule's rule type id. Examples:

apm.error_rate
siem.signals
siem.threshold

Also renamed the following because rule.* fields are meant to be ecs fields pulled from the source/event document, not refer to our rule fields.
rule.name --> kibana.alert.rule.name will refer to the rule's name.

rule.category --> kibana.alert.rule.category will refer to the rule's category.

rule.id --> kibana.alert.rule.uuid will refer to the rule's uuid.

To test

Add the following to your kibana.dev.yml:

xpack.securitySolution.enableExperimental: ['tGridEnabled']

Create a custom rule with query *:*.

Run yarn test:generate within x-pack/plugins/security_solution

Observe the magic (alerts should show up in table).

x-pack/plugins/rule_registry/server/utils/create_lifecycle_rule_type.test.ts

elasticmachine · 2021-08-09T13:35:34Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2021-08-09T13:35:35Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

elasticmachine · 2021-08-09T13:35:37Z

Pinging @elastic/security-solution (Team: SecuritySolution)

…ld_names

elasticmachine · 2021-08-09T13:45:25Z

Pinging @elastic/apm-ui (Team:apm)

ecezalp

LGTM - thanks for consolidating the fields!

...lib/detection_engine/rule_execution_log/rule_registry_log_client/rule_registry_log_client.ts

x-pack/plugins/timelines/server/search_strategy/timeline/index.ts

michaelolo24

Thank you for doing this! Tested locally with @machadoum and looks great, just one minor comment 😄

gmmorris

Reviewed the Alerting code- looks like the only change is the input and output of a couple of unit tests and looks like it was made to make maintenance easier so all I can say is thanks Yara :)

LGTM

michaelolo24 · 2021-08-10T21:08:02Z

Just a comment to uncomment these when it's all sorted: #108034

yctercero · 2021-08-10T21:30:29Z

packages/kbn-rule-data-utils/src/technical_field_names.ts

@@ -17,25 +17,18 @@ const CONSUMERS = `${KIBANA_NAMESPACE}.consumers` as const;
 const ECS_VERSION = 'ecs.version' as const;
 const EVENT_ACTION = 'event.action' as const;
 const EVENT_KIND = 'event.kind' as const;
-const RULE_CATEGORY = 'rule.category' as const;


rule.* should only refer to source/event document fields, not our kibana rules.

yctercero · 2021-08-10T21:36:49Z

x-pack/plugins/rule_registry/common/assets/field_maps/technical_rule_field_map.ts

-  [Fields.ALERT_PRODUCER]: { type: 'keyword' },
+  [Fields.ALERT_RULE_TYPE_ID]: { type: 'keyword', required: true },
+  [Fields.ALERT_RULE_CONSUMER]: { type: 'keyword', required: true },
+  [Fields.ALERT_RULE_PRODUCER]: { type: 'keyword' },


Producer can be derived from the rule type id (apm.error_count) --> apm , so while convenient, we don't need to require it as long as we have the consumer and rule type id, we have enough for rbac.

…ld_names

dgieselaar

APM changes LGTM!

yctercero · 2021-08-11T07:31:51Z

@elasticmachine merge upstream

…ield_names

yctercero · 2021-08-11T08:11:37Z

Just a comment to uncomment these when it's all sorted: #108034

@michaelolo24 Note that after most recent CI failure, took the opportunity to make these changes here so once this goes in, should be golden 👍🏽

kibanamachine · 2021-08-11T10:25:43Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	4.3MB	4.3MB	-662.0B
`observability`	481.8KB	480.8KB	-1015.0B
`securitySolution`	6.5MB	6.5MB	-4.0B
total			-1.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`apm`	44.9KB	44.7KB	-225.0B
`infra`	150.1KB	149.9KB	-225.0B
`uptime`	35.7KB	35.4KB	-225.0B
total			-675.0B

History

💔 Build #144319 failed cb03411142d18ddfbb531f1963e9bab38a14802c
💔 Build #144294 failed e4b9d3c
💔 Build #144282 failed fc44fcc
💔 Build #144244 failed 37614f4
💔 Build #144233 failed 15b9719

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @yctercero

### Summary ### Fields used moving forward `kibana.alert.rule.consumer` will refer to the context in which a rule instance is created. Rules created in: - stack --> `alerts` - security solution --> `siem` - apm --> `apm` `kibana.alert.rule.producer` will refer to the plugin that registered a rule type. Rules registered in: - stack --> `alerts` - security solution --> `siem` - apm --> `apm` So an `apm.error_rate` rule created in stack will have: - consumer: `alerts` and producer: `apm` An `apm.error_rate` rule created in apm will have: - consumer: `apm` and producer: `apm` `kibana.alert.rule.rule_type_id` will refer to a rule's rule type id. Examples: - `apm.error_rate` - `siem.signals` - `siem.threshold` Also renamed the following because `rule.*` fields are meant to be ecs fields pulled from the source/event document, not refer to our rule fields. `rule.name` --> `kibana.alert.rule.name` will refer to the rule's name. `rule.category` --> `kibana.alert.rule.category` will refer to the rule's category. `rule.id` --> `kibana.alert.rule.uuid` will refer to the rule's uuid.

kibanamachine · 2021-08-11T10:28:50Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

### Summary ### Fields used moving forward `kibana.alert.rule.consumer` will refer to the context in which a rule instance is created. Rules created in: - stack --> `alerts` - security solution --> `siem` - apm --> `apm` `kibana.alert.rule.producer` will refer to the plugin that registered a rule type. Rules registered in: - stack --> `alerts` - security solution --> `siem` - apm --> `apm` So an `apm.error_rate` rule created in stack will have: - consumer: `alerts` and producer: `apm` An `apm.error_rate` rule created in apm will have: - consumer: `apm` and producer: `apm` `kibana.alert.rule.rule_type_id` will refer to a rule's rule type id. Examples: - `apm.error_rate` - `siem.signals` - `siem.threshold` Also renamed the following because `rule.*` fields are meant to be ecs fields pulled from the source/event document, not refer to our rule fields. `rule.name` --> `kibana.alert.rule.name` will refer to the rule's name. `rule.category` --> `kibana.alert.rule.category` will refer to the rule's category. `rule.id` --> `kibana.alert.rule.uuid` will refer to the rule's uuid. Co-authored-by: Yara Tercero <yctercero@users.noreply.github.com>

yctercero added 3 commits August 7, 2021 22:02

updating security solution to use kibana.alert.consumer

c921579

removing unused fields

3b2fa1f

get timeline search strategy using the correct fields

206a9f6

yctercero self-assigned this Aug 9, 2021

yctercero added Feature:RAC label obsolete Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detections and Resp Security Detection Response Team Team:Threat Hunting Security Solution Threat Hunting Team v7.15.0 v8.0.0 labels Aug 9, 2021

michaelolo24 mentioned this pull request Aug 9, 2021

[RAC] [RBAC] Adds bulk update route to rule registry and bulk update function to alerts client #106297

Merged

3 tasks

yctercero changed the title ~~Update ss field names~~ [RAC] - Update field names Aug 9, 2021

yctercero commented Aug 9, 2021

View reviewed changes

x-pack/plugins/rule_registry/server/utils/create_lifecycle_rule_type.test.ts Outdated Show resolved Hide resolved

updating snapshots and types

864e746

yctercero marked this pull request as ready for review August 9, 2021 13:35

yctercero requested review from a team as code owners August 9, 2021 13:35

Merge branch 'master' of github.com:elastic/kibana into update_ss_fie…

9820d67

…ld_names

botelastic bot added the Team:APM All issues that need APM UI Team support label Aug 9, 2021

yctercero added the release_note:skip Skip the PR/issue when compiling release notes label Aug 9, 2021

updating signals template version

008a886

ecezalp approved these changes Aug 9, 2021

View reviewed changes

...lib/detection_engine/rule_execution_log/rule_registry_log_client/rule_registry_log_client.ts Outdated Show resolved Hide resolved

x-pack/plugins/timelines/server/search_strategy/timeline/index.ts Outdated Show resolved Hide resolved

michaelolo24 approved these changes Aug 9, 2021

View reviewed changes

update fields for rule type id across the board

e4a767a

yctercero requested a review from a team as a code owner August 9, 2021 16:38

types and tests not yet passing, but wanted to see what CI throws

0ef93cc

gmmorris approved these changes Aug 10, 2021

View reviewed changes

michaelolo24 mentioned this pull request Aug 10, 2021

[Security Solution][Bug] - Disable alert table RBAC until fields sorted #108034

Merged

updating tests

15b9719

yctercero commented Aug 10, 2021

View reviewed changes

yctercero added 2 commits August 10, 2021 14:45

cleanup

c7382f3

Merge branch 'master' of github.com:elastic/kibana into update_ss_fie…

37614f4

…ld_names

yctercero removed the request for review from a team August 10, 2021 21:54

yctercero added 2 commits August 10, 2021 18:29

tweaks to tests, please pass ci

fc44fcc

you know what they say, 150th try's the charm

e4b9d3c

yctercero enabled auto-merge (squash) August 11, 2021 05:39

dgieselaar approved these changes Aug 11, 2021

View reviewed changes

yctercero added the auto-backport Deprecated - use backport:version if exact versions are needed label Aug 11, 2021

yctercero added 2 commits August 11, 2021 00:57

updating snapshots, ci you win

4f1f4f9

Merge branch 'master' of github.com:yctercero/kibana into update_ss_f…

759decb

…ield_names

yctercero force-pushed the update_ss_field_names branch from cb03411 to 759decb Compare August 11, 2021 08:02

yctercero merged commit cec5d3f into elastic:master Aug 11, 2021

kibanamachine mentioned this pull request Aug 11, 2021

[7.x] [RAC] - Update field names (#107857) #108170

Merged

michaelolo24 mentioned this pull request Aug 11, 2021

[Security Solution][RAC] - 7.15 Checklist #108071

Closed

48 tasks

yctercero deleted the update_ss_field_names branch August 11, 2021 16:26

mgiota mentioned this pull request Oct 4, 2021

[RAC] Use alerting framework to check alert permissions in the UI #113726

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RAC] - Update field names #107857

[RAC] - Update field names #107857

yctercero commented Aug 8, 2021 •

edited

Loading

elasticmachine commented Aug 9, 2021

elasticmachine commented Aug 9, 2021

elasticmachine commented Aug 9, 2021

elasticmachine commented Aug 9, 2021

ecezalp left a comment

michaelolo24 left a comment

gmmorris left a comment

michaelolo24 commented Aug 10, 2021

yctercero Aug 10, 2021

yctercero Aug 10, 2021

dgieselaar left a comment

yctercero commented Aug 11, 2021

yctercero commented Aug 11, 2021

kibanamachine commented Aug 11, 2021

kibanamachine commented Aug 11, 2021

[RAC] - Update field names #107857

[RAC] - Update field names #107857

Conversation

yctercero commented Aug 8, 2021 • edited Loading

Summary

Fields used moving forward

To test

elasticmachine commented Aug 9, 2021

elasticmachine commented Aug 9, 2021

elasticmachine commented Aug 9, 2021

elasticmachine commented Aug 9, 2021

ecezalp left a comment

Choose a reason for hiding this comment

michaelolo24 left a comment

Choose a reason for hiding this comment

gmmorris left a comment

Choose a reason for hiding this comment

michaelolo24 commented Aug 10, 2021

yctercero Aug 10, 2021

Choose a reason for hiding this comment

yctercero Aug 10, 2021

Choose a reason for hiding this comment

dgieselaar left a comment

Choose a reason for hiding this comment

yctercero commented Aug 11, 2021

yctercero commented Aug 11, 2021

kibanamachine commented Aug 11, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

History

kibanamachine commented Aug 11, 2021

💚 Backport successful

yctercero commented Aug 8, 2021 •

edited

Loading