[Event Log] Extend ECS event schema with fields needed for Detection Engine #95067

banderror · 2021-03-22T15:23:12Z

Related to: #94143

Summary

This PR adds new fields to the schema (EventSchema, IEvent):

standard ECS fields: error.*, event.*, log.level, log.logger, rule.*
custom field set kibana.detection_engine

We need these fields on the Detections side to implement detection rule execution log. See the related proposal (#94143) for more details.

Also, this PR bumps ECS used in Event Log from 1.6.0 to the current 1.8.0 version. They are 100% same in terms of fields used in Event Log, so no changes in the schema were caused by this version increment.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2021-03-22T15:23:14Z

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

pmuellr

This looks fine to me.

I'm not sure we actually have any tests that test all the fields, but I think at this point, we should add one. We end up testing the alert-specific fields in other alerting tests, so we'll know if something went bad with specific fields (hopefully). But, in general, I don't think we actually test that we can write and then read all the fields back. Seems like it's time to do that - we should have done it before I think. That way we'll know if we end up screwing up some of the fields during similar "updates".

I don't think there's any value in doing this as a unit test, we might as well get an e2e function test for this, that way we'll know everything including mappings is working the way we want. Those tests are in here: x-pack/test/plugin_api_integration . I think we'd just add a test to write an event with all the existing fields set to some value, and then read it back, make sure all the fields are the same (hopefully just an 'equal` test with the two objects).

The README.md should also be updated - it's currently our only doc for event log.

Presumably at some point there will be some additional provider and action values, so those would need to be doc'd as well - or it's not clear if we would enhance the way we write the events from alerting to somehow add the new fields for SIEM alerts.

Note that it is kinda weird to have this alerting stuff in the event log doc, but seemed like the best place to put it at the time. At some point, alerting-specific stuff should probably point into the alerting doc instead. Then same with any SIEM-specific stuff ...

banderror · 2021-03-23T11:48:37Z

Ok cool, thanks for your review @pmuellr 👍

Adding a smoke test as you described makes sense to me. I'll add one.

I can also update the README. I can see there's the "Event Documents" section where the event structure is described. I would personally avoid describing all the fields since this is already done in the ECS documentation. I would reduce the amount of info in the README and add the corresponding links to ECS docs. Is there anything else you think needs to be updated in this PR?

Presumably at some point there will be some additional provider and action values, so those would need to be doc'd as well - or it's not clear if we would enhance the way we write the events from alerting to somehow add the new fields for SIEM alerts.

Not sure I fully got this part. When we create a new provider and new action names for the detection rule execution log, they will likely be defined and documented on the Detections side.

At some point, alerting-specific stuff should probably point into the alerting doc instead. Then same with any SIEM-specific stuff

Imho event log should not dictate and document which provider and action names are used by its clients. Personally, to me it's a bit confusing to see any client-specific things in the event_log plugin (including the kibana.alerting etc parts in the schema). I'd suggest to eventually move these specifics (both code and docs) to clients.

pmuellr · 2021-03-25T19:13:07Z

The comments re: the readme being alerting specific, and describing ECS fields are appropriate, but the reason we documented them here is to make it easier for customers using the event log to diagnose alerting problems is to have a single place to look at for all the info, for right now. Otherwise everything would be spread across alerting, event log and ECS with no single place describing everything.

We'll need to split it eventually ...

pmuellr

LGTM, thanks a bunch Georgii!!

pmuellr · 2021-03-25T19:17:01Z

x-pack/plugins/event_log/README.md


 ```js
 {
+  // Base ECS fields.
+  // https://www.elastic.co/guide/en/ecs/current/ecs-base.html


nice, thx for the links!

pmuellr · 2021-03-25T19:18:07Z

x-pack/plugins/event_log/README.md

    server_uuid: "UUID of kibana server, for diagnosing multi-Kibana scenarios",
    alerting: {
      instance_id: "alert instance id, for relevant documents",
      action_group_id: "alert action group, for relevant documents",
-      action_subgroup_id: "alert action subgroup, for relevant documents",
+      action_subgroup: "alert action subgroup, for relevant documents",


pmuellr · 2021-03-25T19:18:29Z

x-pack/plugins/event_log/README.md

@@ -363,3 +411,14 @@ yarn test:jest x-pack/plugins/event_log --watch

 See: [`x-pack/test/plugin_api_integration/test_suites/event_log`](https://github.com/elastic/kibana/tree/master/x-pack/test/plugin_api_integration/test_suites/event_log).

+To develop integration tests, first start the test server from the root of the repo:


spong · 2021-03-25T22:53:57Z

x-pack/plugins/event_log/README.md

+  // https://www.elastic.co/guide/en/ecs/current/ecs-rule.html
+  rule: {
+    author: ["Elastic"],
+    id: "a823fd56-5467-4727-acb1-66809737d943",


This reminded me of recent id vs rule_id conversations -- there's not a dedicated spot for us to put rule_id (the static id of a rule between clusters), but we can just continue to reference it in the message ala:

[+] Starting Signal Rule execution name: "Persistence via TelemetryController Scheduled Task Hijack" id: "6c25a284-808a-11eb-87f7-018e452bbfc9" rule id: "68921d85-d0dc-48b3-865f-43291ca2c4f2" signals index: ".siem-signals-spong-default"

@spong Thank you for pointing to this problem 👍

ECS has a standard rule.uuid field, I'm not sure about its meaning though, the docs are not super clear to me. But even if neither rule.id nor rule.uuid are suitable for storing a global static rule id, we could do it with a custom field like kibana.detection_engine.rule.guid. And then submit a PR to ECS with rule.guid field.

I'll need to figure this out - which fields to use for rule ids :)

spong

Updated docs, e2e tests, and new fields look good too -- you've got it all! Thank you @banderror! 🙂 LGTM! 👍

kibanamachine · 2021-03-29T12:56:31Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #115185 succeeded c0176cfc8a941449efdde49e3224de3a8698e851
💚 Build #114895 succeeded bbf99eca3522e940b2f307096580dc291a1a93ad
💚 Build #114582 succeeded 86bd8d2ee9de4332ccc1c7c978c67d2a95d3f8ee
💚 Build #114348 succeeded ad3832c7e4851649d7b9a499e3961c5d58138220

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @banderror

…Engine (elastic#95067) **Related to:** elastic#94143 ## Summary This PR adds new fields to the schema (`EventSchema`, `IEvent`): - standard ECS fields: `error.*`, `event.*`, `log.level`, `log.logger`, `rule.*` - custom field set `kibana.detection_engine` We need these fields on the Detections side to implement detection rule execution log. See the related proposal (elastic#94143) for more details. Also, this PR bumps ECS used in Event Log from `1.6.0` to the current `1.8.0` version. They are 100% same in terms of fields used in Event Log, so no changes in the schema were caused by this version increment.

kibanamachine · 2021-03-29T13:02:27Z

💚 Backport successful

✅ 7.x / #95654

This backport PR will be merged automatically after passing CI.

…Engine (#95067) (#95654) **Related to:** #94143 ## Summary This PR adds new fields to the schema (`EventSchema`, `IEvent`): - standard ECS fields: `error.*`, `event.*`, `log.level`, `log.logger`, `rule.*` - custom field set `kibana.detection_engine` We need these fields on the Detections side to implement detection rule execution log. See the related proposal (#94143) for more details. Also, this PR bumps ECS used in Event Log from `1.6.0` to the current `1.8.0` version. They are 100% same in terms of fields used in Event Log, so no changes in the schema were caused by this version increment. Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>

banderror added release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Feature:EventLog labels Mar 22, 2021

banderror requested a review from a team as a code owner March 22, 2021 15:23

banderror self-assigned this Mar 22, 2021

banderror requested a review from pmuellr March 22, 2021 15:23

banderror added v7.13.0 v8.0.0 auto-backport Deprecated - use backport:version if exact versions are needed labels Mar 22, 2021

pmuellr requested changes Mar 22, 2021

View reviewed changes

banderror force-pushed the extend-ecs-schema-of-event-log branch from ad3832c to 86bd8d2 Compare March 23, 2021 14:28

banderror marked this pull request as draft March 23, 2021 14:29

banderror force-pushed the extend-ecs-schema-of-event-log branch 2 times, most recently from bbf99ec to c0176cf Compare March 25, 2021 13:45

banderror marked this pull request as ready for review March 25, 2021 13:45

banderror requested a review from pmuellr March 25, 2021 13:46

pmuellr mentioned this pull request Mar 25, 2021

[event log][discuss] should we stop using common top-level ecs fields for our own data? #95421

Closed

pmuellr approved these changes Mar 25, 2021

View reviewed changes

spong reviewed Mar 25, 2021

View reviewed changes

spong approved these changes Mar 25, 2021

View reviewed changes

banderror added 7 commits March 29, 2021 12:29

Generate new mappings needed for Detection Engine

d539cfd

Bump ECS to the current 1.8.0 version (no changes in the schema)

354cdda

Slightly improve schema generation scripts

221d660

Update documentation

5f94e8b

Add a new integration test

3841ebc

Refactor changed integration tests

62a3281

Update documentation with info about developing integration tests

e848b32

banderror force-pushed the extend-ecs-schema-of-event-log branch from c0176cf to a01b8fb Compare March 29, 2021 10:29

Remove custom detection_engine fields from generated schema

8eb6bae

banderror force-pushed the extend-ecs-schema-of-event-log branch from a01b8fb to 8eb6bae Compare March 29, 2021 10:50

banderror merged commit d16101f into elastic:master Mar 29, 2021

banderror deleted the extend-ecs-schema-of-event-log branch March 29, 2021 13:00

kibanamachine mentioned this pull request Mar 29, 2021

[7.x] [Event Log] Extend ECS event schema with fields needed for Detection Engine (#95067) #95654

Merged

pmuellr mentioned this pull request May 4, 2021

[RAC] Rule monitoring: Event Log for Rule Registry #98353

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Event Log] Extend ECS event schema with fields needed for Detection Engine #95067

[Event Log] Extend ECS event schema with fields needed for Detection Engine #95067

banderror commented Mar 22, 2021

elasticmachine commented Mar 22, 2021

pmuellr left a comment

banderror commented Mar 23, 2021

pmuellr commented Mar 25, 2021

pmuellr left a comment

pmuellr Mar 25, 2021

pmuellr Mar 25, 2021

pmuellr Mar 25, 2021

spong Mar 25, 2021

banderror Mar 29, 2021

spong left a comment

kibanamachine commented Mar 29, 2021

kibanamachine commented Mar 29, 2021

		@@ -363,3 +411,14 @@ yarn test:jest x-pack/plugins/event_log --watch

		See: [`x-pack/test/plugin_api_integration/test_suites/event_log`](https://github.com/elastic/kibana/tree/master/x-pack/test/plugin_api_integration/test_suites/event_log).

		To develop integration tests, first start the test server from the root of the repo:

[Event Log] Extend ECS event schema with fields needed for Detection Engine #95067

[Event Log] Extend ECS event schema with fields needed for Detection Engine #95067

Conversation

banderror commented Mar 22, 2021

Summary

Checklist

elasticmachine commented Mar 22, 2021

pmuellr left a comment

Choose a reason for hiding this comment

banderror commented Mar 23, 2021

pmuellr commented Mar 25, 2021

pmuellr left a comment

Choose a reason for hiding this comment

pmuellr Mar 25, 2021

Choose a reason for hiding this comment

pmuellr Mar 25, 2021

Choose a reason for hiding this comment

pmuellr Mar 25, 2021

Choose a reason for hiding this comment

spong Mar 25, 2021

Choose a reason for hiding this comment

banderror Mar 29, 2021

Choose a reason for hiding this comment

spong left a comment

Choose a reason for hiding this comment

kibanamachine commented Mar 29, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Mar 29, 2021

💚 Backport successful