Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Event Log] Extend ECS event schema with fields needed for Detection Engine #95067

Merged
merged 8 commits into from
Mar 29, 2021
Merged

[Event Log] Extend ECS event schema with fields needed for Detection Engine #95067

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d539cfd
Generate new mappings needed for Detection Engine
banderror Mar 22, 2021
354cdda
Bump ECS to the current 1.8.0 version (no changes in the schema)
banderror Mar 22, 2021
221d660
Slightly improve schema generation scripts
banderror Mar 22, 2021
5f94e8b
Update documentation
banderror Mar 23, 2021
3841ebc
Add a new integration test
banderror Mar 25, 2021
62a3281
Refactor changed integration tests
banderror Mar 25, 2021
e848b32
Update documentation with info about developing integration tests
banderror Mar 25, 2021
8eb6bae
Remove custom detection_engine fields from generated schema
banderror Mar 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 64 additions & 9 deletions x-pack/plugins/event_log/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,23 @@ actitivies.
## Overview

This plugin provides a persistent log of "events" that can be used by other
plugins to record their processing, for later acccess. Currently it's only
used by the alerts and actions plugins.
plugins to record their processing, for later accces. It is used by:

The "events" are ECS documents, with some custom properties for Kibana, and
alerting-specific properties within those Kibana properties. The number of
ECS fields is limited today, but can be extended fairly easily. We are being
conservative in adding new fields though, to help prevent indexing explosions.
- `alerting` and `actions` plugins
- [work in progress] `security_solution` (detection rules execution log)

The "events" are [ECS documents](https://www.elastic.co/guide/en/ecs/current/index.html)
containing both standard ECS fields and some custom fields for Kibana.

- Standard fields are those which are defined in the ECS specification.
Examples: `@timestamp`, `message`, `event.provider`. The number of ECS fields
supported in Event Log is limited today, but can be extended fairly easily.
We are being conservative in adding new fields though, to help prevent
indexing explosions.
- Custom fields are not part of the ECS spec. We defined a top-level `kibana`
field set where we have some Kibana-specific fields like `kibana.server_uuid`
and `kibana.saved_objects`. Plugins added a few custom fields as well,
for example `kibana.alerting` field set.

A client API is available for other plugins to:

Expand Down Expand Up @@ -47,16 +57,25 @@ The structure of the event documents can be seen in the
generated via a script when the structure changes. See the
[README.md](generated/README.md) for how to change the document structure.

Below is an document in the expected structure, with descriptions of the fields:
Below is a document in the expected structure, with descriptions of the fields:

```js
{
// Base ECS fields.
// https://www.elastic.co/guide/en/ecs/current/ecs-base.html
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, thx for the links!

"@timestamp": "ISO date",
tags: ["tags", "here"],
message: "message for humans here",

// ECS version. This is set by the Event Log and should not be specified
// by a client of Event Log.
// https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html
ecs: {
version: "version of ECS used by the event log",
},

// Event fields. All of them are supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-event.html
event: {
provider: "see below",
action: "see below",
Expand All @@ -65,19 +84,44 @@ Below is an document in the expected structure, with descriptions of the fields:
end: "ISO date of end time for events that capture a duration",
outcome: "success | failure, for events that indicate an outcome",
reason: "additional detail on failure outcome",
// etc
},

// Error fields. All of them are supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-error.html
error: {
message: "an error message, usually associated with outcome: failure",
// etc
},

// Log fields. Only a subset is supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-log.html
log: {
level: "info | warning | any log level keyword you need",
logger: "name of the logger",
},

// Rule fields. All of them are supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-rule.html
rule: {
author: ["Elastic"],
id: "a823fd56-5467-4727-acb1-66809737d943",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reminded me of recent id vs rule_id conversations -- there's not a dedicated spot for us to put rule_id (the static id of a rule between clusters), but we can just continue to reference it in the message ala:

[+] Starting Signal Rule execution name: "Persistence via TelemetryController Scheduled Task Hijack" id: "6c25a284-808a-11eb-87f7-018e452bbfc9" rule id: "68921d85-d0dc-48b3-865f-43291ca2c4f2" signals index: ".siem-signals-spong-default"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spong Thank you for pointing to this problem 👍

ECS has a standard rule.uuid field, I'm not sure about its meaning though, the docs are not super clear to me. But even if neither rule.id nor rule.uuid are suitable for storing a global static rule id, we could do it with a custom field like kibana.detection_engine.rule.guid. And then submit a PR to ECS with rule.guid field.

I'll need to figure this out - which fields to use for rule ids :)

// etc
},

// User fields. Only user.name is supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-user.html
user: {
name: "name of Kibana user",
},
kibana: { // custom ECS field

// Custom fields that are not part of ECS.
kibana: {
server_uuid: "UUID of kibana server, for diagnosing multi-Kibana scenarios",
alerting: {
instance_id: "alert instance id, for relevant documents",
action_group_id: "alert action group, for relevant documents",
action_subgroup_id: "alert action subgroup, for relevant documents",
action_subgroup: "alert action subgroup, for relevant documents",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

status: "overall alert status, after alert execution",
},
saved_objects: [
Expand Down Expand Up @@ -363,3 +407,14 @@ yarn test:jest x-pack/plugins/event_log --watch

See: [`x-pack/test/plugin_api_integration/test_suites/event_log`](https://github.com/elastic/kibana/tree/master/x-pack/test/plugin_api_integration/test_suites/event_log).

To develop integration tests, first start the test server from the root of the repo:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome!


```sh
node scripts/functional_tests_server --config x-pack/test/plugin_api_integration/config.ts
```

Then start the test runner:

```sh
node scripts/functional_test_runner --config x-pack/test/plugin_api_integration/config.ts --include x-pack/test/plugin_api_integration/test_suites/event_log/index.ts
```
29 changes: 22 additions & 7 deletions x-pack/plugins/event_log/generated/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,26 @@
# Generating event schema

The files in this directory were generated by manually running the script
../scripts/create-schemas.js from the root directory of the repository.
`../scripts/create-schemas.js` from the root directory of the repository.

These files should not be edited by hand.
**These files should not be edited by hand.**

Please follow the following steps:
1. clone the [ECS](https://github.com/elastic/ecs) repo locally so that it resides along side your kibana repo, and checkout the ECS version you wish to support (for example, the `1.6` branch, for version 1.6)
2. In the `x-pack/plugins/event_log/scripts/mappings.js` file you'll want to make th efollowing changes:
1. Update `EcsKibanaExtensionsMappings` to include the mapping of the fields you wish to add.
2. Update `EcsEventLogProperties` to include the fields in the generated mappings.json.
3. cd to the `kibana` root folder and run: `node ./x-pack/plugins/event_log/scripts/create_schemas.js`

1. Clone the [ECS](https://github.com/elastic/ecs) repo locally so that it
resides along side your kibana repo, and checkout the ECS version you wish to
support (for example, the `1.8` branch, for version 1.8).

2. In the `x-pack/plugins/event_log/scripts/mappings.js` file you'll want to
make the following changes:
- Update `EcsCustomPropertyMappings` to include the mapping of the custom
fields you wish to add.
- Update `EcsPropertiesToGenerate` to include the fields in the generated
`mappings.json`.
- Make sure to list all array fields in `EcsEventLogMultiValuedProperties`.

3. Cd to the `kibana` root folder and run:

```sh
node ./x-pack/plugins/event_log/scripts/create_schemas.js
```
181 changes: 169 additions & 12 deletions x-pack/plugins/event_log/generated/mappings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@
"@timestamp": {
"type": "date"
},
"message": {
"norms": false,
"type": "text"
},
"tags": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"message": {
"norms": false,
"type": "text"
},
"ecs": {
"properties": {
"version": {
Expand All @@ -23,40 +23,197 @@
}
}
},
"error": {
"properties": {
"code": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"message": {
"norms": false,
"type": "text"
},
"stack_trace": {
"doc_values": false,
"fields": {
"text": {
"norms": false,
"type": "text"
}
},
"ignore_above": 1024,
"index": false,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"event": {
"properties": {
"action": {
"ignore_above": 1024,
"type": "keyword"
},
"provider": {
"category": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"code": {
"ignore_above": 1024,
"type": "keyword"
},
"start": {
"created": {
"type": "date"
},
"dataset": {
"ignore_above": 1024,
"type": "keyword"
},
"duration": {
"type": "long"
},
"end": {
"type": "date"
},
"hash": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"ingested": {
"type": "date"
},
"kind": {
"ignore_above": 1024,
"type": "keyword"
},
"module": {
"ignore_above": 1024,
"type": "keyword"
},
"original": {
"doc_values": false,
"ignore_above": 1024,
"index": false,
"type": "keyword"
},
"outcome": {
"ignore_above": 1024,
"type": "keyword"
},
"provider": {
"ignore_above": 1024,
"type": "keyword"
},
"reason": {
"ignore_above": 1024,
"type": "keyword"
},
"reference": {
"ignore_above": 1024,
"type": "keyword"
},
"risk_score": {
"type": "float"
},
"risk_score_norm": {
"type": "float"
},
"sequence": {
"type": "long"
},
"severity": {
"type": "long"
},
"start": {
"type": "date"
},
"timezone": {
"ignore_above": 1024,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"url": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"error": {
"log": {
"properties": {
"message": {
"norms": false,
"type": "text"
"level": {
"ignore_above": 1024,
"type": "keyword"
},
"logger": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"rule": {
"properties": {
"author": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"category": {
"ignore_above": 1024,
"type": "keyword"
},
"description": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"license": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"reference": {
"ignore_above": 1024,
"type": "keyword"
},
"ruleset": {
"ignore_above": 1024,
"type": "keyword"
},
"uuid": {
"ignore_above": 1024,
"type": "keyword"
},
"version": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
Expand Down Expand Up @@ -101,6 +258,7 @@
}
},
"saved_objects": {
"type": "nested",
"properties": {
"rel": {
"type": "keyword",
Expand All @@ -118,8 +276,7 @@
"type": "keyword",
"ignore_above": 1024
}
},
"type": "nested"
}
}
}
}
Expand Down
Loading