Skip to content

Commit

Permalink
[Event Log] Extend ECS event schema with fields needed for Detection …
Browse files Browse the repository at this point in the history 
…Engine (#95067)

**Related to:** #94143

## Summary

This PR adds new fields to the schema (`EventSchema`, `IEvent`):

- standard ECS fields: `error.*`, `event.*`, `log.level`, `log.logger`, `rule.*`
- custom field set `kibana.detection_engine`

We need these fields on the Detections side to implement detection rule execution log. See the related proposal (#94143) for more details.

Also, this PR bumps ECS used in Event Log from `1.6.0` to the current `1.8.0` version. They are 100% same in terms of fields used in Event Log, so no changes in the schema were caused by this version increment.
  • Loading branch information
@banderror
banderror authored Mar 29, 2021
1 parent a1748cb commit d16101f
Show file tree
Hide file tree
Showing 7 changed files with 519 additions and 128 deletions.
73 changes: 64 additions & 9 deletions x-pack/plugins/event_log/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,23 @@ actitivies.
## Overview

This plugin provides a persistent log of "events" that can be used by other
plugins to record their processing, for later acccess. Currently it's only
used by the alerts and actions plugins.
plugins to record their processing, for later accces. It is used by:

The "events" are ECS documents, with some custom properties for Kibana, and
alerting-specific properties within those Kibana properties. The number of
ECS fields is limited today, but can be extended fairly easily. We are being
conservative in adding new fields though, to help prevent indexing explosions.
- `alerting` and `actions` plugins
- [work in progress] `security_solution` (detection rules execution log)

The "events" are [ECS documents](https://www.elastic.co/guide/en/ecs/current/index.html)
containing both standard ECS fields and some custom fields for Kibana.

- Standard fields are those which are defined in the ECS specification.
Examples: `@timestamp`, `message`, `event.provider`. The number of ECS fields
supported in Event Log is limited today, but can be extended fairly easily.
We are being conservative in adding new fields though, to help prevent
indexing explosions.
- Custom fields are not part of the ECS spec. We defined a top-level `kibana`
field set where we have some Kibana-specific fields like `kibana.server_uuid`
and `kibana.saved_objects`. Plugins added a few custom fields as well,
for example `kibana.alerting` field set.

A client API is available for other plugins to:

Expand Down Expand Up @@ -47,16 +57,25 @@ The structure of the event documents can be seen in the
generated via a script when the structure changes. See the
[README.md](generated/README.md) for how to change the document structure.

Below is an document in the expected structure, with descriptions of the fields:
Below is a document in the expected structure, with descriptions of the fields:

```js
{
// Base ECS fields.
// https://www.elastic.co/guide/en/ecs/current/ecs-base.html
"@timestamp": "ISO date",
tags: ["tags", "here"],
message: "message for humans here",

// ECS version. This is set by the Event Log and should not be specified
// by a client of Event Log.
// https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html
ecs: {
version: "version of ECS used by the event log",
},

// Event fields. All of them are supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-event.html
event: {
provider: "see below",
action: "see below",
Expand All @@ -65,19 +84,44 @@ Below is an document in the expected structure, with descriptions of the fields:
end: "ISO date of end time for events that capture a duration",
outcome: "success | failure, for events that indicate an outcome",
reason: "additional detail on failure outcome",
// etc
},

// Error fields. All of them are supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-error.html
error: {
message: "an error message, usually associated with outcome: failure",
// etc
},

// Log fields. Only a subset is supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-log.html
log: {
level: "info | warning | any log level keyword you need",
logger: "name of the logger",
},

// Rule fields. All of them are supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-rule.html
rule: {
author: ["Elastic"],
id: "a823fd56-5467-4727-acb1-66809737d943",
// etc
},

// User fields. Only user.name is supported.
// https://www.elastic.co/guide/en/ecs/current/ecs-user.html
user: {
name: "name of Kibana user",
},
kibana: { // custom ECS field

// Custom fields that are not part of ECS.
kibana: {
server_uuid: "UUID of kibana server, for diagnosing multi-Kibana scenarios",
alerting: {
instance_id: "alert instance id, for relevant documents",
action_group_id: "alert action group, for relevant documents",
action_subgroup_id: "alert action subgroup, for relevant documents",
action_subgroup: "alert action subgroup, for relevant documents",
status: "overall alert status, after alert execution",
},
saved_objects: [
Expand Down Expand Up @@ -363,3 +407,14 @@ yarn test:jest x-pack/plugins/event_log --watch

See: [`x-pack/test/plugin_api_integration/test_suites/event_log`](https://github.com/elastic/kibana/tree/master/x-pack/test/plugin_api_integration/test_suites/event_log).

To develop integration tests, first start the test server from the root of the repo:

```sh
node scripts/functional_tests_server --config x-pack/test/plugin_api_integration/config.ts
```

Then start the test runner:

```sh
node scripts/functional_test_runner --config x-pack/test/plugin_api_integration/config.ts --include x-pack/test/plugin_api_integration/test_suites/event_log/index.ts
```
29 changes: 22 additions & 7 deletions x-pack/plugins/event_log/generated/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,26 @@
# Generating event schema

The files in this directory were generated by manually running the script
../scripts/create-schemas.js from the root directory of the repository.
`../scripts/create-schemas.js` from the root directory of the repository.

These files should not be edited by hand.
**These files should not be edited by hand.**

Please follow the following steps:
1. clone the [ECS](https://github.com/elastic/ecs) repo locally so that it resides along side your kibana repo, and checkout the ECS version you wish to support (for example, the `1.6` branch, for version 1.6)
2. In the `x-pack/plugins/event_log/scripts/mappings.js` file you'll want to make th efollowing changes:
1. Update `EcsKibanaExtensionsMappings` to include the mapping of the fields you wish to add.
2. Update `EcsEventLogProperties` to include the fields in the generated mappings.json.
3. cd to the `kibana` root folder and run: `node ./x-pack/plugins/event_log/scripts/create_schemas.js`

1. Clone the [ECS](https://github.com/elastic/ecs) repo locally so that it
resides along side your kibana repo, and checkout the ECS version you wish to
support (for example, the `1.8` branch, for version 1.8).

2. In the `x-pack/plugins/event_log/scripts/mappings.js` file you'll want to
make the following changes:
- Update `EcsCustomPropertyMappings` to include the mapping of the custom
fields you wish to add.
- Update `EcsPropertiesToGenerate` to include the fields in the generated
`mappings.json`.
- Make sure to list all array fields in `EcsEventLogMultiValuedProperties`.

3. Cd to the `kibana` root folder and run:

```sh
node ./x-pack/plugins/event_log/scripts/create_schemas.js
```
181 changes: 169 additions & 12 deletions x-pack/plugins/event_log/generated/mappings.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@
"@timestamp": {
"type": "date"
},
"message": {
"norms": false,
"type": "text"
},
"tags": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"message": {
"norms": false,
"type": "text"
},
"ecs": {
"properties": {
"version": {
Expand All @@ -23,40 +23,197 @@
}
}
},
"error": {
"properties": {
"code": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"message": {
"norms": false,
"type": "text"
},
"stack_trace": {
"doc_values": false,
"fields": {
"text": {
"norms": false,
"type": "text"
}
},
"ignore_above": 1024,
"index": false,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"event": {
"properties": {
"action": {
"ignore_above": 1024,
"type": "keyword"
},
"provider": {
"category": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"code": {
"ignore_above": 1024,
"type": "keyword"
},
"start": {
"created": {
"type": "date"
},
"dataset": {
"ignore_above": 1024,
"type": "keyword"
},
"duration": {
"type": "long"
},
"end": {
"type": "date"
},
"hash": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"ingested": {
"type": "date"
},
"kind": {
"ignore_above": 1024,
"type": "keyword"
},
"module": {
"ignore_above": 1024,
"type": "keyword"
},
"original": {
"doc_values": false,
"ignore_above": 1024,
"index": false,
"type": "keyword"
},
"outcome": {
"ignore_above": 1024,
"type": "keyword"
},
"provider": {
"ignore_above": 1024,
"type": "keyword"
},
"reason": {
"ignore_above": 1024,
"type": "keyword"
},
"reference": {
"ignore_above": 1024,
"type": "keyword"
},
"risk_score": {
"type": "float"
},
"risk_score_norm": {
"type": "float"
},
"sequence": {
"type": "long"
},
"severity": {
"type": "long"
},
"start": {
"type": "date"
},
"timezone": {
"ignore_above": 1024,
"type": "keyword"
},
"type": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"url": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"error": {
"log": {
"properties": {
"message": {
"norms": false,
"type": "text"
"level": {
"ignore_above": 1024,
"type": "keyword"
},
"logger": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
"rule": {
"properties": {
"author": {
"ignore_above": 1024,
"type": "keyword",
"meta": {
"isArray": "true"
}
},
"category": {
"ignore_above": 1024,
"type": "keyword"
},
"description": {
"ignore_above": 1024,
"type": "keyword"
},
"id": {
"ignore_above": 1024,
"type": "keyword"
},
"license": {
"ignore_above": 1024,
"type": "keyword"
},
"name": {
"ignore_above": 1024,
"type": "keyword"
},
"reference": {
"ignore_above": 1024,
"type": "keyword"
},
"ruleset": {
"ignore_above": 1024,
"type": "keyword"
},
"uuid": {
"ignore_above": 1024,
"type": "keyword"
},
"version": {
"ignore_above": 1024,
"type": "keyword"
}
}
},
Expand Down Expand Up @@ -101,6 +258,7 @@
}
},
"saved_objects": {
"type": "nested",
"properties": {
"rel": {
"type": "keyword",
Expand All @@ -118,8 +276,7 @@
"type": "keyword",
"ignore_above": 1024
}
},
"type": "nested"
}
}
}
}
Expand Down
Loading

0 comments on commit d16101f

Please sign in to comment.