Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] Improve ECS categorization field mappings for mysql module #17491

Merged
merged 3 commits into from
Apr 6, 2020
Merged

[Filebeat] Improve ECS categorization field mappings for mysql module #17491

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0a116b7
convert mysql pipeline to yaml
leehinman Apr 2, 2020
4f2bcab
Improve ECS categorization field mappings for mysql module
leehinman Apr 3, 2020
34aea09
Merge branch 'master' into 16172_mysql_ecs_1.4
leehinman Apr 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Improve ECS categorization field mappings for mssql module. {issue}16171[16171] {pull}17376[17376]
- Added access_key_id, secret_access_key and session_token into aws module config. {pull}17456[17456]
- Add dashboard for Google Cloud Audit and AWS CloudTrail. {pull}17379[17379]
- Improve ECS categorization field mappings for mysql module. {issue}16172[16172] {pull}XXXXX[XXXXX]

*Heartbeat*

Expand Down
65 changes: 0 additions & 65 deletions filebeat/module/mysql/error/ingest/pipeline.json
View file
Open in desktop

This file was deleted.

70 changes: 70 additions & 0 deletions filebeat/module/mysql/error/ingest/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
description: Pipeline for parsing MySQL error logs
processors:
- grok:
field: message
patterns:
- '%{MYSQLDATETIME}%{SPACE}(%{NUMBER:mysql.thread_id:long}%{SPACE})?(\[%{DATA:log.level}\]%{SPACE})?%{GREEDYMULTILINE:message}'
- '%{GREEDYDATA:message}'
ignore_missing: true
pattern_definitions:
LOCALDATETIME: (?:%{YEAR}-%{MONTHNUM}-%{MONTHDAY}|%{NUMBER})%{SPACE}%{TIME}
MYSQLDATETIME: (?:%{LOCALDATETIME:_tmp.local_timestamp}|%{TIMESTAMP_ISO8601:_tmp.timestamp})
GREEDYMULTILINE: |-
(.|
)+
- grok:
field: message
patterns:
- '(\[%{DATA:event.code}\])%{SPACE}(\[%{DATA:event.provider}\])%{SPACE}%{GREEDYMULTILINE}'
- '%{GREEDYDATA}'
ignore_missing: true
ignore_failure: true
pattern_definitions:
GREEDYMULTILINE: |-
(.|
)+
- rename:
field: '@timestamp'
target_field: event.created
- date:
if: ctx._tmp?.local_timestamp != null && ctx.event?.timezone == null
field: _tmp.local_timestamp
formats:
- yyMMdd H:m:s
- yyMMdd H:m:s
- yyyy-MM-dd H:m:s
- yyyy-MM-dd H:m:s
- date:
if: ctx._tmp?.local_timestamp != null && ctx.event?.timezone != null
field: _tmp.local_timestamp
timezone: '{{ event.timezone }}'
formats:
- yyMMdd H:m:s
- yyMMdd H:m:s
- yyyy-MM-dd H:m:s
- yyyy-MM-dd H:m:s
- date:
if: ctx._tmp?.timestamp != null
field: _tmp.timestamp
formats:
- ISO8601
- remove:
field: _tmp
ignore_missing: true
- set:
field: event.kind
value: event
- append:
field: event.category
value: database
- append:
field: event.type
value: info
- append:
field: event.type
value: error
if: "ctx?.log?.level != null && ctx.log.level.toLowerCase() == 'error'"
on_failure:
- set:
field: error.message
value: '{{ _ingest.on_failure_message }}'
2 changes: 1 addition & 1 deletion filebeat/module/mysql/error/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@ var:
os.windows:
- "c:/programdata/MySQL/MySQL Server*/error.log*"

ingest_pipeline: ingest/pipeline.json
ingest_pipeline: ingest/pipeline.yml
input: config/error.yml
77 changes: 77 additions & 0 deletions filebeat/module/mysql/error/test/error.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,16 @@
[
{
"@timestamp": "2016-12-09T13:08:33.000-02:00",
"event.category": [
"database"
],
"event.dataset": "mysql.error",
"event.kind": "event",
"event.module": "mysql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "error",
"input.type": "log",
"log.offset": 0,
Expand All @@ -12,9 +19,16 @@
},
{
"@timestamp": "2016-12-09T12:08:33.335Z",
"event.category": [
"database"
],
"event.dataset": "mysql.error",
"event.kind": "event",
"event.module": "mysql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "error",
"input.type": "log",
"log.level": "Warning",
Expand All @@ -25,9 +39,16 @@
},
{
"@timestamp": "2016-12-09T12:08:33.335Z",
"event.category": [
"database"
],
"event.dataset": "mysql.error",
"event.kind": "event",
"event.module": "mysql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "error",
"input.type": "log",
"log.level": "Warning",
Expand All @@ -38,9 +59,16 @@
},
{
"@timestamp": "2016-12-09T12:08:33.336Z",
"event.category": [
"database"
],
"event.dataset": "mysql.error",
"event.kind": "event",
"event.module": "mysql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "error",
"input.type": "log",
"log.level": "Note",
Expand All @@ -51,9 +79,16 @@
},
{
"@timestamp": "2016-12-09T12:08:33.345Z",
"event.category": [
"database"
],
"event.dataset": "mysql.error",
"event.kind": "event",
"event.module": "mysql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "error",
"input.type": "log",
"log.level": "Warning",
Expand All @@ -64,9 +99,16 @@
},
{
"@timestamp": "2016-12-09T12:08:33.351Z",
"event.category": [
"database"
],
"event.dataset": "mysql.error",
"event.kind": "event",
"event.module": "mysql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "error",
"input.type": "log",
"log.level": "Note",
Expand All @@ -77,9 +119,16 @@
},
{
"@timestamp": "2016-12-09T12:08:33.784Z",
"event.category": [
"database"
],
"event.dataset": "mysql.error",
"event.kind": "event",
"event.module": "mysql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "error",
"input.type": "log",
"log.flags": [
Expand All @@ -93,9 +142,16 @@
},
{
"@timestamp": "2016-12-09T22:21:02.443Z",
"event.category": [
"database"
],
"event.dataset": "mysql.error",
"event.kind": "event",
"event.module": "mysql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "error",
"input.type": "log",
"log.level": "Note",
Expand All @@ -106,9 +162,16 @@
},
{
"@timestamp": "2016-12-09T14:18:50.000-02:00",
"event.category": [
"database"
],
"event.dataset": "mysql.error",
"event.kind": "event",
"event.module": "mysql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "error",
"input.type": "log",
"log.level": "Warning",
Expand All @@ -118,9 +181,16 @@
},
{
"@timestamp": "2016-12-09T14:18:50.000-02:00",
"event.category": [
"database"
],
"event.dataset": "mysql.error",
"event.kind": "event",
"event.module": "mysql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "error",
"input.type": "log",
"log.level": "Note",
Expand All @@ -130,9 +200,16 @@
},
{
"@timestamp": "2016-12-09T14:18:50.000-02:00",
"event.category": [
"database"
],
"event.dataset": "mysql.error",
"event.kind": "event",
"event.module": "mysql",
"event.timezone": "-02:00",
"event.type": [
"info"
],
"fileset.name": "error",
"input.type": "log",
"log.offset": 1422,
Expand Down
Loading