Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove event.timezone from events from some json logs #13918

Merged
merged 3 commits into from
Oct 4, 2019
Merged

Remove event.timezone from events from some json logs #13918

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
797b70f
Remove event.timezone from events from some json logs
jsoriano Oct 4, 2019
e720a0e
Fix changelog
jsoriano Oct 4, 2019
a12be01
Merge remote-tracking branch 'origin/master' into remove-event-timezo…
jsoriano Oct 4, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add support to set the document id in the json reader. {pull}5844[5844]
- Add input httpjson. {issue}13545[13545] {pull}13546[13546]
- Filebeat Netflow input: Remove beta label. {pull}13858[13858]
- Remove `event.timezone` from events that don't need it in some modules that support log formats with and without timezones. {pull}[]

*Heartbeat*
- Add non-privileged icmp on linux and darwin(mac). {pull}13795[13795] {issue}11498[11498]
Expand Down
3 changes: 2 additions & 1 deletion filebeat/module/elasticsearch/audit/config/audit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@ paths:
exclude_files: [".gz$"]

processors:
- add_locale: ~
# Locale for timezone is only needed in non-json logs
- add_locale.when.not.regexp.message: "^{"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we discussed (on Zoom) about leaving this as-is but modifying the JSON processing ingest pipelines to remove event.timezone? That way we wouldn't be duplicating the regexp check in multiple places.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, but while doing the change I thought that we would be removing event.timezone even if a user adds it for any other reason, what would be counter-intuitive and probably hard to debug (though probably this is a quite corner case).

If we avoid executing add_locale when not needed as is done here, we still keep the possibility of using add_locale or manually adding this field if a user wants it.
Even if we have the regexp in two places, any change there will make tests with example logs fail, and I don't expect to change these patterns so much.

I'd slightly prefer to don't run add_locale if not needed, but I don't have a strong opinion, both options look fine to me, so as you prefer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I prefer the option of not having to unnecessarily run this processor if we don't need to either. I just wish there was a way not to duplicate the same check in multiple places, but I don't see a way around that given that some processing happens on the beats side and some on the ES side.

Let's go with what you have here.

10 changes: 0 additions & 10 deletions filebeat/module/elasticsearch/audit/test/test-audit-730.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 0,
Expand Down Expand Up @@ -48,7 +47,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 423,
Expand Down Expand Up @@ -78,7 +76,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 846,
Expand Down Expand Up @@ -107,7 +104,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 1269,
Expand Down Expand Up @@ -136,7 +132,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 1706,
Expand All @@ -162,7 +157,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 2170,
Expand All @@ -188,7 +182,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 2576,
Expand Down Expand Up @@ -217,7 +210,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 2984,
Expand Down Expand Up @@ -246,7 +238,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 3402,
Expand All @@ -272,7 +263,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 3823,
Expand Down
2 changes: 0 additions & 2 deletions filebeat/module/elasticsearch/audit/test/test-audit-docker.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
"event.action": "anonymous_access_denied",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"http.request.method": "GET",
"input.type": "log",
Expand All @@ -29,7 +28,6 @@
"event.action": "authentication_failed",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"http.request.method": "GET",
"input.type": "log",
Expand Down
7 changes: 0 additions & 7 deletions filebeat/module/elasticsearch/audit/test/test-audit.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
"event.action": "authentication_failed",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 0,
Expand All @@ -27,7 +26,6 @@
"event.action": "authentication_failed",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 274,
Expand All @@ -53,7 +51,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 558,
Expand All @@ -78,7 +75,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 941,
Expand All @@ -103,7 +99,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 1309,
Expand Down Expand Up @@ -131,7 +126,6 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 1676,
Expand All @@ -153,7 +147,6 @@
"event.action": "authentication_success",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"http.request.body.content": "\n{\n \"query\" : {\n \"term\" : { \"user\" : \"kimchy\" }\n }\n}\n",
"http.request.method": "GET",
Expand Down
3 changes: 2 additions & 1 deletion filebeat/module/elasticsearch/deprecation/config/log.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,5 @@ multiline:
match: after

processors:
- add_locale: ~
# Locale for timezone is only needed in non-json logs
- add_locale.when.not.regexp.message: "^{"
13 changes: 0 additions & 13 deletions filebeat/module/elasticsearch/deprecation/test/test-json.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -25,7 +24,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -42,7 +40,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -59,7 +56,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -76,7 +72,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -93,7 +88,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -110,7 +104,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -127,7 +120,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -144,7 +136,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -161,7 +152,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -178,7 +168,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -195,7 +184,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand All @@ -212,7 +200,6 @@
"elasticsearch.node.name": "es1_1",
"event.dataset": "elasticsearch.deprecation",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "deprecation",
"input.type": "log",
"log.level": "WARN",
Expand Down
4 changes: 2 additions & 2 deletions filebeat/module/elasticsearch/server/config/log.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@ multiline:
match: after

processors:
- add_locale: ~

# Locale for timezone is only needed in non-json logs
- add_locale.when.not.regexp.message: "^{"
Loading