Teach elasticsearch/audit fileset to handle timestamps correctly
#15942
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ycombinator
added
bug
module
review
Filebeat
|
Pinging @elastic/stack-monitoring (Stack monitoring) |
ycombinator
added
Team:Integrations
ycombinator
commented
Jan 29, 2020
| @@ -6,5 +6,4 @@ paths: | |||
| exclude_files: [".gz$"] | |||
|
|
|||
| processors: | |||
| # Locale for timezone is only needed in non-json logs | |||
| - add_locale.when.not.regexp.message: "^{" | |||
| - add_locale: ~ | |||
There was a problem hiding this comment.
We add this processor for JSON logs as well because some versions of ES (e.g. 7.1.1) generate JSON audit logs with the @timestamp field which does not contain a time zone.
Later versions of ES will generate JSON audit logs with a timestamp field which does contain the time zone. In such cases, we will remove the event.timezone field in the ingest pipeline to avoid confusion about which timezone is correct.
Related: #13918.
mtojek
approved these changes
Jan 30, 2020
jsoriano
approved these changes
Jan 30, 2020
ycombinator
removed
the
needs_backport
|
Thank you @ycombinator to follow up on this |
ycombinator
added a commit
to ycombinator/beats
that referenced
this pull request
Feb 5, 2020
…lastic#15942) * Add samples from ES 7.1.1 audit log * When JSON logs contain @timestamp and TZ is detected, use detected TZ * Add CHANGELOG entry * Remove redundant guard * Revert unrelated change * s/teach/improve/ in changelog entry
ycombinator
added a commit
that referenced
this pull request
Feb 5, 2020
…15942) (#15963) * Add samples from ES 7.1.1 audit log * When JSON logs contain @timestamp and TZ is detected, use detected TZ * Add CHANGELOG entry * Remove redundant guard * Revert unrelated change * s/teach/improve/ in changelog entry
ycombinator
added a commit
to ycombinator/beats
that referenced
this pull request
Feb 7, 2020
…lastic#15942) * Add samples from ES 7.1.1 audit log * When JSON logs contain @timestamp and TZ is detected, use detected TZ * Add CHANGELOG entry * Remove redundant guard * Revert unrelated change * s/teach/improve/ in changelog entry
ycombinator
added a commit
that referenced
this pull request
Feb 8, 2020
…15942) (#15962) * Add samples from ES 7.1.1 audit log * When JSON logs contain @timestamp and TZ is detected, use detected TZ * Add CHANGELOG entry * Remove redundant guard * Revert unrelated change * s/teach/improve/ in changelog entry
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR teaches the
elasticsearch/auditfileset to correctly parse the subtle variations of timestamp fields that can be found in the JSON-formatted Elasticsearch audit logs across multiple versions of Elasticsearch.Why is it important?
Regardless of the version of Elasticsearch generating audit logs, the
elasticsearch/auditfileset will be able to parse them, specifically the timestamps within them, correctly.Checklist
- [ ] I have commented my code, particularly in hard-to-understand areas- [ ] I have made corresponding changes to the documentationRelated issues