Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add GC log level for Elasticsearch #34054

Closed
danielmitterdorfer opened this issue Dec 15, 2022 · 1 comment · Fixed by #34159
Closed

Add GC log level for Elasticsearch #34054

danielmitterdorfer opened this issue Dec 15, 2022 · 1 comment · Fixed by #34159
Labels
enhancement Module:elasticsearch Elasticsearch Beats modules needs_team Indicates that the issue/PR needs a Team:* label v8.7.0

Comments

@danielmitterdorfer
Copy link
Member

Elasticsearch 8.7.0 will change the JVM log output format to include a level parameter (see elastic/elasticsearch#92382); the available log levels are described in the java man page. As there is an existing ingest pipeline to ingest GC logs, it needs to be adapted as well (only the JVM9HEADER will need to change).

Here are some example log lines:

Before this change:

[2022-12-15T06:19:16.936+0000][15181][gc,start    ] GC(0) Pause Young (Normal) (G1 Evacuation Pause)

After this change:

[2022-12-15T06:22:27.932+0000][16320][info][gc,start    ] GC(0) Pause Young (Normal) (G1 Evacuation Pause)
[2022-12-15T06:22:27.932+0000][16320][debug][gc,age      ] GC(0) Desired survivor size 14680064 bytes, new threshold 15 (max threshold 15)
[2022-12-15T06:22:27.935+0000][16320][info ][gc,phases   ] GC(0)   Pre Evacuate Collection Set: 0.1ms
[2022-12-15T06:22:27.935+0000][16320][trace][gc,age      ] GC(0) Age table with threshold 15 (max threshold 15)

Notice how the log level might have trailing spaces (third line in the example).

I'm not familiar enough with grok but I believe the grok pattern would need to change to something like:

 JVM9HEADER: \[%{TIMESTAMP_ISO8601:timestamp}\]\[%{DATA:elasticsearch.gc.level}%{SPACE}\]\[%{POSINT:process.pid}\]\[%{DATA:elasticsearch.gc.tags}%{SPACE}\]

where elasticsearch.gc.level needs to be optional to support also pre 8.7.0 clusters. It should also end up in log.level when present.
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 15, 2022
@botelastic
Copy link

botelastic bot commented Dec 15, 2022

This issue doesn't have a Team:<team> label.

@leehinman leehinman mentioned this issue Dec 15, 2022
Open
danielmitterdorfer added a commit to danielmitterdorfer/beats that referenced this issue Jan 3, 2023
@danielmitterdorfer
Recognize log level in Elasticsearch JVM logs
c3ecb6a 
Elasticsearch will add a log level to its JVM logs to allow users to
help them detect errors / warnings more easily. With this commit we
detect this new field if present and continue to recognize the prior log
format without a log level.

Relates elastic/elasticsearch#92382
Closes elastic#34054
@danielmitterdorfer danielmitterdorfer mentioned this issue Jan 3, 2023
Merged
6 tasks
leehinman pushed a commit that referenced this issue Jan 9, 2023
@danielmitterdorfer
Recognize log level in Elasticsearch JVM logs (#34159)
27b8103 
* Recognize log level in Elasticsearch JVM logs

Elasticsearch will add a log level to its JVM logs to allow users to
help them detect errors / warnings more easily. With this commit we
detect this new field if present and continue to recognize the prior log
format without a log level.

Relates elastic/elasticsearch#92382
Closes #34054
chrisberkhout pushed a commit that referenced this issue Jun 1, 2023
@danielmitterdorfer @chrisberkhout
Recognize log level in Elasticsearch JVM logs (#34159)
6bfa9d4 
* Recognize log level in Elasticsearch JVM logs

Elasticsearch will add a log level to its JVM logs to allow users to
help them detect errors / warnings more easily. With this commit we
detect this new field if present and continue to recognize the prior log
format without a log level.

Relates elastic/elasticsearch#92382
Closes #34054
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Module:elasticsearch Elasticsearch Beats modules needs_team Indicates that the issue/PR needs a Team:* label v8.7.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Recognize log level in Elasticsearch JVM logs danielmitterdorfer/beats
1 participant
@danielmitterdorfer