[Logging] Support for ECS and custom event tags #46119

pgomulka · 2019-08-29T10:45:10Z

To support Elastic Common Schema, some of the fields would have to be renamed and some possibly added to Elasticsearch JSON logs.
There is a project containing a log4j2 layout that can be used to make this task easier. https://github.com/elastic/java-ecs-logging

Also since this would require new fields being added, we should consider making it easier to add more fields in the future to support "special log events" that would make monitoring of Elasticsearch easier. Example would be a special log event when cluster state changes. Draft was done here #44336

elasticmachine · 2019-08-29T10:45:13Z

Pinging @elastic/es-core-infra

pgomulka · 2019-09-05T10:00:09Z

changes between current format and ECS
timestamp -> @timestamp
level -> log.level
component -> log.logger also have a different format org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction vs previously o.e.d.r.a.a.i.RestCreateIndexAction

Additional fields from ES are not a problem, but could be migrated.
These could be be part of service. set? so for instance service.node.name
node.name
node.id
cluster.name
cluster.uuid

we would map these to the custom fields. The changes from the fields above would apply
Search Slow logs
Index Slow logs
Deprecation logs

what other fields should we add to our logs to make it more functional once parsed?

additional fields added by ECSLayout
service.name -> would be cluster name?
process.thread.name -> elasticsearch[node-0][masterService#updateTask][T#1]

ECSLayout defines exceptions these way.
"error.code": "java.lang.IllegalArgumentException",
"error.message": "persistent setting [cluster.routing.allocation.e5nable], not recognized",
"error.stack_trace": array of strings (lines of stactrace

considering moving markers to json fields

cachedout · 2019-09-06T08:32:00Z

cc: @elastic/stack-monitoring for visibility

felixbarny · 2019-09-09T14:08:27Z

ECS specifies service.name and service.id. In the docs, it says this:

In the case of Elasticsearch the service.name could contain the cluster name. For Beats the service.name is by default a copy of the service.type field if no name is specified.

pgomulka · 2019-09-20T11:02:18Z

sample ECS json log line

{"@timestamp":"2019-09-20T10:54:39.539Z", "log.level": "INFO", "message":"adding index lifecycle policy [watch-history-ilm-policy]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[node-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","type":"console","node.id":"suUibtbUTW6fkc5IyaGpNg","node.name":"node-0","cluster.uuid":"4GbuHHtrR4Klf_u8Hmdsng"}

with exception

{"@timestamp":"2019-09-20T11:01:09.040Z", "log.level": "WARN", "message":"path: /_cluster/settings, params: {}", "service.name":"ES_ECS","process.thread.name":"elasticsearch[node-0][http_server_worker][T#3]","log.logger":"rest.suppressed","type":"console","node.id":"_oWcocLVQCOe8ExcWMwZMg","node.name":"node-0","cluster.uuid":"Wz-9LsTlS2mgpEL2yaxhLA","error.code":"com.fasterxml.jackson.core.JsonParseException","error.message":"Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@f835b9d; line: 3, column: 36]","error.stack_trace":[
 "com.fasterxml.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@f835b9d; line: 3, column: 36]",
 "\tat com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702)",
 "\tat com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558)",
 "\tat com.fasterxml.jackson.core.base.ParserMinimalBase._throwUnquotedSpace(ParserMinimalBase.java:522)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(UTF8StreamJsonParser.java:1963)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseLongName(UTF8StreamJsonParser.java:1860)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseMediumName2(UTF8StreamJsonParser.java:1840)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseMediumName(UTF8StreamJsonParser.java:1797)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1732)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:776)",
 "\tat org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readGenericMap(AbstractXContentParser.java:335)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:322)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:375)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.lambda$readMap$0(AbstractXContentParser.java:322)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readGenericMap(AbstractXContentParser.java:342)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:322)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:302)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.map(AbstractXContentParser.java:266)",
 "\tat org.elasticsearch.rest.action.admin.cluster.RestClusterUpdateSettingsAction.prepareRequest(RestClusterUpdateSettingsAction.java:58)",
 "\tat org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:87)",
 "\tat org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:58)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$24(AuthenticationService.java:570)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:579)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.finishAuthentication(AuthenticationService.java:560)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:510)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$16(AuthenticationService.java:404)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)",
 "\tat org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:374)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$3(CachingUsernamePasswordRealm.java:175)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.lambda$doAuthenticate$0(ReservedRealm.java:110)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:202)",
 "\tat org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:88)",
 "\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:166)",
 "\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:103)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:365)",
 "\tat org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:408)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:335)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:345)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:288)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:359)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:269)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:252)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:390)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:122)",
 "\tat org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:55)",
 "\tat org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:220)",
 "\tat org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:293)",
 "\tat org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:164)",
 "\tat org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322)",
 "\tat org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372)",
 "\tat org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301)",
 "\tat org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:69)",
 "\tat org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31)",
 "\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)",
 "\tat io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)",
 "\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)",
 "\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)",
 "\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)",
 "\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:597)",
 "\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:551)",
 "\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)",
 "\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)",
 "\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)",
 "\tat java.base/java.lang.Thread.run(Thread.java:835)"]}

pgomulka · 2020-04-20T15:27:25Z

closed by #47105

Adds breaking change docs for #47105. (ECS layout and plaintext log files removal) Relates to #46119

pgomulka added the :Core/Infra/Logging Log management and logging utilities label Aug 29, 2019

pgomulka self-assigned this Aug 29, 2019

pgomulka mentioned this issue Sep 6, 2019

WIP draft for logging #44336

Closed

pgomulka mentioned this issue Sep 13, 2019

Refactor ESLogMessage to not define fields upfront #46702

Merged

pgomulka mentioned this issue Nov 14, 2019

Logging improvements #49087

Closed

14 tasks

pgomulka mentioned this issue Apr 20, 2020

Make Elasticsearch JSON logs ECS compliant #47105

Merged

pgomulka closed this as completed Apr 20, 2020

jeffspahr mentioned this issue Aug 11, 2020

JSON logging can't be ingested into Elasticsearch elastic/logstash#7610

Open

pgomulka mentioned this issue Jun 22, 2021

Adds minimal traceparent header support to Elasticsearch #74210

Merged

jrodewig mentioned this issue Oct 15, 2021

[DOCS] Migration information about ES logging breaking changes #79146

Merged

pgomulka added a commit that referenced this issue Oct 20, 2021

[DOCS] Migration information about ES logging breaking changes (#79146)

6aa0db8

Adds breaking change docs for #47105. (ECS layout and plaintext log files removal) Relates to #46119

amitmbm mentioned this issue Feb 28, 2022

log4j configuration for slow log files(search and index) is removed in ES 8.0 ? #84405

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Logging] Support for ECS and custom event tags #46119

[Logging] Support for ECS and custom event tags #46119

pgomulka commented Aug 29, 2019 •

edited

Loading

elasticmachine commented Aug 29, 2019

pgomulka commented Sep 5, 2019 •

edited

Loading

cachedout commented Sep 6, 2019

felixbarny commented Sep 9, 2019

pgomulka commented Sep 20, 2019

pgomulka commented Apr 20, 2020 •

edited

Loading

[Logging] Support for ECS and custom event tags #46119

[Logging] Support for ECS and custom event tags #46119

Comments

pgomulka commented Aug 29, 2019 • edited Loading

elasticmachine commented Aug 29, 2019

pgomulka commented Sep 5, 2019 • edited Loading

cachedout commented Sep 6, 2019

felixbarny commented Sep 9, 2019

pgomulka commented Sep 20, 2019

pgomulka commented Apr 20, 2020 • edited Loading

pgomulka commented Aug 29, 2019 •

edited

Loading

pgomulka commented Sep 5, 2019 •

edited

Loading

pgomulka commented Apr 20, 2020 •

edited

Loading