What field to use for Java / Jee logs

[disaster37]

Hi,

you think that I use the right field for Java / Jee logs ?

Sample of logs

2019-07-18 09:43:41,689 ERROR [main] org.springframework.boot.SpringApplication (SpringApplication.java:837) - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'directoryInboundGatewayFlow' defined in class path resource [fr/hm/fwk/batch/directory/config/DefaultDirectoryBatchIntegrationConfiguration.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'directoryInboundGatewayFlow.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0.source': Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: Source directory [/mnt/CMN0B00_ACDC-DEMAT] does not exist.
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:581)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
	at org.springframework.beans.factory.support.AbstractBeanFactory.dt_access$1246(AbstractBeanFactory.java)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:759)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:548)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:386)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1242)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1230)
	at fr.sihm.batch.demat.BatchApplication.main(BatchApplication.java:13)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'directoryInboundGatewayFlow.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0.source': Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: Source directory [/mnt/CMN0B00_ACDC-DEMAT] does not exist.
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
	at org.springframework.beans.factory.support.AbstractBeanFactory.dt_access$1246(AbstractBeanFactory.java)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.integration.config.dsl.IntegrationFlowBeanPostProcessor.registerComponent(IntegrationFlowBeanPostProcessor.java:377)
	at org.springframework.integration.config.dsl.IntegrationFlowBeanPostProcessor.processStandardIntegrationFlow(IntegrationFlowBeanPostProcessor.java:237)
	at org.springframework.integration.config.dsl.IntegrationFlowBeanPostProcessor.postProcessBeforeInitialization(IntegrationFlowBeanPostProcessor.java:110)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1686)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573)
	... 24 common frames omitted
Caused by: java.lang.IllegalArgumentException: Source directory [/mnt/CMN0B00_ACDC-DEMAT] does not exist.
	at org.springframework.util.Assert.isTrue(Assert.java:116)
	at org.springframework.integration.file.FileReadingMessageSource.onInit(FileReadingMessageSource.java:326)
	at org.springframework.integration.file.dsl.FileInboundChannelAdapterSpec$1.onInit(FileInboundChannelAdapterSpec.java:82)
	at org.springframework.integration.context.IntegrationObjectSupport.afterPropertiesSet(IntegrationObjectSupport.java:176)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1753)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1690)
	... 37 common frames omitted

So, with this sample, I should to set the following fields:

• event.dataset: log4j
• event.module: springboot
• @timestamp: 2019-07-18 09:43:41,689
• log.level: error
• event.severity: 100 (integer code that corresponding to log.level, to help user to display only log that have severity under or equal to warning)
• service.ephemeral_id: main (the thread name)
• log.class: org.springframework.boot.SpringApplication
• log.file: SpringApplication.java
• log.line: 837
• log.exception: org.springframework.beans.factory.BeanCreationException
• log.stacktrace: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'directoryInboundGatewayFlow' ... (all line after that)
• message: Application run failed
• service.name: the springboot instance name (global field on fielbeat prospector)
• service.type: the application name (global field on fielbeat prospector)
• service.module: specific appender set in log4j, to generate specific log fiele about webservice call, or sql call)

[webmat]

Hi! We do have in our plans to provide better support for detailed stack trace errors, but this isn't out yet.

In the meantime, here's what I think works well in your mapping:

• event.module
• @timestamp
• log.level
• event.severity
• message
• service.name
• service.type (more on service.name and .type below)

Here's where I would suggest doing the mapping otherwise:

• event.dataset:
  • this field is meant to distinguish between the different kinds of logs or event sources from that same event.module. So perhaps in your case I would expect springboot.exception (or .error), and maybe springboot.log or springboot.access, depending on the types of logs & events you have. If it's all in the same log file, but there are really different types of events, you could still assign them a different dataset. This way you can easily grab the errors and not the other logs and so on.
  • We also have the convention of prepending the module name to the dataset name (as you can see in my examples above)
  • I just spotted what you have in service.module, perhaps this is what should go in event.dataset. Distinguish between errors, db logs, webservice access logs, etc.
• service.ephemeral_id: this is meant to distinguish between different runs of a service. If you don't have a service ID that changes between runs, you don't need to populate it.

Since ECS doesn't have support for stack trace details yet, I would instead suggest using custom fields for this. Currently some of the mappings you're considering would overwrite other valid information (e.g. log.line is really the line number in the log file where the event came from).

So to avoid conflicts with future versions of ECS introducing support for this, you should nest under a field name like your company name or project name. Something that's unlikely to be added to ECS (brand names shouldn't get into ECS). So let's say your project is called Phoenix, you could nest under phoenix.exception.* where you have the fields for code line, exception, stack trace, class name, function name, thread name, etc. Or if you're a shop that creates many Java or Springboot applications, perhaps it would make sense for all exceptions to be nested at java.exception.* or springboot.exception.*. This way these exceptions are always "in the same place" no matter the application you're looking at. This enables you to reuse the same visualizations, no matter what project.

service name and type

And a final note on service.name and .type: we've established a pattern where service.type is the name of the software itself, whereas service.name is how you name this specific instance of the running service.

So for 3rd party software you could have .type=mysql and .name=finance-database. In the case of custom software, like your situation, I'd suggest giving .type=[name of the software project] and .name could distinguish between your production and various test environments, it could also distinguish various geolocated deployments. In other words, the name of that running instance of the service.

Let me know if that helps, and if you have other questions

[jderose9]

I have the same question After reading this I thought the advice was to replicate the exception.error node from APM under the ECS error node. Are you saying it would be better just to create a unique top level field?

I'd like to capture exceptions from applications in multiple languages in a similar format. In addition to the stack trace, there are module locations, thread ids, return codes and that sort of stuff. It's proving fairly difficult to map everything, especially when you start factoring in platform specific fields. Some (such as .net) also have nested exceptions which means either entering them as separate documents in elasticsearch and providing some sort of parent id field, or walking through the tree and creating further levels of nesting.

[felixbarny]

Do you think that the logger name may be a good fit for the event.dataset field? The logger name is mostly the full name of the class which initiated the logger. But it can also be a completely custom name for the logger. Some examples:

org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet]
org.springframework.web.servlet.DispatcherServlet
login-audit

See also https://www.slf4j.org/apidocs/org/slf4j/LoggerFactory.html#getLogger(java.lang.String)

[webmat]

With the various ECS loggers maturing, not only this is addressed pretty well, but the next few Stack releases will continue making that easier than ever.

Please open new issues if we're missing specific fields related to application logs 👍