Promtail 400 bad request when log message is too long

[szikk]

Hello,

I am facing an issue using promtail to parse the stout of a k8s pod. The logs are in json format and everything works fine until the the stacktrace is too long:

Scrape config:

- job_name: custom-config
          pipeline_stages:
            - cri: {}
            - json:
                expressions:
                  timestamp: timestamp
                  level: level
                  service: service
                  traceId: traceId
                  spanId: spanId
                  thread: thread
                  class: logger
                  message: message
                  context: context
                  stack_trace: stack_trace
            - labels:
                timestamp:
                level:
                service:
                traceId:
                spanId:
                thread:
                class:
                message:
                context:
                stack_trace:
            - timestamp:
                format: RFC3339
                source: timestamp
            - output:
                source: message
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels:
                - __meta_kubernetes_pod_controller_name
              regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
              action: replace
              target_label: __tmp_controller_name
            - source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_name
                - __meta_kubernetes_pod_label_app
                - __tmp_controller_name
                - __meta_kubernetes_pod_name
              regex: ^;*([^;]+)(;.*)?$
              action: replace
              target_label: app
            - source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_instance
                - __meta_kubernetes_pod_label_release
              regex: ^;*([^;]+)(;.*)?$
              action: replace
              target_label: instance
            - source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_component
                - __meta_kubernetes_pod_label_component
              regex: ^;*([^;]+)(;.*)?$
              action: replace
              target_label: component
            - action: replace
              source_labels:
              - __meta_kubernetes_pod_node_name
              target_label: node_name
            - action: replace
              source_labels:
              - __meta_kubernetes_namespace
              target_label: namespace
            - action: replace
              replacement: $1
              separator: /
              source_labels:
              - namespace
              - app
              target_label: job
            - action: replace
              source_labels:
              - __meta_kubernetes_pod_name
              target_label: pod
            - action: replace
              source_labels:
              - __meta_kubernetes_pod_container_name
              target_label: container
            - action: replace
              replacement: /var/log/pods/*$1/*.log
              separator: /
              source_labels:
              - __meta_kubernetes_pod_uid
              - __meta_kubernetes_pod_container_name
              target_label: __path__
            - action: replace
              regex: true/(.*)
              replacement: /var/log/pods/*$1/*.log
              separator: /
              source_labels:
              - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
              - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
              - __meta_kubernetes_pod_container_name
              target_label: __path__

Log message

{
	"timestamp": "2024-01-10T11:04:11.074",
	"level": "ERROR",
	"service": "profile-service",
	"traceId": "659e79aacc9b494a407b55d29315e273",
	"spanId": "7e54469782462843",
	"thread": "http-nio-8080-exec-10",
	"class": "c.b.p.exception.CustomExceptionHandler",
	"message": "Internal server error response generated",
	"stack_trace": "\ncom.test.profileservice.exception.FirebaseUserException: Deleting user failed\n at com.test.profileservice.service.FirebaseUserManager.deleteFirebaseUser(FirebaseUserManager.java:19)\n at com.test.profileservice.service.ProfileServiceImpl.deleteUserProfile(ProfileServiceImpl.java:116)\n at com.test.profileservice.controller.ProfileController.deleteUserProfile(ProfileController.java:275)\n at jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)\n at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.base/java.lang.reflect.Method.invoke(Method.java:568)\n at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)\n at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.invoke(AuthorizationManagerBeforeMethodInterceptor.java:199)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)\n at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)\n at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)\n at com.test.profileservice.controller.ProfileController$$SpringCGLIB$$0.deleteUserProfile(<generated>)\n at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\n at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.base/java.lang.reflect.Method.invoke(Method.java:568)\n at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:207)\n at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:152)\n at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)\n at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884)\n at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)\n at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)\n at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)\n at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011)\n at org.springframework.web.servlet.FrameworkServlet.doDelete(FrameworkServlet.java:936)\n at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:596)\n at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)\n at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)\n at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479)\n at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340)\n at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:128)\n at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)\n at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)\n at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:145)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)\n at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)\n at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)\n at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)\n at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323)\n at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224)\n at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)\n at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)\n at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)\n at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)\n at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)\n at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)\n at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)\n at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166)\n at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)\n at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)\n at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)\n at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)\n at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)\n at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:738)\n at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)\n at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)\n at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)\n at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894)\n at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)\n at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)\n at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)\n at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)\n at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n at java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.google.firebase.auth.FirebaseAuthException: No user record found for the given identifier (USER_NOT_FOUND).\n at com.google.firebase.auth.internal.AuthErrorHandler.createException(AuthErrorHandler.java:119)\n at com.google.firebase.auth.internal.AuthErrorHandler.createException(AuthErrorHandler.java:35)\n at com.google.firebase.internal.AbstractHttpErrorHandler.handleHttpResponseException(AbstractHttpErrorHandler.java:57)\n at com.google.firebase.internal.ErrorHandlingHttpClient.send(ErrorHandlingHttpClient.java:108)\n at com.google.firebase.auth.internal.AuthHttpClient.sendRequest(AuthHttpClient.java:75)\n at com.google.firebase.auth.internal.AuthHttpClient.sendRequest(AuthHttpClient.java:69)\n at com.google.firebase.auth.FirebaseUserManager.post(FirebaseUserManager.java:351)\n at com.google.firebase.auth.FirebaseUserManager.deleteUser(FirebaseUserManager.java:179)\n at com.google.firebase.auth.AbstractFirebaseAuth$14.execute(AbstractFirebaseAuth.java:852)\n at com.google.firebase.auth.AbstractFirebaseAuth$14.execute(AbstractFirebaseAuth.java:849)\n at com.google.firebase.internal.CallableOperation.call(CallableOperation.java:36)\n at com.google.firebase.auth.AbstractFirebaseAuth.deleteUser(AbstractFirebaseAuth.java:830)\n at com.test.profileservice.service.FirebaseUserManager.deleteFirebaseUser(FirebaseUserManager.java:17)\n ... 138 common frames omitted\nCaused by: com.google.api.client.http.HttpResponseException: 400 Bad Request\nPOST https://identitytoolkit.googleapis.com/v1/projects/youbastard/accounts:delete\n{\n \"error\": {\n \"code\": 400,\n \"message\": \"USER_NOT_FOUND\",\n \"errors\": [\n {\n \"message\": \"USER_NOT_FOUND\",\n \"domain\": \"global\",\n \"reason\": \"invalid\"\n }\n ]\n }\n}\n at com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)\n at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1118)\n at com.google.firebase.internal.ErrorHandlingHttpClient.send(ErrorHandlingHttpClient.java:96)\n ... 147 common frames omitted\n "
}

Error message in promtail

level=error ts=2024-01-10T11:04:11.128441781Z caller=client.go:430 component=client host=loki-stack:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): stream '{app=\"profile-service\", container=\"profile-service\", filename=\"/var/log/pods/default_profile-service-579b4765c7-4xw84_6b09d696-71da-4dd7-8e37-743933056ac7/profile-service/0.log\", instance=\"profile-service\", job=\"default/profile-service\", level=\"ERROR\", message=\"Internal server error response generated\", namespace=\"default\", node_name=\"aks-agentpool-32864626-vmss000018\", pod=\"profile-service-579b4765c7-4xw84\", service=\"profile-service\", spanId=\"7e54469782462843\", stack_trace=\"\\ncom.test.profileservice.exception.FirebaseUserException: Deleting user failed\\n\\tat com.test.profileservice.service.FirebaseUserManager.deleteFirebaseUser(FirebaseUserManager.java:19)\\n\\tat com.test.profileservice.service.ProfileServiceImpl.deleteUserProfile(ProfileServiceImpl.java:116)\\n\\tat com.test.profileservice.controller.ProfileController.deleteUserProfile(ProfileController.java:275)\\n\\tat jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)\\n\\tat java.base/jdk.internal.reflect.Dele"

Loki limits:

  limits_config:
    enforce_metric_name: false
    reject_old_samples: true
    reject_old_samples_max_age: 168h
    max_entries_limit_per_query: 40000
    max_line_size: 0

[JStickler]

Questions have a better chance of being answered if you ask them on the community forums.

[szikk]

I have posted this 16 days ago:
https://community.grafana.com/t/promtail-400-bad-request-when-stacktrace-is-too-large/111665