OpenTelemetry gRPC OTLP traces exporter sometimes throws NPE on app shutdown when HTTP2 connection stream is closed #41351

michalvavrik · 2024-06-21T09:47:42Z

Describe the bug

I have app where I mock OTel collector with a Vert.x gRPC server so that I can avoid running container where I don't need to. I can see exported traces alright, but when Quarkus app is shutting down, sometimes can see logged NPE. The NPE doesn't affect me (though it fails my test that checks log), but looking at the error, I'm not really sure how my mock could cause that.

Expected behavior

I wonder if I have a wrongly written this gRPC server, but the error is bit strange. I'd prefer someone to check it. I tried it with a Jaeger container and could not reproduce the issue. On the other hand, as I think there is a race, it's logical that race cannot be reproduced in some scenarios.

Actual behavior

2024-06-21 11:35:40,472 INFO  [io.quarkus] (main) app-full-microprofile 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.11.0) started in 1.039s. Listening on: http://0.0.0.0:8080
2024-06-21 11:35:40,479 INFO  [io.quarkus] (main) Profile prod activated. 
2024-06-21 11:35:40,479 INFO  [io.quarkus] (main) Installed features: [cdi, micrometer, opentelemetry, rest, rest-client, security, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, smallrye-jwt, smallrye-openapi, vertx]
2024-06-21 11:35:43,233 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-2) Uncaught exception received by Vert.x [Error Occurred After Shutdown]: java.lang.NullPointerException: Cannot invoke "io.smallrye.context.SmallRyeContextManager.defaultThreadContext()" because the return value of "io.smallrye.context.SmallRyeContextManagerProvider.getManager()" is null
	at io.smallrye.context.SmallRyeThreadContext.getCurrentThreadContextOrDefaultContexts(SmallRyeThreadContext.java:160)
	at io.smallrye.mutiny.context.DefaultContextPropagationInterceptor.getThreadContext(DefaultContextPropagationInterceptor.java:12)
	at io.smallrye.mutiny.context.BaseContextPropagationInterceptor.decorate(BaseContextPropagationInterceptor.java:24)
	at io.smallrye.mutiny.infrastructure.Infrastructure.decorate(Infrastructure.java:149)
	at io.smallrye.mutiny.groups.UniCreate.completionStage(UniCreate.java:140)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.VertxGrpcExporter.initiateSend(VertxGrpcExporter.java:112)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.VertxGrpcExporter$ClientRequestOnSuccessHandler$2.handle(VertxGrpcExporter.java:402)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.VertxGrpcExporter$ClientRequestOnSuccessHandler$2.handle(VertxGrpcExporter.java:397)
	at io.vertx.core.impl.future.FutureImpl$2.onFailure(FutureImpl.java:117)
	at io.vertx.core.impl.future.FutureImpl$ListenerArray.onFailure(FutureImpl.java:316)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:81)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:278)
	at io.vertx.core.impl.future.Mapping.onFailure(Mapping.java:45)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:81)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:278)
	at io.vertx.core.http.impl.HttpClientRequestBase.fail(HttpClientRequestBase.java:182)
	at io.vertx.core.http.impl.HttpClientRequestBase.handleException(HttpClientRequestBase.java:177)
	at io.vertx.core.http.impl.HttpClientRequestImpl.handleException(HttpClientRequestImpl.java:90)
	at io.vertx.core.http.impl.Http2ClientConnection$StreamImpl.handleException(Http2ClientConnection.java:543)
	at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:328)
	at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:321)
	at io.vertx.core.http.impl.VertxHttp2Stream.onException(VertxHttp2Stream.java:96)
	at io.vertx.core.http.impl.Http2ConnectionBase.onStreamClosed(Http2ConnectionBase.java:153)
	at io.vertx.core.http.impl.VertxHttp2ConnectionHandler$1.onStreamClosed(VertxHttp2ConnectionHandler.java:93)
	at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1036)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:992)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:517)
	at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:153)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:217)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:432)
	at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.channelInactive(VertxHttp2ConnectionHandler.java:186)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
	at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:280)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

2024-06-21 11:35:43,234 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-2) Uncaught exception received by Vert.x [Error Occurred After Shutdown]: java.lang.NullPointerException: Cannot invoke "io.smallrye.context.SmallRyeContextManager.defaultThreadContext()" because the return value of "io.smallrye.context.SmallRyeContextManagerProvider.getManager()" is null
	at io.smallrye.context.SmallRyeThreadContext.getCurrentThreadContextOrDefaultContexts(SmallRyeThreadContext.java:160)
	at io.smallrye.mutiny.context.DefaultContextPropagationInterceptor.getThreadContext(DefaultContextPropagationInterceptor.java:12)
	at io.smallrye.mutiny.context.BaseContextPropagationInterceptor.decorate(BaseContextPropagationInterceptor.java:24)
	at io.smallrye.mutiny.infrastructure.Infrastructure.decorate(Infrastructure.java:149)
	at io.smallrye.mutiny.groups.UniCreate.completionStage(UniCreate.java:140)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.VertxGrpcExporter.initiateSend(VertxGrpcExporter.java:112)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.VertxGrpcExporter$ClientRequestOnSuccessHandler$2.handle(VertxGrpcExporter.java:402)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.VertxGrpcExporter$ClientRequestOnSuccessHandler$2.handle(VertxGrpcExporter.java:397)
	at io.vertx.core.impl.future.FutureImpl$2.onFailure(FutureImpl.java:117)
	at io.vertx.core.impl.future.FutureImpl$ListenerArray.onFailure(FutureImpl.java:316)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:81)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:278)
	at io.vertx.core.impl.future.Mapping.onFailure(Mapping.java:45)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:81)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:278)
	at io.vertx.core.http.impl.HttpClientRequestBase.fail(HttpClientRequestBase.java:182)
	at io.vertx.core.http.impl.HttpClientRequestBase.handleException(HttpClientRequestBase.java:177)
	at io.vertx.core.http.impl.HttpClientRequestImpl.handleException(HttpClientRequestImpl.java:90)
	at io.vertx.core.http.impl.Http2ClientConnection$StreamImpl.handleException(Http2ClientConnection.java:543)
	at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:328)
	at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:321)
	at io.vertx.core.http.impl.VertxHttp2Stream.onException(VertxHttp2Stream.java:96)
	at io.vertx.core.http.impl.Http2ConnectionBase.onStreamClosed(Http2ConnectionBase.java:153)
	at io.vertx.core.http.impl.VertxHttp2ConnectionHandler$1.onStreamClosed(VertxHttp2ConnectionHandler.java:93)
	at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1036)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:992)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:517)
	at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:153)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:217)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:432)
	at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.channelInactive(VertxHttp2ConnectionHandler.java:186)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
	at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:280)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

2024-06-21 11:35:43,235 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (vert.x-eventloop-thread-2) Uncaught exception received by Vert.x [Error Occurred After Shutdown]: java.lang.NullPointerException: Cannot invoke "io.smallrye.context.SmallRyeContextManager.defaultThreadContext()" because the return value of "io.smallrye.context.SmallRyeContextManagerProvider.getManager()" is null
	at io.smallrye.context.SmallRyeThreadContext.getCurrentThreadContextOrDefaultContexts(SmallRyeThreadContext.java:160)
	at io.smallrye.mutiny.context.DefaultContextPropagationInterceptor.getThreadContext(DefaultContextPropagationInterceptor.java:12)
	at io.smallrye.mutiny.context.BaseContextPropagationInterceptor.decorate(BaseContextPropagationInterceptor.java:24)
	at io.smallrye.mutiny.infrastructure.Infrastructure.decorate(Infrastructure.java:149)
	at io.smallrye.mutiny.groups.UniCreate.completionStage(UniCreate.java:140)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.VertxGrpcExporter.initiateSend(VertxGrpcExporter.java:112)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.VertxGrpcExporter$ClientRequestOnSuccessHandler$2.handle(VertxGrpcExporter.java:402)
	at io.quarkus.opentelemetry.runtime.exporter.otlp.VertxGrpcExporter$ClientRequestOnSuccessHandler$2.handle(VertxGrpcExporter.java:397)
	at io.vertx.core.impl.future.FutureImpl$2.onFailure(FutureImpl.java:117)
	at io.vertx.core.impl.future.FutureImpl$ListenerArray.onFailure(FutureImpl.java:316)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:81)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:278)
	at io.vertx.core.impl.future.Mapping.onFailure(Mapping.java:45)
	at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:81)
	at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:278)
	at io.vertx.core.http.impl.HttpClientRequestBase.fail(HttpClientRequestBase.java:182)
	at io.vertx.core.http.impl.HttpClientRequestBase.handleException(HttpClientRequestBase.java:177)
	at io.vertx.core.http.impl.HttpClientRequestImpl.handleException(HttpClientRequestImpl.java:90)
	at io.vertx.core.http.impl.Http2ClientConnection$StreamImpl.handleException(Http2ClientConnection.java:543)
	at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:328)
	at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:321)
	at io.vertx.core.http.impl.VertxHttp2Stream.onException(VertxHttp2Stream.java:96)
	at io.vertx.core.http.impl.Http2ConnectionBase.onStreamClosed(Http2ConnectionBase.java:153)
	at io.vertx.core.http.impl.VertxHttp2ConnectionHandler$1.onStreamClosed(VertxHttp2ConnectionHandler.java:93)
	at io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1036)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:992)
	at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:517)
	at io.netty.handler.codec.http2.DefaultHttp2Connection.close(DefaultHttp2Connection.java:153)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$BaseDecoder.channelInactive(Http2ConnectionHandler.java:217)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.channelInactive(Http2ConnectionHandler.java:432)
	at io.vertx.core.http.impl.VertxHttp2ConnectionHandler.channelInactive(VertxHttp2ConnectionHandler.java:186)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
	at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:280)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

How to Reproduce?

Steps to reproduce the behavior:

git clone git@github.com:michalvavrik/quarkus-startstop.git -b feature/make-app-more-real-latest-repro
cd quarkus-startstop
mvn clean verify -pl testsuite -Dtest=StartStopTest#fullMicroProfileJVM -Dstart-stop.iterations=25

With branch feature/make-app-more-real-latest-repro I reproduced it only once, however I consider it important as there, the use case is very simple, it's just a gRPC server that inspect exported traces. No magic.

UPDATE: I already managed to reproduce it few times on the feature/make-app-more-real-latest-repro branch.

I also have a branch where at one point, I reproduced it almost every single time, try git checkout feature/make-app-real-repro. In this branch, there is more noise that is unnecessary for my tests.

Output of `uname -a` or `ver`

Fedora 38

Output of `java -version`

OpenJDK 21

Quarkus version or git rev

999-SNAPSHOT

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Apache Maven 3.9.4

Additional information

No response

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2024-06-21T09:47:46Z

/cc @alesj (grpc), @brunobat (opentelemetry,tracing), @cescoffier (grpc), @radcortez (opentelemetry,tracing)

Fixes: quarkusio#41351

geoand · 2024-06-25T15:14:33Z

Can you try #41438 please?

Avoid exporter related exceptions when application has been shutdown

michalvavrik · 2024-06-26T07:08:21Z

Can you try #41438 please?

Sorry @geoand , I didn't have time between your question and merging. However I am going to enable this in our test suite, therefore we will know as it is run at least once a day. Thank you for the fix

geoand · 2024-06-26T08:02:31Z

Cool, let me know how it goes

Fixes: quarkusio#41351 (cherry picked from commit 39fd819)

Fixes: quarkusio#41351

Fixes: quarkusio#41351 (cherry picked from commit 39fd819)

rsvoboda · 2024-09-16T09:00:08Z

With 3.8.6 I've seen this:

__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2024-09-13 12:43:50,981 INFO  [io.quarkus] (main) app-full-microprofile 1.0.0-SNAPSHOT native (powered by Quarkus 3.8.6.redhat-00004) started in 0.033s. Listening on: http://0.0.0.0:8080
2024-09-13 12:43:50,982 INFO  [io.quarkus] (main) Profile prod activated. 
2024-09-13 12:43:50,982 INFO  [io.quarkus] (main) Installed features: [cdi, micrometer, opentelemetry, rest-client-reactive, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, smallrye-openapi, vertx]
2024-09-13 12:43:52,949 SEVERE [io.qua.ope.run.exp.otl.VertxGrpcExporter] (vert.x-eventloop-thread-2) Failed to export spans. The request could not be executed. Full error message: null [Error Occurred After Shutdown]
2024-09-13 12:43:52,949 SEVERE [io.qua.ope.run.exp.otl.VertxGrpcExporter] (vert.x-eventloop-thread-2) Failed to export spans. The request could not be executed. Full error message: null [Error Occurred After Shutdown]
2024-09-13 12:43:52,950 INFO  [io.quarkus] (Shutdown thread) app-full-microprofile stopped in 1.002s

It's not stacktrace like in the original report, but still something is not ok. CC @geoand

I've seen this in NATIVE mode, not yet in JVM mode. Faster on startup, faster on shutdown :) ?

geoand · 2024-09-16T09:06:21Z

This isn't necessary an indication that something went wrong - it could just be that the application started up and shutdown before any traces could be exported

brunobat · 2024-09-16T10:26:04Z

Correct. Server was shutting down, the event loop was down already and no spans could be exported.

Fixes: quarkusio#41351

michalvavrik added the kind/bug Something isn't working label Jun 21, 2024

quarkus-bot bot added area/grpc gRPC area/smallrye area/tracing labels Jun 21, 2024

geoand added a commit to geoand/quarkus that referenced this issue Jun 25, 2024

Avoid exporter related exceptions when application has been shutdown

39fd819

Fixes: quarkusio#41351

geoand mentioned this issue Jun 25, 2024

Avoid exporter related exceptions when application has been shutdown #41438

Merged

brunobat closed this as completed in #41438 Jun 26, 2024

brunobat added a commit that referenced this issue Jun 26, 2024

Merge pull request #41438 from geoand/#41351

48c21c8

Avoid exporter related exceptions when application has been shutdown

quarkus-bot bot added this to the 3.13 - main milestone Jun 26, 2024

gsmet modified the milestones: 3.13 - main, 3.12.1 Jun 26, 2024

gsmet pushed a commit to gsmet/quarkus that referenced this issue Jun 26, 2024

Avoid exporter related exceptions when application has been shutdown

3f9342c

Fixes: quarkusio#41351 (cherry picked from commit 39fd819)

michalvavrik mentioned this issue Jul 11, 2024

Bump Quarkus to 3.12.1 and remove whitelisted log item fixed in 3.12.1 quarkus-qe/quarkus-startstop#380

Closed

holly-cummins pushed a commit to holly-cummins/quarkus that referenced this issue Jul 31, 2024

Avoid exporter related exceptions when application has been shutdown

66d94e9

Fixes: quarkusio#41351

This was referenced Aug 5, 2024

No graceful shutdown with quarkus native #42321

Closed

Fix whitelist regex for 41351 issue quarkus-qe/quarkus-startstop#382

Merged

gsmet modified the milestones: 3.12.1, 3.8.6 Aug 14, 2024

gsmet pushed a commit to gsmet/quarkus that referenced this issue Aug 14, 2024

Avoid exporter related exceptions when application has been shutdown

7a72dcf

Fixes: quarkusio#41351 (cherry picked from commit 39fd819)

danielsoro pushed a commit to danielsoro/quarkus that referenced this issue Sep 20, 2024

Avoid exporter related exceptions when application has been shutdown

43f3909

Fixes: quarkusio#41351

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenTelemetry gRPC OTLP traces exporter sometimes throws NPE on app shutdown when HTTP2 connection stream is closed #41351

OpenTelemetry gRPC OTLP traces exporter sometimes throws NPE on app shutdown when HTTP2 connection stream is closed #41351

michalvavrik commented Jun 21, 2024 •

edited

Loading

quarkus-bot bot commented Jun 21, 2024

geoand commented Jun 25, 2024

michalvavrik commented Jun 26, 2024

geoand commented Jun 26, 2024

rsvoboda commented Sep 16, 2024 •

edited

Loading

geoand commented Sep 16, 2024

brunobat commented Sep 16, 2024

OpenTelemetry gRPC OTLP traces exporter sometimes throws NPE on app shutdown when HTTP2 connection stream is closed #41351

OpenTelemetry gRPC OTLP traces exporter sometimes throws NPE on app shutdown when HTTP2 connection stream is closed #41351

Comments

michalvavrik commented Jun 21, 2024 • edited Loading

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

quarkus-bot bot commented Jun 21, 2024

geoand commented Jun 25, 2024

michalvavrik commented Jun 26, 2024

geoand commented Jun 26, 2024

rsvoboda commented Sep 16, 2024 • edited Loading

geoand commented Sep 16, 2024

brunobat commented Sep 16, 2024

michalvavrik commented Jun 21, 2024 •

edited

Loading

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

rsvoboda commented Sep 16, 2024 •

edited

Loading