Trino JDBC 440+ is not compatible with OpenTelemetry Java instrumentation < 2.1.0

[gaeljw]

Issue

Given an application with:

• direct dependency to io.trino:trino-jdbc:440 (or 441)
• direct dependency to OpenTelemetry libraries using io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:2.0.0-alpha and more precisely:
  • io.opentelemetry:opentelemetry-sdk:1.34.1
  • io.opentelemetry:opentelemetry-exporter-prometheus:1.34.1-alpha
  • io.opentelemetry.semconv:opentelemetry-semconv:1.21.0-alpha
  • io.opentelemetry.instrumentation:opentelemetry-runtime-telemetry-java17:2.0.0-alpha

The applications fails at runtime with following error java.lang.NoSuchFieldError: ERROR_TYPE.

Stack trace:

[error] Caused by: java.lang.NoSuchFieldError: ERROR_TYPE
[error]     at io.trino.jdbc.$internal.opentelemetry.instrumentation.api.semconv.http.HttpCommonAttributesExtractor.onEnd(HttpCommonAttributesExtractor.java:113)
[error]     at io.trino.jdbc.$internal.opentelemetry.instrumentation.api.semconv.http.HttpClientAttributesExtractor.onEnd(HttpClientAttributesExtractor.java:91)
[error]     at io.trino.jdbc.$internal.opentelemetry.instrumentation.api.instrumenter.Instrumenter.doEnd(Instrumenter.java:230)
[error]     at io.trino.jdbc.$internal.opentelemetry.instrumentation.api.instrumenter.Instrumenter.end(Instrumenter.java:147)
[error]     at io.trino.jdbc.$internal.opentelemetry.instrumentation.okhttp.v3_0.internal.TracingInterceptor.intercept(TracingInterceptor.java:53)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
[error]     at io.trino.jdbc.$internal.client.OkHttpUtil.lambda$userAgent$0(OkHttpUtil.java:72)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
[error]     at io.trino.jdbc.$internal.opentelemetry.instrumentation.okhttp.v3_0.internal.ConnectionErrorSpanInterceptor.intercept(ConnectionErrorSpanInterceptor.java:38)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
[error]     at io.trino.jdbc.$internal.opentelemetry.instrumentation.okhttp.v3_0.ContextInterceptor.intercept(ContextInterceptor.java:28)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
[error]     at io.trino.jdbc.$internal.okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
[error]     at io.trino.jdbc.$internal.opentelemetry.instrumentation.okhttp.v3_0.TracingCallFactory$TracingCall.execute(TracingCallFactory.java:100)
[error]     at io.trino.jdbc.$internal.client.JsonResponse.execute(JsonResponse.java:113)
[error]     at io.trino.jdbc.$internal.client.StatementClientV1.executeRequest(StatementClientV1.java:401)
[error]     at io.trino.jdbc.$internal.client.StatementClientV1.<init>(StatementClientV1.java:132)
[error]     at io.trino.jdbc.$internal.client.StatementClientFactory.newStatementClient(StatementClientFactory.java:28)
[error]     at io.trino.jdbc.TrinoConnection.startQuery(TrinoConnection.java:771)
[error]     at io.trino.jdbc.TrinoStatement.internalExecute(TrinoStatement.java:252)
[error]     at io.trino.jdbc.TrinoStatement.execute(TrinoStatement.java:240)

This doesn't happen with Trino JDBC 439.

Short-term workaround

Upgrading to OpenTelemetry 2.1.0 solves the issue.

Especially thanks to the underlying upgrade to io.opentelemetry.semconv:opentelemetry-semconv:1.23.1-alpha which contains the ERROR_TYPE value added in the semantic conventions 1.22.0 (open-telemetry/semantic-conventions#205).

Long-term solution

While the short-term workaround is fine and easy to apply, I believe this raises questions about Trino partial shading of OpenTelemetry dependencies.

In my understanding, shading is here to be "isolated" from the OpenTelemetry versions used by the wrapping application but as this example shows, it doesn't work as expected. Shouldn't all OpenTelemetry dependencies be shaded for this to work fine?

On the other hand, shading brings other issues like #19958 (and #20074).

I'm definitely not an expert in shading and the requirements needed for a JDBC driver but it seems to me that the setup is not entirely right here.

[mosabua]

Can you take a look @mattstep and @electrum ?

[findinpath]

cc @wendigo