-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Java] [OpenTelemetry] Trino generates a huge cardinality of metrics http_client_duration #19958
Comments
I don't see an easy workaround for this. I tried to set I think this feature should really be made opt-in rather than the default. |
Here's a minimal reproduction case: https://github.com/gaeljw/otel9972 I hope it will help. To reproduce:
You'll see a bunch of metrics having the issue. For the record, here are the top lines I get in my case:
You can see the |
From open-telemetry/opentelemetry-java-instrumentation#9972 (comment)
|
As per the last comment at open-telemetry/opentelemetry-java-instrumentation#9972 (comment), would anyone object if I provide a PR to not shade OTEL Extension dependency? @nineinchnick I believe you worked on the OTEL stuff in Trino |
…trinodb#19958) Otherwise, OTEL's HttpMetricsAdvice class checks doesn't work and this result in many harmful labels to be exposed in OkHttp metrics like the http_response_content_length and http_url. trinodb#19958
…rinodb#19958) Otherwise, OTEL's HttpMetricsAdvice class checks doesn't work and this result in many harmful labels to be exposed in OkHttp metrics like the http_response_content_length and http_url. Fixes: trinodb#19958
OTEL has been added in the JDBC driver to support traces, specifically to propagate them to the server. It wasn't supposed to register any metrics, so I think we should try to turn that off. |
Note that with the work done in open-telemetry/opentelemetry-java-instrumentation#10671, the metrics won't be exposed anymore. This doesn't fix the underlying root cause as #20074 aims to, but it solves the consequences. |
I confirm that with recent trino-jdbc version (445), the metrics are not exposed anymore. |
I just noticed that using the Trino JDBC driver and OpenTelemetry metrics results in a huge cardinality for metrics
http_client_duration
generated by the Trino client.Examples of such metrics (Prometheus format):
As these are histograms, there is also a bunch of other metrics
http_client_duration_milliseconds_bucket
.The high cardinality comes from the fact that
http_url
label is different for each request sent to Trino. Same forhttp_response_content_length
.IMHO it doesn't make sense to be enabled by default (I'm not using auto instrumentation, only manual instrumentation moreover), and even if chosen it's not relevant in Trino's context to have these metrics as the URL is changing for each query.
I'm using Trino JDBC 433 and OpenTelemetry 1.31.0 but the issue was present already with OpenTelemetry 1.29.0 and Trino JDBC 427.
The text was updated successfully, but these errors were encountered: