Exclude URLs from Tracing

[jakob-o]

Is your feature request related to a problem? Please describe.
As already mentioned here open-telemetry/opentelemetry-specification#173 I'd like to be able to exclude or sample a list of URLs / URL-Patterns from instrumentation. In my case particularly to avoid generating many events from health- and liveness-checks.

Describe the solution you'd like
I opened the issue open-telemetry/opentelemetry-java#1552 to discuss if / how tracing might be disabled from instrumentation.
To my knowledge there currently is no API / SDK method to disable tracing centrally on the context.
If I understood correctly a (maybe temporary) solution might be to create a non-recording / invalid span in the HTTP instrumentation, which due to the ParentOrElse-Sampler, would lead to ignoring child spans as well, if the request matches a URL pattern.
Any hint to where this would be architecturally appropriately implemented is highly appreciated.
Maybe in the HttpServerTracer?

Describe alternatives you've considered
We already attempted to use the otel.trace.classes.exclude but only succeeded in completely disabling WebMvc instrumentation.

/CC @gabrielthunig @spaletta

[iNikem]

If I understood correctly a (maybe temporary) solution might be to create a non-recording / invalid span in the HTTP instrumentation, which due to the ParentOrElse-Sampler, would lead to ignoring child spans as well, if the request matches a URL pattern.
Any hint to where this would be architecturally appropriately implemented is highly appreciated.
Maybe in the HttpServerTracer

That seems like a reasonable approach :)

[anuraaga]

Been trying auto instrumentation in a container lately, and was slightly annoyed myself with tracing of health check. Then a customer trying it out independently had the same feedback - it's exactly the kind of input we're hoping for in trials :) So I may actually mark this required for GA, the UX is impacted a lot with having no control of tracing by URL pattern.

[pavolloffay]

I have started looking into this.

Is there a proposed design for this? In OpenTracing this was an instrumentation feature. The instrumentation check if the URL matches exclude pattern if yes then the span wasn't created. However if the excluded URL uses another instrumentation (or makes downstream call) that would create a span. The question is whether we want to exclude just specific URLs or the whole trace starting at that URL.

[iNikem]

Yes, we have a proposal right in the task's description:

If I understood correctly a (maybe temporary) solution might be to create a non-recording / invalid span in the HTTP instrumentation, which due to the ParentOrElse-Sampler, would lead to ignoring child spans as well, if the request matches a URL pattern.
Any hint to where this would be architecturally appropriately implemented is highly appreciated.
Maybe in the HttpServerTracer?

This should lead to ignoring the whole subtree starting from that SERVER span.

[pavolloffay]

+1 I think that is the right way to go open-telemetry/opentelemetry-specification#173 (comment). It would also make sense to have a consolidated config property for this.

@iNikem what is the API to create non-recording span? In OT there was sampling.priority=bool tag that could be applied on the span builder https://github.com/opentracing/specification/blob/master/semantic_conventions.yaml#L26

[iNikem]

I think the right way is to use one of the factory methods on io.opentelemetry.trace.DefaultSpan.

[pavolloffay]

Yeah, I wanted to avoid having two paths of span creation.

[anuraaga]

Sorry if double-spam, I thought I had already posted this. How about we have a special Sampler itself configured that delegates to the default, except for when the path matches the allow list, then it's ParentOnly? It means we need to refactor to make sure our tracers set attributes on Span.Builder instead of Span - a little annoying but we should have been doing that already so maybe good motivation for it.

[anuraaga]

Note that DefaultSpan might get renamed to something that would be naming-wise not a good fit with what we want to do here open-telemetry/opentelemetry-specification#994 (review)

[iNikem]

refactor to make sure our tracers set attributes on Span.Builder instead of Span

Yes, we want to do that eventually.

[cemo]

Is there a way now to exclude health check traces? I checked processors but could not find any solution too.

[iNikem]

No, this functionality is not yet implemented.

[irl-segfault]

bump. Is there any workaround here in the meantime? Sampling of health checks is not ideal.

[iNikem]

bump. Is there any workaround here in the meantime? Sampling of health checks is not ideal.

The only known workaround is to write custom sampler.

But I have plans to address this issue during the next month or so.

[iNikem]

The corresponding Sampler has been implemented in the contrib repo. It can be added to your deployment using extension mechanism. There is no immediate plans to add that sampler into this distribution, as this requires changes in Otel Specification and that requires some effort.

[trask]

linking related: open-telemetry/oteps#240

[shameekagarwal]

my architecture -
app -> otel agent (auto instrumentation) -> otel collector -> observability backend
for now, i wanted to filter out swagger and health check calls (caused due to k8s readiness probe)

solution 1 - filter processor. issue - resulted in orphaned spans

solution 2 - tail sampling processor. till now, looks like it works

[nedcerneckis]

Is there any update on this?

I understand the concerns raised by @iNikem about the specification being hard to change but this feature has been requested by numerous people for quite some time now.

I think this is a very important and much-needed feature for a lot of users of OpenTelemetry.

@trask Has anyone tried to contribute to the project with this feature? I would like to take a go at implementing it for the Otel Java agent.

[trask]

hi @nedcerneckis!

I understand the concerns raised by @iNikem about the specification being hard to change but this feature has been requested by numerous people for quite some time now.

I don't believe we're blocked by specification work, since we can add this initially as an opt-in experimental feature (bypassing the need for a specification)

Has anyone tried to contribute to the project with this feature? I would like to take a go at implementing it for the Otel Java agent.

not yet, that would be great, check out #1060 (comment) for very high-level sketch that could fit well with existing feature set

[kenfinnigan]

For Lumigo's distribution I implemented a SamplerCustomizer for use with AutoConfigure, see here.

My approach was to allow urls for client and server to be filter, or separate env vars to filter urls for only client or server spans. The urls are defined as an array of regex, such as [".*/health.*", ".*/actuator.*"]

[nedcerneckis]

Thank you very much @trask! Much appreciated for the info.

I'm assuming this needs to be a little more advanced than just excluding certain HTTP endpoints and include other types of rules inside this YAML config file?

[EvertonSA]

Based on @jack-berg examples, and the possibility to embed extension directly into the agent, I've created this simple project that provides both the java agent and the docker auto instrumentation that will ignore /health and /metrics calls.

life savier, for me this is the easiest solution, but I only needed the jar file:

stages:
  - get_jar
  - distribute

variables:
  OTEL_LIB_VERSION: 1.32.0

Get Jar:
  stage: get_jar
  image: ghcr.io/vmaleze/opentelemetry-java-ignore-spans:$OTEL_LIB_VERSION
  script:
    - cp /javaagent.jar ./
  artifacts:
    paths:
      - './javaagent.jar'
    expire_in: 1 week

Deploy JAR:
  image: maven:3.6-openjdk-17-slim
  stage: distribute
  rules:
  script:
    - mvn deploy -DskipTests

[apischan]

I'm one who also waiting for that feature, but I solved this problem from another side (this approach is probably not for everyone).
The task was to disable actuator healthckecks /actuator/health on tracing UI. So here is what I did.
I've added the filter on opentelemetry-collector's side adding processor filter to force it to skip spans from actuator.

Here is the documentation about processors: https://opentelemetry.io/docs/collector/configuration/#processors
Here is the list of available filters: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor

In total now I have such configuration on my project (which I'll certainly would like to change on config property when it'll be implemented):

config:
  exporters:
      ...

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch, filter/drophttp] # do not forget to add processor otherwise it will not work
        exporters: [otlp]

  processors:
    filter/drophttp:
      error_mode: ignore
      traces:
        span:
          - attributes["http.url"] == "/actuator/health"
          - attributes["url.path"] == "/actuator/health"

I would be appreciated to myself if I would find this comment hours before. 😄

[Kortex]

I did the same but I ended up with orphaned spans. How are you dealing with those?

[shameekagarwal]

my architecture - app -> otel agent (auto instrumentation) -> otel collector -> observability backend for now, i wanted to filter out swagger and health check calls (caused due to k8s readiness probe)

solution 1 - filter processor. issue - resulted in orphaned spans
solution 2 - tail sampling processor. till now, looks like it works

@Kortex, does tail sampling not work?

[Kortex]

@shameekagarwal I did not try out tail sampling.

Would it be possible to share your configuration?

[shameekagarwal]

i linked my comment, where the code is in a collapsible....expand solution 2

[apischan]

So I decided to abandon the filter/drophttp processor and after the hours of debugging I've finally end up with the following configuration (thanks to @shameekagarwal):

  processors:
    tail_sampling:
      policies: [
        {
          name: filter_http_url,
          type: string_attribute,
          string_attribute: {
            key: http.url,
            values: [ /actuator/health, /swagger-ui.*, /v3/api-docs.*, /favicon.ico ],
            enabled_regex_matching: true,
            invert_match: true
          }
        },
        {
          name: filter_url_path,
          type: string_attribute,
          string_attribute: {
            key: url.path,
            values: [ /actuator/health, /swagger-ui.*, /v3/api-docs.*, /favicon.ico ],
            enabled_regex_matching: true,
            invert_match: true
          }
        }
      ]

NOTE: there are two of them because in your project can be enabled both spring-web and spring-webmvc tracing and they are using two different keys containing URL http.url and http.path respectively. (personally I decided to disable spring-webmvc tracing)

otel:
  instrumentation:
    spring-web:
      enabled: true
    spring-webmvc:
      enabled: true

Not sure why @shameekagarwal has key http.route, but in my case it is not (Spring Boot 3.2.2; Opentelemetry 1.35.0).
Anyway the good thing that everything is working now. :)

[EvertonSA]

one comment here, tail sampling works absolutely fantastic for almost all issues here (related to actuator) but I have a SSE stream that can be open for more than 1 hour. Tail sampling can't keep up with such long trace.

[jack-berg]

Re-opening until lingering issues are resolved which prevent declarative config from being used with the otel java agent: open-telemetry/opentelemetry-java-contrib#1440 (comment)

[jack-berg]

This is resolved in 2.9.0, which fixes support for declarative config.

See java agent with declarative configuration example for an end to end demonstration of this:

• a spring boot app
• using the otel java agent
• using declarative config
• packaging the rule based routing sampler as an extension
• configuring the sampler to be the parent based sampler, with the root set to the rule based sampler excluding certain routes

[jiten686]

@jack-berg Could provide an example how to use it. What I did is

1. Download the open telemetry java agent -2.9.0 jar.

2. Run my application with following script.
   -v /home/user/otel-javaagent-test:/opt/otel-javaagent -e JAVA_TOOL_OPTIONS='-javaagent:/opt/otel-javaagent/opentelemetry-javaagent.jar -Dotel.javaagent.configuration-file=/opt/otel-javaagent/otel-agent.properties -Dotel.javaagent.extensions=/opt/otel-javaagent/opentelemetry-samplers-1.40.0-alpha.jar' \

3. otel-agent.properties has following properties.

otel.service.name=service1
otel.propagators=tracecontext,baggage
otel.instrumentation.svc-web.enabled=true
otel.instrumentation.apache-httpclient.enabled=true
otel.instrumentation.okhttp.enabled=true
otel.exporter.otlp.protocol=grpc
otel.exporter.otlp.endpoint=http://**.***.**.**:****
otel.metrics.exporter=none
otel.logs.exporter=none
**otel.experimental.config.file=/opt/otel-javaagent/sdk-config.yml

4. sdk-config file is also attached in txt format sdk-config.txt

I am not seeing any traces for any APIs.

[trask]

hi @jiten686, are you able to get the example in the examples repo to work? https://github.com/open-telemetry/opentelemetry-java-examples/tree/main/javaagent#declarative-configuration

[jiten686]

Hi @trask , Oh I missed it. Now It is working. Thank you.
I have a question. Previously, I was using an older version of the Java agent and running it along with an agent-properties file containing the following properties.

otel.service.name=service1
otel.propagators=tracecontext,baggage
otel.instrumentation.svc-web.enabled=true
otel.instrumentation.apache-httpclient.enabled=true
otel.instrumentation.okhttp.enabled=true
otel.exporter.otlp.protocol=grpc
otel.exporter.otlp.endpoint=http://**.***.**.**:****
otel.metrics.exporter=none
otel.logs.exporter=none

However, as mention in above example, we need to use an sdk-config.yaml file to specify rules under rule_based_routing. Is there a way to keep all these configurations in the properties file instead of switching to sdk-config.yaml? If so, how can these configurations be added?"

[trask]

Is there a way to keep all these configurations in the properties file instead of switching to sdk-config.yaml?

no, you'll need to migrate those to the equivalent yaml configuration

[sergeykad]

Is it possible to configure these exclusions using environment variables?
I am using OpenTelemetry Operator for Kubernetes and it does not support declarative configuration.

open-telemetry/opentelemetry-operator#3392

[baole229801]

Hi @jiten686 , what you have changed to make it works? I config the same as you and traces disappeared for all APIs.

[jiten686]

@baole229801 Instead of using agent properties file, add the all those configuration in sdk-config.yaml file and set env variable as -e OTEL_EXPERIMENTAL_CONFIG_FILE=path/to/sdk-config.yaml.