OTTL Filter Processor doesn't seem to work for traces

[akshat-io]

Component(s)

processor/filter

What happened?

Description

As per the docs, the filter processor with the config to drop all HTTP spans, should have dropped all spans having 'GET' HTTP method. But they are still showing in Jaeger UI.

Steps to Reproduce

1. Clone the repo: https://github.com/akshat-io/spring-cloud-open-telemetry-filter-processor-issue
2. Use JDK 17 on your system.
3. Change directory into project: cd spring-cloud-open-telemetry-filter-processor-issue
4. Run command: ./mvnw clean install
5. After running 4th step successfully, run command: docker compose up
6. It should spin up both Java Applications, OpenTelemetry Collector and Jaeger.
7. Go to Jaeger UI: http://localhost:16686/search
8. Call Product API by opening below links in the browser:
   http://localhost:8080/product/100002
   http://localhost:8080/product/100003
   http://localhost:8080/product/100004
9. Jaeger UI at http://localhost:16686/search should be showing some traces.

Expected Result

No traces should be showing in Jaeger UI. As all spans with HTTP request method should have filtered out by the otel-config filter defined here.

Actual Result

The traces are still coming for HTTP spans.

Multiple conditions have been checked, including custom attributes, but the same issue is happening with all.

Collector version

0.81.0

Environment information

Environment

OS: Ubuntu 20.04
Compiler(if manually compiled): Did not compile collector manually, instead used docker image for collector version "0.81.0"
Container: Docker & Docker Compose
(Used docker to start the applications, collector and jaeger).

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  filter/drophttp:
    error_mode: ignore
    traces:
      span:
        - attributes["http.method"] != nil
  batch:

exporters:
  logging:
    loglevel: debug
  jaeger:
    endpoint: jaeger-service:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ logging, jaeger ]

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• processor/filter: @TylerHelmuth @boostchicken

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

You configured filter/drophttp but did not add it to the traces pipeline:

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ filter/drophttp, batch ]
      exporters: [ logging, jaeger ]

[krishnaexplore]

i've used in reverse for some other requirement (my_attribute), it only captures that span not entire trace.. what should i configure to capture entire trace

     filter/drop_others
        error_mode: ignore
        traces:
          span:
            - attributes["my_attribute"] == nil

[akshat-io]

@krishnaexplore I was going to ask the same thing as well. My use-case is also similar to you.

I want to get the complete trace, but only when a custom attribute productId is added to the parent (root) span of that trace.

My filter processor is as follows:

processors:
  filter/productidonly:
    error_mode: ignore
    traces:
      span:
        - attributes["productId"] == nil

I have modified the same repo to reproduce this:
The repo: https://github.com/akshat-io/spring-cloud-open-telemetry-filter-processor-issue

I have added a custom attribute productId to the span but only when the productId queried is 100004.

The filter is working and only when we hit below URL.
http://localhost:8080/product/100004

The trace comes for Product Service with this productId only, but not the other products like 100001, 100002 etc.

But now it doesn't contain lower level spans, which I need.

@TylerHelmuth What can be the approach to get the whole trace on the basis of some custom attribute, which is only added on the root / parent span and not to the child spans.

[TylerHelmuth]

The filterprocessor cannot be used to drop entire traces, that is a stateful operation that requires gathering all the spans in a trace. Use the tailsamplingprocessor for that.

• https://opentelemetry.io/docs/concepts/sampling/#tail-sampling

[krishnaexplore]

@TylerHelmuth thanks for response,

1. can i retain entire trace if one of the attribute of span matches? (or span name)?

2. I think we can't use tail-sampling, one of the application processes the request (jms) for minutes to hours

[TylerHelmuth]

can i retain entire trace if one of the attribute of span matches? (or span name)?

With the tail sampling processor you can, but not with the filter processor.

I think we can't use tail-sampling, one of the application processes the request (jms) for minutes to hours

That's a problem.

A solution could be to try to get the same identifying attributes on all spans in the trace you want to drop when you produce the spans.

[krishnaexplore]

Thanks @TylerHelmuth , seems that is the right approach

@akshat-io let me know if you need any help!!

[TylerHelmuth]

I am going to close this issue as dropping an entire trace is not something the filterprocessor can do. If you have issues about the tailsamplingprocessor please open a new issue.

[akshat-io]

@krishnaexplore
Let me know if you got it working. Because I am not able to filter the entire trace on the basis of the attribute coming in only root span.

Here is my otel config:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  tail_sampling:
    decision_wait: 5s
    num_traces: 2
    expected_new_traces_per_sec: 2
    policies:
      [
        {
          name: test-product-id,
          type: ottl_condition,
          ottl_condition: {
            error_mode: ignore,
            span: [
              "attributes[\"productId\"] == nil",
            ]
          }
        }
      ]
  batch:

exporters:
  logging:
    loglevel: debug
  jaeger:
    endpoint: jaeger-service:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ tail_sampling, batch ]
      exporters: [ logging, jaeger ]

The productId attribute only gets added to span, when the product with productId == 100004 is queried from the product service.

I have committed and pushed the code.

@krishnaexplore Let me know if you can help me with this.