Skip Patterns while pushing traces

[rajatgupta5]

Hi team
I've tried out SigNoz with Spring boot micro-services which are already integrated with Zipkin. In the dashboard, I can see the /webjars and /swagger endpoints. It would be great to have some 'skip pattern' to avoid un-desired URLs. Thanks!

[ankitnayan]

@rajatgupta5 Great to see your Zipkin OpenTelemetry receiver working.

Definitely, an interesting feature. OpenTelemetry Collector has processors which are designed for such purposes. I will try this out soon and shall put this in our docs.

[ankitnayan]

We can learn more about different OpenTelemetry processors at Otel Processor Docs

Guide to include/exclude spans using opentelemetry processor [Link to Otel Docs]

Requirement

• Add a folder of otel-collector-configs at /deploy from where folks can choose for different use cases they have.
• Update docs in website

[velutha]

Hey @ankitnayan I did take a look at the issue following our slack conversation. I tried out using a filter processor with otel_collector. From the open telemetry collector docs,

It takes a pipeline type, of which only metrics is supported, followed by an action

1. Did you mean for only metrics to be filtered out at ingestion ?
2. It would not solve the issue raised since http endpoints would produce traces and filter processor would not help there. There is an issue asking for extending pipelines to have filter processor on a traces

From what I understand as of now, this fits at the processing time with a filter layer added on top ( I think this sub-system is intended to be extensible?) . However, I am not sure if this is something that needs to be done. Let me know what you think of this.

[ankitnayan]

@velutha Your understanding is correct. The include/exclude spans are just for allowing/disallowing mutations to those spans.

Do you want to extend that filter processor to drop spans also? Or we can build a separate processor for dropping spans. Where do you think is the best place for this functionality?

Also, as part of logic in dropping spans, we can remove all child spans of the specified span for now and build on more capabilities later. That shall handle the case of dropping completely endpoints like /swagger, etc.

We have a SigNoz's fork of OpenTelemetry Collector, where we can have our distribution and if we align with the design and speed of the official repo then we can create a PR for the same. The speed things move there would probably be different than ours so feel free to build things out in our repo.

I am also building an exporter that will be in SigNoz's fork of opentelemetry-collector.

[velutha]

Where do you think is the best place for this functionality

This is how I'm thinking about of advantages and trade-offs between the two approaches

Collection layer

• Extending functionality on otel_collector
• Filtering at edge
• Performance overhead in collector
• Complex logic on filters in difficult to incorporate

Processing layer

• Complex logic around filters should be easier to integrate
• Pave way for additional functionality with different mutations applicable on spans at a global level
• Additional processing added to flattener-processor (should be okay in reality)
• Time for development

I am also building an exporter that will be in SigNoz's fork of opentelemetry-collector.

It would be great if you could add your thoughts about adding it in collectors / exporters approach. Currently I am pro towards adding it at the processing layer. Better control and can work with existing client side collectors make it favourable.

[ankitnayan]

@velutha Great points!

As by me, the cons for adding to the processing layer would be

• Allow yaml like configuration reader in processors as we have at otel-collector
• We are planning to make Kafka non-compulsory for deployments at a small scale. This would leave such users without this feature.
• We are still to see powerful use cases of stream-processors. Right now, we are just using it for flattening data which can be done at ingestion time. We still need to check whether we are looking at Kafka as a queue in which case we might support other queues like RabbitMQ, SQS, etc, OR whether we need stream processing in which case we need to support alternatives like Kinesis. In either of the case, our current approach of using processors is not extendible.

Though ideally, the processing layer may seem like a better fit for this usecase. I would suggest continuing to develop this at the collector to make our project more adaptive to future changes.

What do you say?

[velutha]

Agree. I would like to work on the fork of otel collector rather than upstream. Let me take a look at the code and get back to you

[ashish0x02]

is there any update on this?