Monitor Tab Extremly Slow and Load Over 100MB Data in Total At Once

[laggage]

It takes over minutes to load data with over 70MB on monitor tab

Is there any approach to config the ui page data of Operations metrics?

[dosubot[bot]]

Hello, @laggage! I'm here to help you with any bugs, questions, or contributions. Let's work together to solve this issue.

To address performance issues with the Monitor tab in the Jaeger UI, especially when dealing with large data sets, you might consider the following approaches:

1. Optimize Elasticsearch Configurations: Adjust Elasticsearch settings to handle large data requests more efficiently. This includes configurations like es.bulk.actions, es.bulk.size, es.bulk.flush-interval, and es.bulk.workers to improve bulk processing. Additionally, manage index rollover frequency and adjust the number of shards and replicas to optimize performance for large datasets ^[1][2].

2. Review Jaeger Query Service Settings: Ensure that the Jaeger Query Service is configured to handle large data volumes effectively. This might involve increasing the maximum document count (es.max-doc-count) and adjusting timeouts (es.timeout) to prevent long-running queries from timing out ^[1][2].

3. Monitor Tab Architecture: Understand that the Monitor tab fetches and renders data using an HTTP JSON API, and the recommended way to retrieve RED metrics is via the jaeger.api_v2.metrics.MetricsQueryService gRPC endpoint. This feature is experimental and designed to help identify interesting traces without needing to know service or operation names upfront ^[3][4][5].

These steps can help improve the performance of the Monitor tab when dealing with large data volumes.

To continue talking to Dosu, mention @dosu.

[yurishkuro]

How many operations do you have in one service?

[laggage]

About 8000 operations

[laggage]

Some operations are just Virus, is there a way i can filter out the operation that only start with /api/** ?

[yurishkuro]

Right now we don't support any filtering capabilities. The page automatically filters by spans kind server, so it's odd that all these span names are server kind - they look more like internal spans.

You also seem to have a very slow connection if it takes 30sec to download 70Mb.

Paging the requests is possible if we first load all the endpoints and then iterate through them in batches, it's just not how the logic is currently implemented - it simply asks Prometheus for a given metric and gets whichever labels / time series are available.

[jkowall]

The way to handle this is to use Otel collector to filter out the data which is not relevant (or possibly whitelist the things which are relevant). https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md

From perplexity.... to filter out specific spans with OpenTelemetry (OTel), you can use the filter processor in the OpenTelemetry Collector. Here are some methods to achieve this:

Using the Filter Processor

The filter processor allows you to drop spans based on various criteria. It utilizes the OpenTelemetry Transformation Language (OTTL) to define conditions for filtering.

Basic Filtering

To filter out specific spans, you can use the following configuration:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.method"] == "GET"'
        - 'resource.attributes["service.name"] == "my-service"'

This configuration will drop spans that match either of the conditions:

1. Spans with the HTTP method "GET"
2. Spans from the service named "my-service"[1][3]

Filtering Based on Span Name

To filter spans based on their names:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - 'name == "unwanted-span-name"'

This will drop spans with the name "unwanted-span-name"[1]

Filtering Using Regular Expressions

You can use regular expressions for more flexible filtering:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - 'IsMatch(name, "^/api/v1/.*")'

This configuration drops spans whose names start with "/api/v1/"[3]

Filtering Root Spans

To filter only root spans:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - 'IsRootSpan() == true'

This will only drop root spans that match the condition[4]

Advanced Filtering

For more complex scenarios, you can combine multiple conditions:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - 'IsRootSpan() == true and attributes["http.method"] == "POST" and resource.attributes["service.name"] == "order-processing"'

This configuration drops root spans from the "order-processing" service that represent POST requests[4]

Considerations

1. Order of Operations: Include filters are always applied before exclude filters for any given filter processor instance[3].

2. Performance Impact: Complex filtering configurations may have a significant impact on collector performance. It's important to test and optimize your configurations[1].

3. Error Handling: The error_mode setting determines how the processor reacts to errors that occur while processing an OTTL condition. Set it to "ignore" to continue processing even if errors occur[5].

4. Orphaned Telemetry: Be cautious when filtering as it may lead to orphaned telemetry. For example, if you drop all span events for a span, the span itself will be left intact[5].

By using these filtering techniques, you can effectively control which spans are processed and forwarded by the OpenTelemetry Collector, allowing you to focus on the most relevant data for your observability needs.

Citations:
[1] https://opentelemetry.io/docs/collector/transforming-telemetry/
[2] https://www.alibabacloud.com/help/en/opentelemetry/use-cases/filter-the-specified-span-by-using-opentelemetry
[3] https://docs.splunk.com/observability/en/gdi/opentelemetry/components/filter-processor.html
[4] https://last9.io/blog/identify-root-spans-in-otel-collector/
[5] https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md

[laggage]

End up with filtering it out on the client side