How to use jaeger-query-ui to use the data collected by open telemetry collector

[sathishsoundharajan]

Hi, I'm using otel-collector as agent and otel-collector as gateway to collect the traces and send it to elasticsearch backend. How can i use jaeger-ui to visulize this ?

Config Files

Agent Config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-agent-config
  labels:
    app: otel-collector-agent
data:
  otel-collector-agent.yml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      
    processors:
      batch: {}
      memory_limiter:
        check_interval: 5s
        limit_mib: 156
        spike_limit_mib: 48
      
    extensions:
      health_check: {}
      memory_ballast:
        size_mib: "78"
      
    exporters:
      logging: {}
      otlp:
        endpoint: otel-collector:4317
        tls:
          insecure: true

    service:
      extensions:
        - health_check
        - memory_ballast
      pipelines:
        traces:
          exporters:
            - otlp
            - logging
          processors:
            - memory_limiter
            - batch
          receivers:
            - otlp

Collector Config

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
data:
  otel-collector.yml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      
    exporters:
      logging: {}

      elasticsearch/trace:
        endpoints: [http://elasticsearch:9200]
        # traces_index: otlp-traces
        tls:
          insecure_skip_verify: true

    extensions:
      health_check: {}
      memory_ballast: {}

    processors:
      batch: {}
      memory_limiter:
        check_interval: 5s
        limit_mib: 156
        spike_limit_mib: 48

    service:
      extensions:
        - health_check
        - memory_ballast
      
      pipelines:
        traces:
          exporters:
            - logging
            - elasticsearch/trace
          receivers:
            - otlp
          processors:
            - memory_limiter
            - batch

I can see the traces like this collected in elastic search.

health status index                  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .geoip_databases       xb503ibjSdS6QxGtBnyScA   1   0         40            0     38.4mb         38.4mb
yellow open   traces-generic-default Q3nFwpg1S4m3GsXPPSZ85Q   1   1         74            0    173.5kb        173.5kb

Screenshot

[jkrech17]

You would need to send the traces to a Jaeger Exporter, which sends them to a Jaeger collector that is configured to write to Elasticsearch

[albertteoh]

@sathishsoundharajan, @jkrech17 is correct, you would need to have a jaeger exporter with an elasticsearch backing store.

If you need an example of how to send traces from an otel collector to Jaeger, you can run the docker-compose example from https://github.com/jaegertracing/jaeger/tree/main/docker-compose/monitor, which spins up a standalone deployment of otel collector with a jaeger all-in-one instance (not backed by elasticsearch).

[sathishsoundharajan]

But @albertteoh @jkrech17 @yurishkuro why can't we directly use the otlp traces collected ? My assumption was open-telemetry framework will collect vendor agnostic trace data which can persisted by the exporters collector supports and use any UI to visualise them ?

But if my understanding is wrong, can you answer these questions for clarification

If i have to run jaeger-collector on top of otel-collector to make this setup work ?

1. How should the setup be in kubernetes ? My initial design was all micro service will running a side-car with otel-collector acting as agent ( This is to make sure i can use the same agent to collect metrics also ) & run set of stateless otel-collector as Kubernetes deployment. But now if i have to add Jaeger-collector on top of it, should have to run set of jaeger-collector also as Kubernetes Deployment ?
2. So the trace data captured and persisted by Jaeger-collector, can be used to visualise using let's say Zipkin Or NewRelic ? ( Because still the trace collected by otel framework is vendor agnostic )
3. Is it good to run jaeger-all-in-one image as Kubernetes deployment and use horizontal scaling with it ? My trace volume will be huge as per my understanding.. ( OR ) running jaeger-collector image as Deployment and Jaeger-query-ui as separate deployment, jaeger-injester as separate Deployment for consuming from Kafka ?

[sathishsoundharajan]

Hi @albertteoh / @jkrech17 / @yurishkuro Your thoughts on this please ..

[yurishkuro]

elasticsearch/trace exporter is not some generic exporter, it is a vendor-specific format that only works with Elastic Observability product. It has nothing to do with either OTLP or Jaeger, it is not compatible with the Jaeger backend.

1- you are not required to use otel-collector since you can instruct your OTEL SDKs to export data via OTLP directly to Jaeger collector. If you do want otel-collector for other reasons (e.g. for forwarding metrics somewhere), then yes you will need a separate deployment of Jaeger collectors.

2- I am not sure where Zipkin or NewRelic are coming into picture if you're collecting data via OTEL SDK. Jaeger can accept data from Zipkin SDKs as well, but not NewRelic (if they use a proprietary format).

3- You can indeed run JAIO if you don't want to scale collectors and query separately, but JAIO will only support direct ingestion SDK->JAIO->storage, it does not support Kafka as a buffer.

[sathishsoundharajan]

@yurishkuro Thank you so much for response.

This is high level plan of what i'm planning to do:

1. Would like to collect traces using OTEL SDK and view them using Jaeger.
2. Use the OTEL SDK to collect metrics and use OTEL Agent to Pull metrics and push to OTEL Collector and from OTEL Collector to Prometheus.

Question:

1. Let's say in future, if i want to switch out of Jaeger and use different visualisation platform (Zipkin, Elastic APM, New Relic etc), My question is can i use the same data i have collected until now and just switch visualisation platform alone to Zipkin / Elastic APM / New Relic ? ). I believe this is not possible, because even though OTEL SDK API is vendor agnostic, the tool we used to visualise this trace data has different data structure, please correct if i'm wrong ?
2. To collect both trace & metrics guess I need OTEL Collector which going to act central gateway from there i can push to Jaeger arch or Prometheus Arch

[jkowall]

On the high level…. * This is fine, but you have to run a Jaeger collector which does the writes to the backend (ElasticSearch, Cassandra, or other backends). * OTEL SDK pushes metrics to the collector, which then can either be scraped by Prometheus or you can use remote write to send the data to Prometheus. I suggest that you also check out the Jaeger/Prometheus integration. Answers: * Unfortunately, each telemetry platform uses a different schema and format, so the data you have collected is useless. The good news is that with OpenTelemetry you can dual write. For example, if you are using Jaeger and want to test New Relic, you can simply add another exporter and pipeline and send it to both. Then shut one off. You will notice that support for Otel will be wildly different for each platform in terms of how they handle it.  We are working on Jaeger schema support inside the OpenSearch observability plugin, so it can read Jaeger and OpenSearch schemas for tracing. * Yes, that is correct. You'll want to use the collector for sampling, summarization, archiving, and other features of the collector.

…

On Tue, Oct 11, 2022 at 5:47 AM, Sathish < ***@***.*** > wrote: @ yurishkuro ( https://github.com/yurishkuro ) Thank you so much for response. This is high level plan of what i'm planning to do: * Would like to collect traces using OTEL SDK and view them using Jaeger. * Use the OTEL SDK to collect metrics and use *OTEL Agent* to *Pull* metrics and push to *OTEL Collector* and from *OTEL Collector to Prometheus*. Question: * Let's say in future, if i want to switch out of Jaeger and use different visualisation platform (Zipkin, Elastic APM, New Relic etc), My question is can i use the same data i have collected until now and just switch visualisation platform alone to Zipkin / Elastic APM / New Relic ? ). I believe this is not possible, because even though OTEL SDK API is vendor agnostic, the tool we used to visualise this trace data has different data structure, please correct if i'm wrong ? * To collect both trace & metrics guess I need OTEL Collector which going to act central gateway from there i can push to Jaeger arch or Prometheus Arch — Reply to this email directly, view it on GitHub ( #3952 (reply in thread) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAOGC3E5CZSXO3YJWEAAU5LWCUZRPANCNFSM6AAAAAAQ7W664E ). You are receiving this because you are subscribed to this thread. Message ID: <jaegertracing/jaeger/repo-discussions/3952/comments/3848846 @ github. com>

[sathishsoundharajan]

Thank you so much for your insights @jkowall .
Will take a look at Jaeger/Prometheus Integration.