helmchart - jaeger-collector & jaeger-query can't reach elasticsearch #5789

AliAyvanaky · 2024-07-30T13:17:58Z

AliAyvanaky
Jul 30, 2024

Hello Guys,

I installed jaeger with helmchart (version v3.1.2).
I just enabled elasticsearch as storage and didn't any changes.

provisionDataStore:
  cassandra: false
  elasticsearch: true
  kafka: false

and

storage:
  # allowed values (cassandra, elasticsearch, grpc-plugin, badger, memory)
  type: elasticsearch

jaeger-collector & jaeger-query pods stuck in CrashLoopBackOff status.
the logs are:
jaeger-collector:

2024/07/30 13:09:29 maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
2024-07-30T13:09:29.113233955Z 2024/07/30 13:09:29 application version: git-commit=b620f0eaefa5f17ec4bd01195149f158023a4e8e, git-version=v1.53.0, build-date=2024-01-08T18:05:40Z
2024-07-30T13:09:29.116530385Z {"level":"info","ts":1722344969.1163018,"caller":"flags/service.go:119","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1722344969.1163962,"caller":"flags/service.go:125","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
2024-07-30T13:09:29.116943866Z {"level":"info","ts":1722344969.1168597,"caller":"flags/admin.go:130","msg":"Mounting health check on admin server","route":"/"}
2024-07-30T13:09:29.116987449Z {"level":"info","ts":1722344969.1169112,"caller":"flags/admin.go:144","msg":"Starting admin HTTP server","http-addr":":14269"}
2024-07-30T13:09:29.117008969Z {"level":"info","ts":1722344969.1169329,"caller":"flags/admin.go:122","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
2024-07-30T13:09:34.631383299Z {"level":"fatal","ts":1722344974.6309576,"caller":"collector/main.go:82","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: Head \"http://elasticsearch-master:9200\": dial tcp: lookup elasticsearch-master on 10.43.0.10:53: no such host: no Elasticsearch node available","stacktrace":"main.main.func1\n\tgithub.com/jaegertracing/jaeger/cmd/collector/main.go:82\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.8.0/command.go:983\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1039\nmain.main\n\tgithub.com/jaegertracing/jaeger/cmd/collector/main.go:155\nruntime.main\n\truntime/proc.go:267"}

jaeger-query:

2024/07/30 13:09:20 maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
2024-07-30T13:09:20.140075099Z 2024/07/30 13:09:20 application version: git-commit=b620f0eaefa5f17ec4bd01195149f158023a4e8e, git-version=v1.53.0, build-date=2024-01-08T18:05:40Z
2024-07-30T13:09:20.140794972Z {"level":"info","ts":1722344960.1406796,"caller":"flags/service.go:119","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1722344960.1407242,"caller":"flags/service.go:125","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
2024-07-30T13:09:20.141036103Z {"level":"info","ts":1722344960.1409767,"caller":"flags/admin.go:130","msg":"Mounting health check on admin server","route":"/"}
2024-07-30T13:09:20.141117595Z {"level":"info","ts":1722344960.141027,"caller":"flags/admin.go:144","msg":"Starting admin HTTP server","http-addr":":16687"}
2024-07-30T13:09:20.141149564Z {"level":"info","ts":1722344960.1410472,"caller":"flags/admin.go:122","msg":"Admin server started","http.host-port":"[::]:16687","health-status":"unavailable"}
2024-07-30T13:09:25.614650964Z {"level":"fatal","ts":1722344965.6144323,"caller":"query/main.go:93","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: Head \"http://elasticsearch-master:9200\": dial tcp: lookup elasticsearch-master on 10.43.0.10:53: no such host: no Elasticsearch node available","stacktrace":"main.main.func1\n\tgithub.com/jaegertracing/jaeger/cmd/query/main.go:93\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.8.0/command.go:983\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.8.0/command.go:1039\nmain.main\n\tgithub.com/jaegertracing/jaeger/cmd/query/main.go:159\nruntime.main\n\truntime/proc.go:267"}

 kubectl get all -n jaeger-2
NAME                                      READY   STATUS             RESTARTS      AGE
pod/jaeger-agent-8vwdp                    1/1     Running            0             12m
pod/jaeger-agent-nqbwj                    1/1     Running            0             12m
pod/jaeger-collector-b8648649d-z7rph      0/1     CrashLoopBackOff   8 (34s ago)   12m
pod/jaeger-elasticsearch-coordinating-0   1/1     Running            0             12m
pod/jaeger-elasticsearch-coordinating-1   1/1     Running            0             12m
pod/jaeger-elasticsearch-data-0           1/1     Running            0             12m
pod/jaeger-elasticsearch-data-1           1/1     Running            0             12m
pod/jaeger-elasticsearch-ingest-0         1/1     Running            0             12m
pod/jaeger-elasticsearch-ingest-1         1/1     Running            0             12m
pod/jaeger-elasticsearch-master-0         1/1     Running            0             12m
pod/jaeger-elasticsearch-master-1         1/1     Running            0             12m
pod/jaeger-query-846bf8d6c-t4dft          1/2     CrashLoopBackOff   8 (36s ago)   12m

NAME                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                         AGE
service/jaeger-agent                           ClusterIP   10.43.218.13    <none>        5775/UDP,6831/UDP,6832/UDP,5778/TCP,14271/TCP   12m
service/jaeger-collector                       ClusterIP   10.43.50.203    <none>        14250/TCP,14268/TCP,14269/TCP                   12m
ali@sds-lin003:~/helm/jaeger/v3.1.2$ kubectl get all -n jaeger-2
NAME                                      READY   STATUS             RESTARTS        AGE
pod/jaeger-agent-8vwdp                    1/1     Running            0               21m
pod/jaeger-agent-nqbwj                    1/1     Running            0               21m
pod/jaeger-collector-b8648649d-z7rph      0/1     CrashLoopBackOff   9 (4m13s ago)   21m
pod/jaeger-elasticsearch-coordinating-0   1/1     Running            0               21m
pod/jaeger-elasticsearch-coordinating-1   1/1     Running            0               21m
pod/jaeger-elasticsearch-data-0           1/1     Running            0               21m
pod/jaeger-elasticsearch-data-1           1/1     Running            0               21m
pod/jaeger-elasticsearch-ingest-0         1/1     Running            0               21m
pod/jaeger-elasticsearch-ingest-1         1/1     Running            0               21m
pod/jaeger-elasticsearch-master-0         1/1     Running            0               21m
pod/jaeger-elasticsearch-master-1         1/1     Running            0               21m
pod/jaeger-query-846bf8d6c-t4dft          1/2     CrashLoopBackOff   9 (4m22s ago)   21m

NAME                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                         AGE
service/jaeger-agent                           ClusterIP   10.43.218.13    <none>        5775/UDP,6831/UDP,6832/UDP,5778/TCP,14271/TCP   21m
service/jaeger-collector                       ClusterIP   10.43.50.203    <none>        14250/TCP,14268/TCP,14269/TCP                   21m
service/jaeger-elasticsearch                   ClusterIP   10.43.226.200   <none>        9200/TCP,9300/TCP                               21m
service/jaeger-elasticsearch-coordinating-hl   ClusterIP   None            <none>        9200/TCP,9300/TCP                               21m
service/jaeger-elasticsearch-data-hl           ClusterIP   None            <none>        9200/TCP,9300/TCP                               21m
service/jaeger-elasticsearch-ingest-hl         ClusterIP   None            <none>        9200/TCP,9300/TCP                               21m
service/jaeger-elasticsearch-master-hl         ClusterIP   None            <none>        9200/TCP,9300/TCP                               21m
service/jaeger-query                           ClusterIP   10.43.16.50     <none>        80/TCP,16685/TCP,16687/TCP                      21m

NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/jaeger-agent   2         2         2       2            2           <none>          21m

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jaeger-collector   0/1     1            0           21m
deployment.apps/jaeger-query       0/1     1            0           21m

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/jaeger-collector-b8648649d   1         1         0       21m
replicaset.apps/jaeger-query-846bf8d6c       1         1         0       21m

NAME                                                 READY   AGE
statefulset.apps/jaeger-elasticsearch-coordinating   2/2     21m
statefulset.apps/jaeger-elasticsearch-data           2/2     21m
statefulset.apps/jaeger-elasticsearch-ingest         2/2     21m
statefulset.apps/jaeger-elasticsearch-master         2/2     21m

As you see here, query and collector dont have any endpoint:

 kubectl get ep -n jaeger-2
NAME                                   ENDPOINTS                                                     AGE
jaeger-agent                           10.42.1.24:6832,10.42.2.33:6832,10.42.1.24:5775 + 7 more...   24m
jaeger-collector                                                                                     24m
jaeger-elasticsearch                   10.42.1.25:9200,10.42.2.36:9200,10.42.1.25:9300 + 1 more...   24m
jaeger-elasticsearch-coordinating-hl   10.42.1.25:9200,10.42.2.36:9200,10.42.1.25:9300 + 1 more...   24m
jaeger-elasticsearch-data-hl           10.42.1.27:9200,10.42.2.38:9200,10.42.1.27:9300 + 1 more...   24m
jaeger-elasticsearch-ingest-hl         10.42.1.26:9200,10.42.2.37:9200,10.42.1.26:9300 + 1 more...   24m
jaeger-elasticsearch-master-hl         10.42.1.28:9200,10.42.2.39:9200,10.42.1.28:9300 + 1 more...   24m
jaeger-query                                                                                         24m

Do you have this issues?
Could you please support me?

@AliAyvanaky · 2024-07-30T13:26:13Z

dosubot[bot]
bot Jul 30, 2024

Hey there, @AliAyvanaky! I'm a bot designed to help you solve bugs, answer questions, and become a contributor. I'll do my best to assist you while we wait for a human maintainer.

To resolve the issue where the jaeger-collector and jaeger-query pods cannot reach the Elasticsearch service, you should verify and configure the following settings in your Jaeger configuration:

Elasticsearch Server URLs: Ensure that the server_urls parameter is correctly set to the URL of your Elasticsearch service. The default value is http://127.0.0.1:9200, but it should be updated to match your actual Elasticsearch service URL.
```
extensions:
  jaeger_storage:
    elasticsearch:
      es_main:
        server_urls: http://elasticsearch-master:9200
      es_archive:
        server_urls: http://elasticsearch-master:9200
```

Network Connectivity: Ensure that there are no network policies or firewall rules blocking the communication between the jaeger-collector/jaeger-query pods and the Elasticsearch service. You can create a Kubernetes NetworkPolicy to allow traffic to Elasticsearch:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-jaeger-to-elasticsearch
  namespace: jaeger-2
spec:
  podSelector:
    matchLabels:
      app: jaeger
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: elasticsearch
    ports:
    - protocol: TCP
      port: 9200  # Default port for Elasticsearch HTTP
    - protocol: TCP
      port: 9300  # Default port for Elasticsearch transport

DNS Resolution: Ensure that the DNS resolution for elasticsearch-master is correctly set up in your Kubernetes cluster. You can check the DNS resolution by running:
```
kubectl exec -it <jaeger-collector-pod> -- nslookup elasticsearch-master
```
Health Check Endpoint: Use the health check endpoint provided by the admin server to verify the service. The admin.http.host-port option specifies the host and port for the admin server, which includes health checks. The default value is :14270.
```
curl http://<admin-server-host>:14270/health
```

By verifying and configuring these settings, you should be able to resolve the connectivity issue between the jaeger-collector/jaeger-query pods and the Elasticsearch service ^[1]^[2]^[3]^[4].

To continue talking to Dosu, mention @dosu.

2 replies

AliAyvanaky Jul 30, 2024
Author

As you see in the logs, url is http://elasticsearch-master

yurishkuro Jul 30, 2024
Maintainer

what did this say?

kubectl exec -it <jaeger-collector-pod> -- nslookup elasticsearch-master

I don't see elasticsearch-master name in your output, only service/jaeger-elasticsearch-master-hl

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jaeger - Distributed Tracing Platform

helmchart - jaeger-collector & jaeger-query can't reach elasticsearch #5789

{{title}}

Replies: 1 comment 2 replies

{{title}}

{{title}}

{{title}}

Select a reply

Jaeger - Distributed Tracing Platform

helmchart - jaeger-collector & jaeger-query can't reach elasticsearch #5789

AliAyvanaky Jul 30, 2024

Replies: 1 comment · 2 replies

dosubot[bot] bot Jul 30, 2024

AliAyvanaky Jul 30, 2024 Author

yurishkuro Jul 30, 2024 Maintainer

AliAyvanaky
Jul 30, 2024

Replies: 1 comment 2 replies

dosubot[bot]
bot Jul 30, 2024

AliAyvanaky Jul 30, 2024
Author

yurishkuro Jul 30, 2024
Maintainer