Helm: Wrong querier address in SingleBinary mode

[elchenberg]

Describe the bug

I deploy Loki in Kubernetes using Helm (chart version: 6.3.4) in SingleBinary mode.

These are the errors that I see in the Loki logs:

level=error ts=2024-04-24T04:39:49.988405567Z caller=http.go:200 org_id=fake msg="Error from client" err="websocket: close 1006 (abnormal closure): unexpected EOF"
level=error ts=2024-04-24T04:39:49.988604905Z caller=tail.go:231 component=querier org_id=fake msg="Error receiving response from grpc tail client" err="rpc error: code = Canceled desc = context canceled"
level=error ts=2024-04-24T04:39:50.011889677Z caller=tail.go:96 component=querier msg="Error reconnecting to disconnected ingesters" err="failed to connect with one or more ingester(s) during tailing: context canceled"
level=error ts=2024-04-24T04:39:50.011923911Z caller=tail.go:146 component=querier msg="Error reconnecting to ingesters" err="failed to connect with one or more ingester(s) during tailing: context canceled"

In the ConfigMap that there is a frontend.tail_proxy_url=http://loki-querier.default.svc.cluster.local:3100 in the config.yaml. When I manually change the configuration to frontend.tail_proxy_url="" in the cluster and restart Loki, the errors disappear.

After comparing how the loki.querySchedulerAddress is generated and how the loki.querierAddress is generated in the _helpers.tpl file I think this could be the solution:

-{{- if "loki.deployment.isDistributed "}}
+{{- $isDistributed := eq (include "loki.deployment.isDistributed" .) "true" -}}
+{{- if $isDistributed -}}

https://github.com/grafana/loki/blob/helm-loki-6.3.4/production/helm/loki/templates/_helpers.tpl#L958

To Reproduce

Run helm template with the following values.yaml and grep for tail_proxy_url:

# values.yaml
deploymentMode: SingleBinary
loki:
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  useTestSchema: true
singleBinary:
  replicas: 1
read:
  replicas: 0
backend:
  replicas: 0
write:
  replicas: 0

This is the output:

      tail_proxy_url: http://loki-querier.default.svc.cluster.local:3100

Expected behavior

No querier connection errors.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: Helm

Screenshots, Promtail config, or terminal output

See above.

[volver-13]

Hi @wrdls @JStickler @elchenberg

I'm running Loki as single binary and noticed the following in the logs:

loki-0 loki level=error ts=2024-08-23T15:41:19.131863731Z caller=tail.go:96 component=querier msg="Error reconnecting to disconnected ingesters" err="failed to connect with one or more ingester(s) during tailing: context canceled"
loki-0 loki level=error ts=2024-08-23T15:41:19.131908031Z caller=tail.go:146 component=querier msg="Error reconnecting to ingesters" err="failed to connect with one or more ingester(s) during tailing: context canceled"

Is this expected?