Loki exits when the configured jaeger is not present

[davkal]

I'd argue that Loki should not just exit when a configured jaeger is not present.

This came up when testing loki and jaeger together in the grafana's devenv docker-compose snippets: Loki would no longer start on its own, and I had to remove the env vars: grafana/grafana#20967

Here is the docker compose output:

loki_1         | Could not initialize jaeger tracer: lookup jaeger on 127.0.0.11:53: no such host

[yindia]

@cyriltovena Would like to work on this. Can this be assigned to me ? I also need small brief

[adityacs]

@cyriltovena @davkal The code here https://github.com/weaveworks/common/blob/master/tracing/tracing.go#L19 is doing os.Exit(1) which fails loki as well on jaeger not being available.

How should we see this for loki?

Currently we don't have any retry logic to connect to jaeger when it comes up. So, when we start loki without jaeger being present and later when jaegar comes up, loki won't connect to jaeger. May be we should add this logic in turn through this https://github.com/weaveworks/common/blob/master/tracing/tracing.go ?

OR

Don't disturb the weaveworks/common logic. Let it be the way it is. We can handle this from loki side to check if the configured jaeger agent is present and then call weaverworks/common logic to initialize jaeger?

[adityacs]

@cyriltovena Kindly let me know your thoughts on this

[cyriltovena]

I think we should change the common project to return an error and not os.Exit, this consumer of the API can decide what to do.