Thanos ruler dies on startup if AM DNS can't be resolved

[OGKevin]

Thanos, Prometheus and Golang version used:

thanosio/thanos:v0.14.0

What happened:
Some details/backstory can be found here: prometheus/alertmanager#2372
TL;DR, thanos ruler dies at startup if it cannot revolve AM DNS.

What you expected to happen:
Ruler to boot up and at leats process rules.

How to reproduce it (as minimally and precisely as possible):

docker run -p10902:10902 thanosio/thanos:v0.14.0 rule --alertmanagers.url=dns+http://k3jhut933.messagebird.io:9093/alertmanager --query=mon.messagbird.io/query

Full logs to relevant components:

Logs

level=info ts=2020-09-18T07:32:40.602076815Z caller=main.go:151 msg="Tracing will be disabled"
level=info ts=2020-09-18T07:32:40.605531707Z caller=head.go:645 component=tsdb msg="Replaying WAL and on-disk memory mappable chunks if any, this may take a while"
level=info ts=2020-09-18T07:32:40.607499462Z caller=head.go:706 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2020-09-18T07:32:40.607541953Z caller=head.go:709 component=tsdb msg="WAL replay completed" duration=1.973454ms
level=info ts=2020-09-18T07:32:40.609255532Z caller=options.go:23 component=rules protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2020-09-18T07:32:40.609579143Z caller=grpc.go:92 component=rules service=gRPC/server component=rule msg="registering as gRPC StoreAPI and RulesAPI"
level=info ts=2020-09-18T07:32:40.610248039Z caller=rule.go:638 component=rules msg="no supported bucket was configured, uploads will be disabled"
level=info ts=2020-09-18T07:32:40.610285622Z caller=rule.go:641 component=rules msg="starting rule node"
level=info ts=2020-09-18T07:32:40.610460329Z caller=intrumentation.go:60 component=rules msg="changing probe status" status=healthy
level=info ts=2020-09-18T07:32:40.610495283Z caller=http.go:56 component=rules service=http/server component=rule msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2020-09-18T07:32:40.610988764Z caller=rule.go:791 component=rules msg="reload rule files" numFiles=0
level=info ts=2020-09-18T07:32:40.612768561Z caller=intrumentation.go:48 component=rules msg="changing probe status" status=ready
level=info ts=2020-09-18T07:32:40.612830648Z caller=grpc.go:119 component=rules service=gRPC/server component=rule msg="listening for serving gRPC" address=0.0.0.0:10901
level=info ts=2020-09-18T07:32:40.620800493Z caller=manager.go:888 component=rules msg="Stopping rule manager..."
level=info ts=2020-09-18T07:32:40.620844692Z caller=manager.go:898 component=rules msg="Rule manager stopped"
level=info ts=2020-09-18T07:32:40.620853668Z caller=manager.go:888 component=rules msg="Stopping rule manager..."
level=info ts=2020-09-18T07:32:40.620857918Z caller=manager.go:898 component=rules msg="Rule manager stopped"
level=warn ts=2020-09-18T07:32:40.620866143Z caller=intrumentation.go:54 component=rules msg="changing probe status" status=not-ready reason="lookup IP addresses \"k3jhut933.messagebird.io\": lookup k3jhut933.messagebird.io on 192.168.65.1:53: no such host"
level=info ts=2020-09-18T07:32:40.620883785Z caller=grpc.go:138 component=rules service=gRPC/server component=rule msg="gracefully stopping internal server"
level=info ts=2020-09-18T07:32:40.620983285Z caller=grpc.go:150 component=rules service=gRPC/server component=rule msg="internal server shutdown" err="lookup IP addresses \"k3jhut933.messagebird.io\": lookup k3jhut933.messagebird.io on 192.168.65.1:53: no such host"
level=warn ts=2020-09-18T07:32:40.621015491Z caller=intrumentation.go:54 component=rules msg="changing probe status" status=not-ready reason="lookup IP addresses \"k3jhut933.messagebird.io\": lookup k3jhut933.messagebird.io on 192.168.65.1:53: no such host"
level=info ts=2020-09-18T07:32:40.621064992Z caller=http.go:81 component=rules service=http/server component=rule msg="internal server shutdown" err="lookup IP addresses \"k3jhut933.messagebird.io\": lookup k3jhut933.messagebird.io on 192.168.65.1:53: no such host"
level=info ts=2020-09-18T07:32:40.621109322Z caller=intrumentation.go:66 component=rules msg="changing probe status" status=not-healthy reason="lookup IP addresses \"k3jhut933.messagebird.io\": lookup k3jhut933.messagebird.io on 192.168.65.1:53: no such host"
level=error ts=2020-09-18T07:32:40.621870546Z caller=main.go:211 err="lookup IP addresses \"k3jhut933.messagebird.io\": lookup k3jhut933.messagebird.io on 192.168.65.1:53: no such host\nrule command failed\nmain.main\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:211\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"

Anything else we need to know:
Could not reproduce on MacOS.

When passing --query.sd-dns-resolver=miekgdns, ruler does not die.

[bwplotka]

Hm, I think this might be expected as we cannot reach DNS server at all - it's not like response has no results.

However I think we can consider making this non crashing, wonder if we can built-in dead men switch 🤔

[OGKevin]

Hmm it looks like its not necessarily not being able to reach the DNS server.

miekgdns and the default DNS server have different behaviours when a host can not be resolved. The default DNS resolved returns a err no such host while the miekgdns returns a nil response and a nil error.

So what we could do is to instead of propagating the no such host error from the default DSN resolver to the top, we could check if it is such error and return nil as well. This way both choices have the same behaviour.

[OGKevin]

Im picking this up BTW 😊.