call libc::res_init() in response to DNS failures #118
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Go's DNS resolution often defers to the libc implementation, and glibc's
resolver has a serious bug: https://sourceware.org/bugzilla/show_bug.cgi?id=984
It will cache the contents of /etc/resolv.conf, which can put the client
in a state where all DNS requests fail forever after a network change.
The conditions where Go calls into libc are complicated and
platform-specific, and the resolver cache involves thread-local state,
so repros tend to be inconsistent. But when you hit this on your laptop
on the subway or whatever, the effect is that everything is broken until
you restart the process.
One way to fix this would be to force using the pure-Go resolver
(net.DefaultResolver.PreferGo = true), which refreshes /etc/resolv.conf
every 5 seconds. I'm wary of doing that, because the Go devs went
through an enormous amount of trouble to enable cgo fallback, for
various platform- and environment-specific reasons. See all the comments
in net/conf.go::initConfVal() and net/conf.go::hostLookupOrder() in the
standard library.
Instead, we're trying the same workaround that the Rust standard library
chose, where we call libc::res_init() after DNS failures. See
rust-lang/rust#41570. The downside here is
that we have to remember to do this after we make network calls, and
that we have to use cgo in the build, but the upside is that it should
never break a DNS environment that was working before.
r? @maxtaco @mmaxim