Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

call libc::res_init() in response to DNS failures #118

Merged
merged 1 commit into from
Jul 19, 2017

Conversation

oconnor663
Copy link

Go's DNS resolution often defers to the libc implementation, and glibc's
resolver has a serious bug: https://sourceware.org/bugzilla/show_bug.cgi?id=984
It will cache the contents of /etc/resolv.conf, which can put the client
in a state where all DNS requests fail forever after a network change.
The conditions where Go calls into libc are complicated and
platform-specific, and the resolver cache involves thread-local state,
so repros tend to be inconsistent. But when you hit this on your laptop
on the subway or whatever, the effect is that everything is broken until
you restart the process.

One way to fix this would be to force using the pure-Go resolver
(net.DefaultResolver.PreferGo = true), which refreshes /etc/resolv.conf
every 5 seconds. I'm wary of doing that, because the Go devs went
through an enormous amount of trouble to enable cgo fallback, for
various platform- and environment-specific reasons. See all the comments
in net/conf.go::initConfVal() and net/conf.go::hostLookupOrder() in the
standard library.

Instead, we're trying the same workaround that the Rust standard library
chose, where we call libc::res_init() after DNS failures. See
rust-lang/rust#41570. The downside here is
that we have to remember to do this after we make network calls, and
that we have to use cgo in the build, but the upside is that it should
never break a DNS environment that was working before.

r? @maxtaco @mmaxim

@oconnor663
Copy link
Author

Vendored into client as keybase/client#7790.

// https://github.com/rust-lang/rust/blob/028569ab1b/src/libstd/sys_common/net.rs#L186-L190
// Note that we still propagate the error here, and we expect callers to
// retry.
resinit.ResInitIfDNSError(err)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inside if?

Go's DNS resolution often defers to the libc implementation, and glibc's
resolver has a serious bug: https://sourceware.org/bugzilla/show_bug.cgi?id=984
It will cache the contents of /etc/resolv.conf, which can put the client
in a state where all DNS requests fail forever after a network change.
The conditions where Go calls into libc are complicated and
platform-specific, and the resolver cache involves thread-local state,
so repros tend to be inconsistent. But when you hit this on your laptop
on the subway or whatever, the effect is that everything is broken until
you restart the process.

One way to fix this would be to force using the pure-Go resolver
(net.DefaultResolver.PreferGo = true), which refreshes /etc/resolv.conf
every 5 seconds. I'm wary of doing that, because the Go devs went
through an enormous amount of trouble to enable cgo fallback, for
various platform- and environment-specific reasons. See all the comments
in net/conf.go::initConfVal() and net/conf.go::hostLookupOrder() in the
standard library.

Instead, we're trying the same workaround that the Rust standard library
chose, where we call libc::res_init() after DNS failures. See
rust-lang/rust#41570. The downside here is
that we have to remember to do this after we make network calls, and
that we have to use cgo in the build, but the upside is that it should
never break a DNS environment that was working before.
@oconnor663 oconnor663 merged commit 9a3cbac into master Jul 19, 2017
@oconnor663 oconnor663 deleted the jack/CORE-4611/res_init branch July 19, 2017 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants