Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

call libc::res_init() in response to DNS failures #7790

Merged
merged 2 commits into from
Jul 19, 2017

Conversation

oconnor663
Copy link
Contributor

@oconnor663 oconnor663 commented Jul 17, 2017

Go's DNS resolution often defers to the libc implementation, and glibc's
resolver has a serious bug: https://sourceware.org/bugzilla/show_bug.cgi?id=984
It will cache the contents of /etc/resolv.conf, which can put the client
in a state where all DNS requests fail forever after a network change.
The conditions where Go calls into libc are complicated and
platform-specific, and the resolver cache involves thread-local state,
so repros tend to be inconsistent. But when you hit this on your laptop
on the subway or whatever, the effect is that everything is broken until
you restart the process.

One way to fix this would be to force using the pure-Go resolver
(net.DefaultResolver.PreferGo = true), which refreshes /etc/resolv.conf
every 5 seconds. I'm wary of doing that, because the Go devs went
through an enormous amount of trouble to enable cgo fallback, for
various platform- and environment-specific reasons. See all the comments
in net/conf.go::initConfVal() and net/conf.go::hostLookupOrder() in the
standard library.

Instead, we're trying the same workaround that the Rust standard library
chose, where we call libc::res_init() after DNS failures. See
rust-lang/rust#41570. The downside here is
that we have to remember to do this after we make network calls, and
that we have to use cgo in the build, but the upside is that it should
never break a DNS environment that was working before.

r? @maxtaco @mmaxim

@oconnor663 oconnor663 force-pushed the jack/CORE-4611/res_init branch 5 times, most recently from 188c426 to 48ac673 Compare July 18, 2017 15:19
@oconnor663 oconnor663 changed the title example commit -- probably wont build on windows call libc::res_init() in response to DNS failures Jul 18, 2017
@oconnor663
Copy link
Contributor Author

The majority of this PR is code vendored from keybase/go-framed-msgpack-rpc#118, so we can review it in either place.

Go's DNS resolution often defers to the libc implementation, and glibc's
resolver has a serious bug: https://sourceware.org/bugzilla/show_bug.cgi?id=984
It will cache the contents of /etc/resolv.conf, which can put the client
in a state where all DNS requests fail forever after a network change.
The conditions where Go calls into libc are complicated and
platform-specific, and the resolver cache involves thread-local state,
so repros tend to be inconsistent. But when you hit this on your laptop
on the subway or whatever, the effect is that everything is broken until
you restart the process.

One way to fix this would be to force using the pure-Go resolver
(net.DefaultResolver.PreferGo = true), which refreshes /etc/resolv.conf
every 5 seconds. I'm wary of doing that, because the Go devs went
through an enormous amount of trouble to enable cgo fallback, for
various platform- and environment-specific reasons. See all the comments
in net/conf.go::initConfVal() and net/conf.go::hostLookupOrder() in the
standard library.

Instead, we're trying the same workaround that the Rust standard library
chose, where we call libc::res_init() after DNS failures. See
rust-lang/rust#41570. The downside here is
that we have to remember to do this after we make network calls, and
that we have to use cgo in the build, but the upside is that it should
never break a DNS environment that was working before.
@oconnor663 oconnor663 merged commit 2deb33e into master Jul 19, 2017
@oconnor663 oconnor663 deleted the jack/CORE-4611/res_init branch July 19, 2017 17:06
@oconnor663
Copy link
Contributor Author

Also revendored kbfs: keybase/kbfs#1076

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants