Skip to content

Commit

Permalink
call libc::res_init() in response to DNS failures
Browse files Browse the repository at this point in the history
Go's DNS resolution often defers to the libc implementation, and glibc's
resolver has a serious bug: https://sourceware.org/bugzilla/show_bug.cgi?id=984
It will cache the contents of /etc/resolv.conf, which can put the client
in a state where all DNS requests fail forever after a network change.
The conditions where Go calls into libc are complicated and
platform-specific, and the resolver cache involves thread-local state,
so repros tend to be inconsistent. But when you hit this on your laptop
on the subway or whatever, the effect is that everything is broken until
you restart the process.

One way to fix this would be to force using the pure-Go resolver
(net.DefaultResolver.PreferGo = true), which refreshes /etc/resolv.conf
every 5 seconds. I'm wary of doing that, because the Go devs went
through an enormous amount of trouble to enable cgo fallback, for
various platform- and environment-specific reasons. See all the comments
in net/conf.go::initConfVal() and net/conf.go::hostLookupOrder() in the
standard library.

Instead, we're trying the same workaround that the Rust standard library
chose, where we call libc::res_init() after DNS failures. See
rust-lang/rust#41570. The downside here is
that we have to remember to do this after we make network calls, and
that we have to use cgo in the build, but the upside is that it should
never break a DNS environment that was working before.
  • Loading branch information
oconnor663 committed Jul 19, 2017
1 parent 475bb3b commit 2deb33e
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions go/libkb/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import (
"time"

"github.com/keybase/go-framed-msgpack-rpc/rpc"
"github.com/keybase/go-framed-msgpack-rpc/rpc/resinit"

"h12.me/socks"
)
Expand Down Expand Up @@ -146,6 +147,13 @@ func NewClient(e *Env, config *ClientConfig, needCookie bool) *Client {
xprt.Dial = func(network, addr string) (c net.Conn, err error) {
c, err = net.Dial(network, addr)
if err != nil {
// If we get a DNS error, it could be because glibc has cached an
// old version of /etc/resolv.conf. The res_init() libc function
// busts that cache and keeps us from getting stuck in a state
// where DNS requests keep failing even though the network is up.
// This is similar to what the Rust standard library does:
// https://github.com/rust-lang/rust/blob/028569ab1b/src/libstd/sys_common/net.rs#L186-L190
resinit.ResInitIfDNSError(err)
return c, err
}
if err = rpc.DisableSigPipe(c); err != nil {
Expand All @@ -159,6 +167,7 @@ func NewClient(e *Env, config *ClientConfig, needCookie bool) *Client {
xprt.TLSClientConfig = &tls.Config{RootCAs: config.RootCAs}
}
if e.GetTorMode().Enabled() {
// TODO: should we call res_init on DNS errors here as well?
dialSocksProxy := socks.DialSocksProxy(socks.SOCKS5, e.GetTorProxy())
xprt.Dial = dialSocksProxy
} else {
Expand Down

0 comments on commit 2deb33e

Please sign in to comment.