call libc::res_init() in response to DNS failures

Go's DNS resolution often defers to the libc implementation, and glibc's
resolver has a serious bug: https://sourceware.org/bugzilla/show_bug.cgi?id=984
It will cache the contents of /etc/resolv.conf, which can put the client
in a state where all DNS requests fail forever after a network change.
The conditions where Go calls into libc are complicated and
platform-specific, and the resolver cache involves thread-local state,
so repros tend to be inconsistent. But when you hit this on your laptop
on the subway or whatever, the effect is that everything is broken until
you restart the process.

One way to fix this would be to force using the pure-Go resolver
(net.DefaultResolver.PreferGo = true), which refreshes /etc/resolv.conf
every 5 seconds. I'm wary of doing that, because the Go devs went
through an enormous amount of trouble to enable cgo fallback, for
various platform- and environment-specific reasons. See all the comments
in net/conf.go::initConfVal() and net/conf.go::hostLookupOrder() in the
standard library.

Instead, we're trying the same workaround that the Rust standard library
chose, where we call libc::res_init() after DNS failures. See
rust-lang/rust#41570. The downside here is
that we have to remember to do this after we make network calls, and
that we have to use cgo in the build, but the upside is that it should
never break a DNS environment that was working before.

rpc/connection.go

@@ -12,6 +12,7 @@ import (
 	"time"
 
 	"github.com/keybase/backoff"
+	"github.com/keybase/go-framed-msgpack-rpc/rpc/resinit"
 	"golang.org/x/net/context"
 )
 
@@ -73,6 +74,15 @@ func (t *connTransport) Dial(context.Context) (Transporter, error) {
 	var err error
 	t.conn, err = t.uri.Dial()
 	if err != nil {
+		// If we get a DNS error, it could be because glibc has cached an old
+		// version of /etc/resolv.conf. The res_init() libc function busts that
+		// cache and keeps us from getting stuck in a state where DNS requests
+		// keep failing even though the network is up. This is similar to what
+		// the Rust standard library does:
+		// https://github.com/rust-lang/rust/blob/028569ab1b/src/libstd/sys_common/net.rs#L186-L190
+		// Note that we still propagate the error here, and we expect callers
+		// to retry.
+		resinit.ResInitIfDNSError(err)
 		return nil, err
 	}
 

rpc/resinit/resinit.go

@@ -0,0 +1,20 @@
+package resinit
+
+import "net"
+
+// If we get a DNS error, it could be because glibc has cached an old version
+// of /etc/resolv.conf. The res_init() libc function busts that cache and keeps
+// us from getting stuck in a state where DNS requests keep failing even though
+// the network is up. This is similar to what the Rust standard library does:
+// https://github.com/rust-lang/rust/blob/028569ab1b/src/libstd/sys_common/net.rs#L186-L190
+func ResInitIfDNSError(err error) {
+	// There are two error cases we need to handle, a raw *net.DNSError, and
+	// one wrapped in a *net.OpError. Detect that second case, and unwrap it.
+	if opErr, isOpErr := err.(*net.OpError); isOpErr {
+		err = opErr.Err
+	}
+	if _, isDNSError := err.(*net.DNSError); isDNSError {
+		// defined per platform in resinit_*.go
+		resInit()
+	}
+}

rpc/resinit/resinit_android.go

@@ -0,0 +1,15 @@
+// Copyright 2015 Keybase, Inc. All rights reserved. Use of
+// this source code is governed by the included BSD license.
+
+// +build android
+
+package resinit
+
+// /* Bionic has res_init() but it's not in any header. Patterned off of: */
+// /* https://mail.gnome.org/archives/commits-list/2013-May/msg01329.html */
+// int res_init (void);
+import "C"
+
+func resInit() {
+	C.res_init()
+}

rpc/resinit/resinit_nix.go

@@ -0,0 +1,14 @@
+// Copyright 2015 Keybase, Inc. All rights reserved. Use of
+// this source code is governed by the included BSD license.
+
+// +build !windows,!android
+
+package resinit
+
+// #cgo LDFLAGS: -lresolv
+// #include<resolv.h>
+import "C"
+
+func resInit() {
+	C.res_init()
+}

rpc/resinit/resinit_windows.go

@@ -0,0 +1,10 @@
+// Copyright 2015 Keybase, Inc. All rights reserved. Use of
+// this source code is governed by the included BSD license.
+
+// +build windows
+
+package resinit
+
+func resInit() {
+	// no-op
+}