net: revisit unconditional use of cgo lookups for darwin

[danp]

https://golang.org/cl/8945 changed to using the native Go stub resolver for most systems. Darwin was excluded, here, due to firewall warnings.

Is this still an issue?

On my 10.11 system with the firewall enabled this test program doesn't produce any warnings, even with cgo disabled:

package main

import (
    "fmt"
    "net"
)

func main() {
    ns, err := net.LookupHost("www.google.com")
    if err != nil {
        panic(err)
    }
    fmt.Println(ns)
}

% CGO_ENABLED=0 GODEBUG=netdns=2 go run main.go
go package net: built with netgo build tag; using Go's DNS resolver
go package net: hostLookupOrder(www.google.com) = files,dns
[216.58.192.164 2607:f8b0:4009:80e::2004]

(nothing pops up when running)

There are probably other reasons to conditionally exclude Darwin for now, such as if /etc/resolver config is used (#12524), but perhaps removing this blanket condition could be a start.

cc @mdempsky and @bradfitz since you authored that change.

[bradfitz]

The Darwin exclusion predates https://golang.org/cl/8945 I thought. Can you do some digging and investigate its history?

What do we gain by using Go's resolver by default on Mac? It seems like we'd need to do #12524 first as you mentioned otherwise we risk doing more harm than good.

[danp]

I did some digging!

Before https://golang.org/cl/8945 the only way to use the Go's resolver (for most platforms, not just darwin) was to build with netgo (and/or without cgo?). This caused the stubs to return false for completed and triggered use of the Go resolver.

I discovered along the way that android is also currently summarily excluded from using Go's resolver due to #10714, though it's done per request instead of per-conf-setup like darwin. In the CL for that, @minux called for the same to be done for iOS and then subsequently found the exclusion for darwin.

All unix platforms except for darwin and android now use Go's resolver unless weirdness is detected along here or here. I think we could gain more consistency across platforms if darwin were to join in with necessary safety checks.

Re #12524, that could be something we detect and fall back to cgo for initially. Once there is support for it in the Go resolver the check could be removed and Go's resolver could be used in that case.

At the very least if we decide not to pursue this it would be good to update the comment around the darwin exclusion to explain it's due to more than possible firewall warnings.

[minux]

Have you tested the pure Go resolver on older OS X systems?

[danp]

I only have immediate access to 10.11 and 10.10 systems, neither of which pop up anything when trying the test program in the description (which I've updated to show exactly how I ran it). Trying to see if any community members have access to older systems for testing. Worth noting that 10.10 would have been current at the time of https://golang.org/cl/8945.

I'm also unable to find anything when searching for this issue happening generally. @bradfitz, did you experience this firsthand? Is it possible it was this standard firewall warning that pops up when a process listens for outside connections, and maybe caused by something else?

[groob]

@dpiddy @bradfitz I don't think this a bug. I tested on 10.11 and the application firewall is not triggered either.

Having a "server" triggers the firewall on OS X, since that would offer an incoming connection. But DNS resolution is outgoing, so the firewall would ignore it.

[danp]

Any further thoughts on this?

I can try and put together ways to detect when we should not use Go's resolver if we want to move forward with removing the blanket condition.

[bradfitz]

@dpiddy, I can test on OS X 10.8, now that we have VM-based builders and ancient versions available.

[danp]

Great! Let me know what I can do to help.

[bradfitz]

I tried on OS X 10.8 with the firewall on in its most restrictive mode (block all incoming connections), and I saw no pop-up dialog using Go's DNS resolver. I set GODEBUG=netdns=go+2 and saw it use Go's DNS resolver and get the right answers.

[bradfitz]

I can try and put together ways to detect when we should not use Go's resolver if we want to move forward with removing the blanket condition.

Propose away.

[danp]

Hope to spend some good time on this soon. So far existence of /etc/resolvers is the first obvious cgo fallback condition.

What I'm more unsure of is this bit in the darwin resolver(5) man page:

However, client configurations are not limited to file storage. The implementation of the DNS multi-client search strategy may also locate client configuratins in other data sources, such as the System Configuration Database. Users of the DNS system should make no assumptions about the source of the configuration data.

I'm not familiar with what those other data sources might be or if we can detect their use.

Issues like docker/for-mac#19 suggest scutil can be used to discover DNS config in the System Configuration Database. Would that give enough hints on when cgo should be preferred? Would it work for all Go-supported OS X versions?

There might also be special names that should prefer cgo.

Any available insight on these things would be greatly appreciated!

And if it would help get things started to open a CL removing the blanket darwin exclusion but falling back to cgo if /etc/resolvers exists I can certainly do that.

[quentinmit]

scutil is a wrapper around the SystemConfiguration framework. scutil --dns prints the current resolver configuration. The default configuration is DNS from DHCP/manual, followed by mdns for a number of specific domains ("local" and the PTR domains).

It looks like Chromium uses the unexported symbol dns_configuration_copy to get the configuration: https://chromium.googlesource.com/chromium/src/+/b4aadc32a7fd6d42ac3cc9adbb20a8ee4a267572/net/dns/dns_config_watcher_mac.cc