net: LookupIP("doesnotexist.domain") returns "server misbehaving" when resolv.conf contains search lists

[mbenkmann]

The following test program returns "lookup doesnotexist.domain: no such host" when run with https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz but returns "lookup doesnotexist.domain on 172.16.2.203:53: server misbehaving" with https://storage.googleapis.com/golang/go1.5.linux-amd64.tar.gz and https://storage.googleapis.com/golang/go1.5.1.linux-amd64.tar.gz

package main

import "net"
import "fmt"

func main() {
  _, err := net.LookupIP("doesnotexist.domain")
  fmt.Println(err)
}

When I change resolv.conf to use nameserver 8.8.8.8 the output is correct. Apparently something has changed in Go 1.5 that prevents it from understanding the reply from our internal DNS server.

nslookup does not have a problem:

> nslookup doesnotexist.domain 172.16.2.203
Server:         172.16.2.203
Address:        172.16.2.203#53

** server can't find doesnotexist.domain: NXDOMAIN

> nslookup doesnotexist.domain 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

** server can't find doesnotexist.domain: NXDOMAIN

dig also has no problem

> dig @172.16.2.203 doesnotexist.domain
; <<>> DiG 9.9.5-3ubuntu0.4-Ubuntu <<>> @172.16.2.203 doesnotexist.domain
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 37809
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;doesnotexist.domain.               IN      A

;; AUTHORITY SECTION:
.                   1200    IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2015092200 1800 900 604800 86400

;; Query time: 2 msec
;; SERVER: 172.16.2.203#53(172.16.2.203)
;; WHEN: Tue Sep 22 13:13:11 CEST 2015
;; MSG SIZE  rcvd: 123

[bradfitz]

Without more information or a way to reproduce it we won't be able to do much.

Can you capture the network traffic when you run the Go program, or dig?

From that we should be able to write a test, once we see what's arriving on the wire from your DNS server.

[mbenkmann]

If you can tell me how to make a local copy of Go's net package work, I can dump out the relevant data structures and maybe even find the issue myself. Unfortunately just copying /opt/go/src/net to my local directory and changing import "net" to import "../net" does not work. It says "imports internal/singleflight: use of internal package not allowed". Copying /opt/go/src/internal, too, does not help. Copying "net" and "internal" to $GOPATH/src does not work (without changing import "net") either. They are simply ignored and the global system version keeps being used.

[bradfitz]

You won't be able to do that. Just build Go from source and modify the net package.

Run "./src/make.bash" from the "src" directory to recompile Go. It takes 30-60 seconds to build everything.

[mbenkmann]

I've found the problem. My resolv.conf has a line "search foo bar" for 2 subdomains foo and bar. For whatever reason queries to anything.bar are answered with error code 2 SERVFAIL. LookupIP("doesnotexist.domain") apparently uses the search field in resolv.conf and queries "doesnotexist.domain", "doesnotexist.domain.foo" and "doesnotexist.domain.bar" in sequence and reports the LAST ERROR CODE returned, which happens to be SERVFAIL. If the order in resolv.conf is changed to "search bar foo", LookupIP reports the NXDOMAIN error (no such host).

While it is apparent that the DNS server is misbehaving, the current LookupIP() implementation is suboptimal, because:

• the error code returned depends on the ordering in the search line of resolv.conf
• the behaviour of Go 1.5 is different from Go 1.4
• the error code reported does not relate to the actual query, which is not what the programmer expects, especially not in a case where the host queried contains a ".". I'm not even sure if it is proper to append the search subdomains to names that contain a ".".

In any case, I would change LookupIP() so that if the DNS server returns different error codes for the different queries made due to to "search", LookupIP() should always return the error code returned for the query with the actual argument passed to LookupIP(). This is the only query that is guaranteed to happen and therefore the only code that can be expected to be consistent.

[bradfitz]

the behaviour of Go 1.5 is different from Go 1.4

Go 1.5 uses Go by default for DNS lookups, and only falls back to libc's resolver for special cases. See https://golang.org/doc/go1.5#net for details.

So, it's not surprising that the Go DNS resolver has some rough edges. It's being exercised a lot more than it has in the past.

/cc @mikioh

[mikioh]

FWIW, IIRC, troubleshooting tools such as dig and/or drill don't use search list for super (not sub) domains in resolv.conf by default.

Looks like you have some idea to LookupIP. I'm not the original designer of LookupIP but I guess he wanted to make LookupIP easier for name-to-address mapping traversal. If you think of having a new API (sorry, we cannot change the behavior of LookupIP because it works as a stub resolver for helping Dial) for some purpose, please follow the procedure: https://github.com/golang/proposal#readme

Thank you.

[bradfitz]

Why did you close this bug?

[mikioh]

PS: If you need more low-level control for DNS, there are external packages. For example, http://godoc.org/github.com/miekg/dns.

[mikioh]

Just hands slipped.

[bradfitz]

I don't think @mbenkmann wanted low-level control of DNS. I interpreted this bug as Go's native DNS resolver just misbehaving compared to libc.