net: TestLookupPTR failures on windows-amd64-longtest builder

[bcmills]

#!watchflakes
post <- builder == "windows-amd64-longtest" && pkg == "net" && test == "TestLookupPTR"

It's not obvious to me whether the test is flaky, the builder has a flaky network connection, or the test is detecting something that is actually wrong.

CC @mikioh @bradfitz @ianlancetaylor (for net), @dmitshur, @toothrot, @cagedmantis, @andybons (for builders)

2020-03-26T17:48:20-918d4d4/windows-amd64-longtest
2020-03-06T14:41:22-2b0f481/windows-amd64-longtest
2020-01-09T18:00:06-8ac98e7/windows-amd64-longtest
2019-12-30T09:34:53-4b43700/windows-amd64-longtest
2019-11-13T08:07:51-bf5f641/windows-amd64-longtest

[bcmills]

There are at least two different failure modes:

--- FAIL: TestLookupPTR (3.92s)
    lookup_windows_test.go:167: skipping failed lookup 1.1.1.1 test: exit status 1: 
        Pinging one.one.one.one [1.1.1.1] with 32 bytes of data:
        Request timed out.
        
        Ping statistics for 1.1.1.1:
            Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),
    lookup_windows_test.go:172: different results 1.1.1.1:	exp:null	got:["one.one.one.one."]
FAIL
FAIL	net	66.044s

--- FAIL: TestLookupPTR (0.07s)
    lookup_windows_test.go:160: failed 1.1.1.1: lookup 1.1.1.1: dnsquery: DNS server failure.
    lookup_windows_test.go:163: no results
    lookup_windows_test.go:172: different results 1.1.1.1:	exp:[]	got:null
FAIL
FAIL	net	57.951s

[bcmills]

2021-02-19T00:41:28-b445d6e/windows-amd64-longtest
2021-02-19T00:40:49-427bd75/windows-amd64-longtest
2021-02-19T00:40:42-f6c4b4b/windows-amd64-longtest
2021-02-19T00:40:30-ac024a0/windows-amd64-longtest
2021-02-19T00:40:22-a3b97e7/windows-amd64-longtest
2021-02-19T00:40:16-985d087/windows-amd64-longtest
2021-02-19T00:40:05-95a44d2/windows-amd64-longtest
2021-02-19T00:40:00-0ca0551/windows-amd64-longtest
2021-01-28T02:19:23-376518d/windows-amd64-longtest
2021-01-27T21:11:22-00f2ff5/windows-amd64-longtest
2020-10-26T18:20:05-ae585ee/windows-amd64-longtest
2020-10-26T18:10:12-c5dea8f/windows-amd64-longtest

[bcmills]

Still ongoing at a low rate:

greplogs -l -e 'FAIL: TestLookupPTR .*(?:\n .*)*different results 1.1.1.1:\s+exp:null\s+got:\["one\.one\.one\.one\."\]' --since=2021-02-20
2021-11-18T02:16:39-353cb71/windows-amd64-longtest
2021-04-09T09:01:07-4d7d7a4/windows-amd64-longtest
2021-03-17T17:13:50-0bd308f/windows-amd64-longtest

greplogs -l -e 'FAIL: TestLookupPTR .*(?:\n .*)*DNS server failure' --since=2021-02-20
2022-06-13T19:06:49-4703546/windows-amd64-longtest
2022-06-13T16:53:11-7eeec1f/windows-amd64-longtest

[bcmills]

One more this week. The mode=go line in the failure log makes me wonder whether this recent spate of failures is related to #33097, although this most recent failure is after CL 413458.

@ianlancetaylor, @bradfitz: could you double-check whether something relating to the PreferGo and netdns=go changes in 1.19 could be causing an increase in TestLookupPTR failures?

greplogs -l -e 'FAIL: TestLookupPTR .*(?:\n .*)*DNS server failure' --since=2022-06-14
2022-06-22T23:27:17-2e773a3/windows-amd64-longtest

[ianlancetaylor]

Note that the mode=go appears on a likely unrelated failure of TestLookupDotsWithRemoteSource. It does not appear in a failure of TestLookupPTR. It's just that both tests happened to fail in that run.

[ianlancetaylor]

Most of the errors are of the form

--- FAIL: TestLookupPTR (3.90s)
    lookup_windows_test.go:167: skipping failed lookup 1.1.1.1 test: exit status 1: 
        Pinging one.one.one.one [1.1.1.1] with 32 bytes of data:
        Request timed out.
        
        Ping statistics for 1.1.1.1:
            Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),
    lookup_windows_test.go:172: different results 1.1.1.1:	exp:null	got:["one.one.one.one."]

The test actually runs the ping program. If the ping program fails, the test says skipping failed lookup. But it doesn't actually skip the test. All of those errors are flakes.

There are also some errors in which the DnsQuery Windows function fails with DNS_ERROR_RCODE_SERVER_FAILURE. The documentation for that error is "DNS server failure." I don't know what is going on there. As far as I can tell the code that is failing is unchanged since 1.18. This case only happens when using netdns=cgo, the default in 1.18 and earlier on Windows.

[gopherbot]

Change https://go.dev/cl/414634 mentions this issue: net: really skip Windows TestLookupPTR test if we say we are skipping it