x/net/route: TestRouteMessage failures on freebsd-386 builders #35513

bcmills · 2019-11-11T21:00:28Z

--- FAIL: TestRouteMessage (0.00s)
    message_test.go:235: dst|gateway|ifp|ifa|brd (pmtu=0) (inet4 127.0.0.1) (link 2 <nil> <nil>) (link 2 lo0 <nil>) (inet4 127.0.0.1) (inet4 0.0.0.0)
    message_test.go:235: dst|gateway (pmtu=0) (inet4 127.0.0.1) (link 2 <nil> <nil>)
    message_test.go:223: {0 4 0 0 1160 3 <nil> [(inet6 0000:0000:0000:0000:0000:0000:0000:0001 0) <nil> <nil> <nil> (link 0 <nil> <nil>) (inet6 0000:0000:0000:0000:0000:0000:0000:0000 0) <nil> (inet6 0000:0000:0000:0000:0000:0000:0000:0000 0)] 0 []}: invalid address
FAIL
FAIL	golang.org/x/net/route	0.006s

2019-11-09T02:19:31-daa7c04/freebsd-386-11_2
2019-11-09T02:19:31-daa7c04/freebsd-386-12_0

CC @tklauser @mikioh @bradfitz

The text was updated successfully, but these errors were encountered:

bradfitz · 2019-11-11T21:48:44Z

Only 386? Always or a flake?

bcmills · 2019-11-11T22:18:32Z

Always or a flake?

Looks like a recurring flake, but getting a better answer to that question is part of my motivation for filing #35515.

bcmills · 2020-10-19T13:19:31Z

bcmills · 2021-02-02T16:04:03Z

bcmills · 2021-05-07T13:54:37Z

@golang/release: it seems like this test either needs an owner to fix it, or ought to be skipped on this builder. The freebsd-386-12_2 builder is currently providing little value for x/net because it's ~always red.

(Marking as release-blocker for Go 1.17 via #11811.)

heschi · 2021-05-13T18:16:06Z

Weekly check-in: this needs to be investigated before beta 1.

cc @neild

neild · 2021-05-14T01:03:56Z

I have not yet figured out what's going on here, but it's not 1.17 specific. The problem is easily reproducible with -count=1000 and 1.16.

heschi · 2021-05-18T19:30:04Z

Removing release-blocker, then.

bcmills · 2021-05-19T14:14:43Z

@heschi, I believe the freebsd-12_2-386 builder is new as of the Go 1.17 cycle.

Unless I am mistaken, from #11811 it follows that part of the process of setting up a new builder needs to include either fixing or adding skips for the existing tests that are already failing on that builder.

Since the builder is new for Go 1.17, I believe this issue should still be a 1.17 release-blocker.

bcmills · 2021-05-19T14:23:46Z

Note that the failure rate on the freebsd-12_2-386 builder is qualitatively, if not quantitatively, different from the failure rate on the previous 11_2 and current 11_4 builders:

I cannot explain why the qualitative appearance of the dashboard does not seem to match the more even distribution of failures in the logs downloaded by fetchlogs. That discrepancy may be worth investigating further, but I don't plan to pursue it at this time.

neild · 2021-05-19T23:38:02Z

The failing test reads routing information from an AF_ROUTE socket and parses it.

The information read from the socket does not match the expected format. The test passes most of the time, because most of the time the parser doesn't realize there's a problem.

You can easily reproduce this on the freebsd-386-11_2 builder with go test ./route -run=RouteMessage/2 -count=1000 -failfast.

Here are the bytes provided to ParseRIB for an example failure:

rt_msghdr
48010504 02000000 45003000 33000000 5f080000 03000000 00000000 00000000
00000000 00000000 00000000 00000000 00400000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 01000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000

sockaddr
                                                      1c1c0000 00000000
00000000 00000000 00000000 00000001 00000000

unknown data
                                             ffffffff 36120200 18000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 38120200 18030000 6c6f3000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000

sockaddr
                  1c1c0000 00000000 00000000 00000000 00000000 00000001
00000000

unknown data
         ffffffff

This data is expected to be an rt_msghdr struct, followed by an array of sockaddrs. The number of sockaddrs is specified in the header--2 in this case.

There are what appear to be two IPv6 sockaddrs in the data here, each consisting of 28 bytes starting with 1c1c.

There is also a chunk of data between the sockaddrs--everything from the ffffffff to the next 1c1c. We parse the first sockaddr, and then attempt to parse the ffffffff as a sockaddr and fail. The times the test passes, the chunk of data contains some different data which parses correctly.

I've poked through the source code for FreeBSD utilities which parse this same data, and none of them seem to handle this case. For example the code here looks pretty similar to that in x/net/route:
https://github.com/freebsd/freebsd-src/blob/main/usr.bin/netstat/route.c#L317

I'm very puzzled by what's going on here. This test is definitely pointing out a real problem. It is flaky only because the "successful" runs aren't detecting that they're parsing invalid data.

ianlancetaylor · 2021-05-20T03:40:40Z

I think you are misreading the data slightly. The number of addresses is not 2, but is determined by the number of bits set in the rtm_addrs field. In this case the value 0x33 indicates that there are four addresses. The first is an AF_INET6 address, the next two are AF_LINK addresses, and the last one is another AF_INET6 address. Viewed this way we can see that the 0xffffffff values appear after the AF_INET6 addresses, which strongly suggests that the address lengths are being rounded up to a multiple of 8 bytes, in this case rounding from 0x1c to 0x20. That makes me suspect that we are running a 386 program on an amd64 kernel. And that makes me suspect that https://golang.org/cl/139577 was mistaken, or perhaps that it simply doesn't apply to whatever system we are running. See what happens if you revert that CL.

neild · 2021-05-21T17:32:33Z

You're right, I was misreading it.

If I'm following the FreeBSD code correctly, after this change, sockaddrs in routing messages returned by the sysctl syscall with the SCTL_MASK32 flag set are 32-bit aligned. The test here is reading routing messages from an AF_ROUTE routing socket. Empirically, this returns 64-bit aligned sockaddrs.

We do also make a sysctl syscall elsewhere in this test, which is also returning 64-bit aligned sockaddrs.

I'm not clear on who sets the SCTL_MASK32 flag (userspace? kernel?), but so far as I can tell all the routing messages we're reading are 64-bit aligned.

gopherbot · 2021-05-21T17:52:56Z

Change https://golang.org/cl/321869 mentions this issue: route: revert routing message alignment for FreeBSD 386 emulation

bcmills added OS-FreeBSD NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 11, 2019

bcmills added this to the Backlog milestone Nov 11, 2019

bcmills changed the title ~~x/net/route: TestRouteMessage failures on freebsd builders~~ x/net/route: TestRouteMessage failures on freebsd-386 builders Feb 2, 2021

bcmills added the release-blocker label May 7, 2021

bcmills modified the milestones: Backlog, Go1.17 May 7, 2021

heschi removed the release-blocker label May 18, 2021

bcmills added Builders x/build issues (builders, bots, dashboards) okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 release-blocker labels May 19, 2021

gopherbot closed this as completed in golang/net@fe42d45 May 21, 2021

golang locked and limited conversation to collaborators May 21, 2022

gopherbot added the FrozenDueToAge label May 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/net/route: TestRouteMessage failures on freebsd-386 builders #35513

x/net/route: TestRouteMessage failures on freebsd-386 builders #35513

bcmills commented Nov 11, 2019

bradfitz commented Nov 11, 2019

bcmills commented Nov 11, 2019

bcmills commented Oct 19, 2020

bcmills commented Feb 2, 2021

bcmills commented May 7, 2021 •

edited

Loading

heschi commented May 13, 2021

neild commented May 14, 2021

heschi commented May 18, 2021

bcmills commented May 19, 2021

bcmills commented May 19, 2021 •

edited

Loading

neild commented May 19, 2021

ianlancetaylor commented May 20, 2021

neild commented May 21, 2021

gopherbot commented May 21, 2021

x/net/route: TestRouteMessage failures on freebsd-386 builders #35513

x/net/route: TestRouteMessage failures on freebsd-386 builders #35513

Comments

bcmills commented Nov 11, 2019

bradfitz commented Nov 11, 2019

bcmills commented Nov 11, 2019

bcmills commented Oct 19, 2020

bcmills commented Feb 2, 2021

bcmills commented May 7, 2021 • edited Loading

heschi commented May 13, 2021

neild commented May 14, 2021

heschi commented May 18, 2021

bcmills commented May 19, 2021

bcmills commented May 19, 2021 • edited Loading

neild commented May 19, 2021

ianlancetaylor commented May 20, 2021

neild commented May 21, 2021

gopherbot commented May 21, 2021

bcmills commented May 7, 2021 •

edited

Loading

bcmills commented May 19, 2021 •

edited

Loading