net: failures with `signal arrived during cgo execution`

[bcmills]

#!watchflakes
post <- goos == "freebsd" && pkg == "net" && `signal arrived during cgo execution`

[bcmills]

One of these was reported in #27992 (comment), but doesn't match the prior failure mode for which that issue was created.

I saw another in a TryBot in https://storage.googleapis.com/go-build-log/55480854/freebsd-amd64-12_3_a94bfe2e.log.

[mknyszek]

Yeah, that definitely looks different from #27992, but I'm not sure this is necessarily going to be a C&RT issue? It seems like there's a segfault in C code during a net package test. signal arrived during cgo execution is just indicating that the crash happened in cgo, so the stack is going to be truncated. Tentatively removing the compiler/runtime label, but feel free to add it back if you disagree.

[bcmills]

signal arrived during cgo execution is just indicating that the crash happened in cgo, so the stack is going to be truncated.

Ah, I see. That makes it tricky to track down the actual failure, though — is there some way we can provide the equivalent of a default runtime.SetCgoTraceback for the cgo dependencies in the standard library? (Can we make assumptions about system libraries that allow for something simpler than the full generality of SetCgoTraceback that we need for user C code?)

(CC @ianlancetaylor)

[ianlancetaylor]

If anything, system libraries are harder to get a traceback from, because they are always heavily optimized and because often the debug info is stored somewhere else. (Without the debug info the traceback is close to useless, it's just a list of PC values.)

For debugging purposes a blank import of github.com/ianlancetaylor/cgosymbolizer will often get a C backtrace, but I really can't recommend making that part of the Go standard library. It's 17,000 lines of C code.

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "net" && `signal arrived during cgo execution`

2023-05-10 22:43 freebsd-amd64-12_3 go@639957eb net.TestLookupDotsWithRemoteSource (log)

SIGSEGV: segmentation violation
PC=0x8009f9f1c m=0 sigcode=1
signal arrived during cgo execution

rax    0xc2c3
rbx    0x437e
rcx    0xc2c3
rdx    0xffffffffffffffff
rdi    0x0
rsi    0x0
...
	/tmp/workdir/go/src/net/lookup_unix.go:96 +0xa5 fp=0xc00032bd98 sp=0xc00032bd38 pc=0x561325
net.(*Resolver).LookupCNAME(0xc000392310?, {0x6dcc10?, 0x879980?}, {0x685212, 0xb})
	/tmp/workdir/go/src/net/lookup.go:472 +0x2b fp=0xc00032bde0 sp=0xc00032bd98 pc=0x55deeb
net.LookupCNAME(...)
	/tmp/workdir/go/src/net/lookup.go:455
net.testDots(0xc0001df380, {0x682ee0, 0x3})
	/tmp/workdir/go/src/net/lookup_test.go:676 +0x12e fp=0xc00032bef0 sp=0xc00032bde0 pc=0x5b918e
net.TestLookupDotsWithRemoteSource(0xc0001df380)
	/tmp/workdir/go/src/net/lookup_test.go:658 +0x157 fp=0xc00032bf70 sp=0xc00032bef0 pc=0x5b8fb7
testing.tRunner(0xc0001df380, 0x699c48)

2023-05-22 16:48 freebsd-amd64-13_0 go@10fbd925 net.TestLookupDotsWithRemoteSource (log)

SIGSEGV: segmentation violation
PC=0x800a115e1 m=0 sigcode=1
signal arrived during cgo execution

rax    0x1ffff8
rbx    0x8008621b0
rcx    0xffffffffffffffff
rdx    0x8008620c0
rdi    0xffffffffffffffff
rsi    0x800862280
...
	/tmp/workdir/go/src/net/lookup_unix.go:96 +0xa5 fp=0xc00023bd98 sp=0xc00023bd38 pc=0x563c85
net.(*Resolver).LookupCNAME(0xc000311cb0?, {0x6e3a90?, 0x883bc0?}, {0x68a838, 0xb})
	/tmp/workdir/go/src/net/lookup.go:472 +0x2b fp=0xc00023bde0 sp=0xc00023bd98 pc=0x56082b
net.LookupCNAME(...)
	/tmp/workdir/go/src/net/lookup.go:455
net.testDots(0xc00020cea0, {0x6884f3, 0x3})
	/tmp/workdir/go/src/net/lookup_test.go:676 +0x12e fp=0xc00023bef0 sp=0xc00023bde0 pc=0x5bd08e
net.TestLookupDotsWithRemoteSource(0xc00020cea0)
	/tmp/workdir/go/src/net/lookup_test.go:658 +0x157 fp=0xc00023bf70 sp=0xc00023bef0 pc=0x5bceb7
testing.tRunner(0xc00020cea0, 0x69f4f0)

2023-05-22 19:05 freebsd-amd64-13_0 go@6761bff4 net.TestLookupDotsWithRemoteSource (log)

SIGSEGV: segmentation violation
PC=0x800a115e1 m=0 sigcode=1
signal arrived during cgo execution

rax    0x1ffff8
rbx    0x8008621b0
rcx    0xffffffffffffffff
rdx    0x8008620c0
rdi    0xffffffffffffffff
rsi    0x800862280
...
	/tmp/workdir/go/src/net/lookup_unix.go:96 +0xa5 fp=0xc0003abd98 sp=0xc0003abd38 pc=0x563c85
net.(*Resolver).LookupCNAME(0xc000358b10?, {0x6e3a90?, 0x883bc0?}, {0x68a838, 0xb})
	/tmp/workdir/go/src/net/lookup.go:472 +0x2b fp=0xc0003abde0 sp=0xc0003abd98 pc=0x56082b
net.LookupCNAME(...)
	/tmp/workdir/go/src/net/lookup.go:455
net.testDots(0xc0003689c0, {0x6884f3, 0x3})
	/tmp/workdir/go/src/net/lookup_test.go:676 +0x12e fp=0xc0003abef0 sp=0xc0003abde0 pc=0x5bd08e
net.TestLookupDotsWithRemoteSource(0xc0003689c0)
	/tmp/workdir/go/src/net/lookup_test.go:658 +0x157 fp=0xc0003abf70 sp=0xc0003abef0 pc=0x5bceb7
testing.tRunner(0xc0003689c0, 0x69f4f0)

2023-05-22 19:37 freebsd-amd64-13_0 go@8c445b7c net.TestLookupDotsWithRemoteSource (log)

SIGSEGV: segmentation violation
PC=0x800a115e1 m=0 sigcode=1
signal arrived during cgo execution

rax    0x1ffff8
rbx    0x8008621b0
rcx    0xffffffffffffffff
rdx    0x8008620c0
rdi    0xffffffffffffffff
rsi    0x800862280
...
	/tmp/workdir/go/src/net/lookup_unix.go:96 +0xa5 fp=0xc00023dd98 sp=0xc00023dd38 pc=0x563c85
net.(*Resolver).LookupCNAME(0xc000580dd0?, {0x6e3a90?, 0x883bc0?}, {0x68a838, 0xb})
	/tmp/workdir/go/src/net/lookup.go:472 +0x2b fp=0xc00023dde0 sp=0xc00023dd98 pc=0x56082b
net.LookupCNAME(...)
	/tmp/workdir/go/src/net/lookup.go:455
net.testDots(0xc0002b4ea0, {0x6884f3, 0x3})
	/tmp/workdir/go/src/net/lookup_test.go:676 +0x12e fp=0xc00023def0 sp=0xc00023dde0 pc=0x5bd08e
net.TestLookupDotsWithRemoteSource(0xc0002b4ea0)
	/tmp/workdir/go/src/net/lookup_test.go:658 +0x157 fp=0xc00023df70 sp=0xc00023def0 pc=0x5bceb7
testing.tRunner(0xc0002b4ea0, 0x69f4f0)

2023-05-23 11:36 freebsd-amd64-13_0 go@380529d5 net.TestLookupDotsWithRemoteSource (log)

SIGSEGV: segmentation violation
PC=0x800a115e1 m=0 sigcode=1
signal arrived during cgo execution

rax    0x1ffff8
rbx    0x8008621b0
rcx    0xffffffffffffffff
rdx    0x8008620c0
rdi    0xffffffffffffffff
rsi    0x800862280
...
	/tmp/workdir/go/src/net/lookup_unix.go:96 +0xa5 fp=0xc0003d5d98 sp=0xc0003d5d38 pc=0x563c85
net.(*Resolver).LookupCNAME(0xc0000c87f0?, {0x6e3a90?, 0x883bc0?}, {0x68a838, 0xb})
	/tmp/workdir/go/src/net/lookup.go:472 +0x2b fp=0xc0003d5de0 sp=0xc0003d5d98 pc=0x56082b
net.LookupCNAME(...)
	/tmp/workdir/go/src/net/lookup.go:455
net.testDots(0xc000133a00, {0x6884f3, 0x3})
	/tmp/workdir/go/src/net/lookup_test.go:676 +0x12e fp=0xc0003d5ef0 sp=0xc0003d5de0 pc=0x5bd08e
net.TestLookupDotsWithRemoteSource(0xc000133a00)
	/tmp/workdir/go/src/net/lookup_test.go:658 +0x157 fp=0xc0003d5f70 sp=0xc0003d5ef0 pc=0x5bceb7
testing.tRunner(0xc000133a00, 0x69f4f0)

2023-05-23 16:36 freebsd-amd64-13_0 go@d9f7efed net.TestLookupDotsWithRemoteSource (log)

SIGSEGV: segmentation violation
PC=0x800a115e1 m=0 sigcode=1
signal arrived during cgo execution

rax    0x1ffff8
rbx    0x8008621b0
rcx    0xffffffffffffffff
rdx    0x8008620c0
rdi    0xffffffffffffffff
rsi    0x800862280
...
	/tmp/workdir/go/src/net/lookup_unix.go:96 +0xa5 fp=0xc0000cfd98 sp=0xc0000cfd38 pc=0x563c85
net.(*Resolver).LookupCNAME(0xc00009c2b0?, {0x6e3a90?, 0x883bc0?}, {0x68a838, 0xb})
	/tmp/workdir/go/src/net/lookup.go:472 +0x2b fp=0xc0000cfde0 sp=0xc0000cfd98 pc=0x56082b
net.LookupCNAME(...)
	/tmp/workdir/go/src/net/lookup.go:455
net.testDots(0xc000596820, {0x6884f3, 0x3})
	/tmp/workdir/go/src/net/lookup_test.go:676 +0x12e fp=0xc0000cfef0 sp=0xc0000cfde0 pc=0x5bd08e
net.TestLookupDotsWithRemoteSource(0xc000596820)
	/tmp/workdir/go/src/net/lookup_test.go:658 +0x157 fp=0xc0000cff70 sp=0xc0000cfef0 pc=0x5bceb7
testing.tRunner(0xc000596820, 0x69f4f0)

2023-05-23 19:06 freebsd-amd64-13_0 go@ef2bb813 net.TestLookupDotsWithRemoteSource (log)

SIGSEGV: segmentation violation
PC=0x800a115e1 m=0 sigcode=1
signal arrived during cgo execution

rax    0x1ffff8
rbx    0x8008621b0
rcx    0xffffffffffffffff
rdx    0x8008620c0
rdi    0xffffffffffffffff
rsi    0x800862280
...
	/tmp/workdir/go/src/net/lookup_unix.go:96 +0xa5 fp=0xc00022bd98 sp=0xc00022bd38 pc=0x5640c5
net.(*Resolver).LookupCNAME(0xc000388f20?, {0x6e3b70?, 0x883bc0?}, {0x68a838, 0xb})
	/tmp/workdir/go/src/net/lookup.go:472 +0x2b fp=0xc00022bde0 sp=0xc00022bd98 pc=0x560c6b
net.LookupCNAME(...)
	/tmp/workdir/go/src/net/lookup.go:455
net.testDots(0xc00039b860, {0x6884f3, 0x3})
	/tmp/workdir/go/src/net/lookup_test.go:676 +0x12e fp=0xc00022bef0 sp=0xc00022bde0 pc=0x5bd4ce
net.TestLookupDotsWithRemoteSource(0xc00039b860)
	/tmp/workdir/go/src/net/lookup_test.go:658 +0x157 fp=0xc00022bf70 sp=0xc00022bef0 pc=0x5bd2f7
testing.tRunner(0xc00039b860, 0x69f528)

— watchflakes

[bcmills]

Iiiiinteresting, all freebsd-amd64-13_0.

attn @golang/freebsd !

[ayang64]

Okay -- so I'll see if i can reproduce but running the tests with github.com/ianlancetaylor/cgosymbolizer and if I bump into it, I'll post the trace.

I'm curious: are we running FreeBSD 14 trybots? It might be interesting to know if this was fixed in later releases -- might give me a place to start bisecting.