runtime: SIGTRAP or SIGQUIT in TestStress on NetBSD

[bradfitz]

Seen once at:

http://build.golang.org/log/fe7210aa075d8bbd6b4e5e5edf22722f27703d1d
(rev daddeb2 but fixed in the following commit)

ok      net/rpc 0.010s
ok      net/rpc/jsonrpc 0.006s
ok      net/smtp    0.026s
ok      net/textproto   0.003s
ok      net/url 0.043s
ok      os  0.322s
ok      os/exec 0.195s
SIGTRAP: trace trap
PC=0x808b195

goroutine 0 [idle]:
runtime.morestack()
    /usr/local/go/src/runtime/asm_386.s:314 +0x25

goroutine 9 [syscall]:
syscall.Syscall(0x3921, 0x1e, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/asm_netbsd_386.s:20 +0x5 fp=0x1841b7ac sp=0x1841b7a8
syscall.Kill(0x3921, 0x1e, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_netbsd_386.go:655 +0x46 fp=0x1841b7d0 sp=0x1841b7ac
os/signal.TestStress.func2(0x18450340, 0x18450380)
    /tmp/gobuilder/netbsd-386-minux-daddeb26865e/go/src/os/signal/signal_test.go:96 +0x79 fp=0x1841b7e8 sp=0x1841b7d0
runtime.goexit()
    /usr/local/go/src/runtime/asm_386.s:2423 +0x1 fp=0x1841b7ec sp=0x1841b7e8
created by os/signal.TestStress
    /tmp/gobuilder/netbsd-386-minux-daddeb26865e/go/src/os/signal/signal_test.go:101 +0x102

goroutine 1 [chan receive]:
testing.RunTests(0x81c5f54, 0x82357c0, 0x6, 0x6, 0x1)
    /usr/local/go/src/testing/testing.go:557 +0x874
testing.(*M).Run(0x1843a060, 0x824b540)
    /usr/local/go/src/testing/testing.go:486 +0x65
main.main()
    os/signal/_test/_testmain.go:66 +0x171

goroutine 4 [syscall]:
os/signal.loop()
    /tmp/gobuilder/netbsd-386-minux-daddeb26865e/go/src/os/signal/signal_unix.go:22 +0x1b
created by os/signal.init.1
    /tmp/gobuilder/netbsd-386-minux-daddeb26865e/go/src/os/signal/signal_unix.go:28 +0x36

goroutine 7 [runnable]:
time.Sleep(0x5f5e100, 0x0)
    /usr/local/go/src/runtime/time.go:59 +0xf0
os/signal.TestStress(0x1840c300)
    /tmp/gobuilder/netbsd-386-minux-daddeb26865e/go/src/os/signal/signal_test.go:102 +0x116
testing.tRunner(0x1840c300, 0x82357cc)
    /usr/local/go/src/testing/testing.go:448 +0x9a
created by testing.RunTests
    /usr/local/go/src/testing/testing.go:556 +0x842

goroutine 8 [select]:
os/signal.TestStress.func1(0x18450340, 0x18450380)
    /tmp/gobuilder/netbsd-386-minux-daddeb26865e/go/src/os/signal/signal_test.go:81 +0x1c8
created by os/signal.TestStress
    /tmp/gobuilder/netbsd-386-minux-daddeb26865e/go/src/os/signal/signal_test.go:88 +0xde

eax    0x1841b7a8
ebx    0x1841e780
ecx    0x184008c0
edx    0x0
edi    0x1841b790
esi    0x184008c0
ebp    0x0
esp    0x1841b77c
eip    0x808b195
eflags 0x246
cs     0x17
fs     0xab
gs     0xb3
FAIL    os/signal   0.255s
ok      os/user 0.003s
ok      path    0.002s
ok      path/filepath   0.021s
ok      reflect 0.068s
ok      regexp  0.078s
ok      regexp/syntax   0.465s

/cc @bsiegert @minux @mdempsky

[mdempsky]

Odd. I'm at a loss. Looking at the NetBSD kernel sources, the only reasons I can find that i386 would trigger SIGTRAP is when using ptrace() or if the process hits a breakpoint instruction.

[mdempsky]

Oh, duh, I think this is the cause:

306 // Cannot grow signal stack.
307 MOVL m_gsignal(BX), SI
308 CMPL g(CX), SI
309 JNE 2(PC)
310 INT $3

(The stack trace shows asm_386.s:314, because that's the next instruction after the INT.)

So somehow NetBSD is running out of stack space in its signal handler?

[mdempsky]

I'm still puzzled though. If we were running on m_gsignal, then we should have been running in a signal handler already when we hit INT 3 and triggered the SIGTRAP.

However, we mask all signals while handling signals. Also, at least on Linux, if you use INT 3 and have masked SIGTRAP, then the process core dumps immediately. Looking at NetBSD's trapsignal() kernel function, I think they do the same.

So the confusing part is, how did we print that SIGTRAP backtrace?

[dvyukov]

I don't think that we are running out of stack space. More likely we invoke the handler on a wrong stack. We've seen such crashes when native code calls sigaltstack and resetups signal stack. Check that SP is within gsignal stack at least in the very first function, sigtramp or how it is called. We ought to have such check.

[mikioh]

This still happens with go1.6beta1+ (071fcd1) on both netbsd/386 and netbsd/amd64.

[ianlancetaylor]

@mikioh If you can repeat this, can you post the trace back with GOTRACEBACK=2? Thanks.

I just ran TestStress 100 times on a netbsd/amd64 AWS system, with no failures.

[mikioh]

@ianlancetaylor, sorry I already dropped netbsd vm for saving storage space on my laptop.

[krytarowski]

I recommend to test NetBSD-current, there were incorporated many fixes around ptrace(2) and signal handling.

[bsiegert]

@krytarowski Any concrete suggestions for pull-ups for NetBSD 7.1?

[krytarowski]

Nothing offhand, I just suggest to check NetBSD-current.

[bradfitz]

Fixed by requiring NetBSD 8+ for Go 1.10.

Documentation bug is #22911