runtime: Crash in GC on Debian riscv64 (qemu)

[phpeterson-usf]

What version of Go are you using (go version)?

$ go version
go version go1.18.4 darwin/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

I am cross-compiling the micro editor on my Mac targeting a Debian riscv64 guest under QEMU. The Debian guest is Linux debian 5.18.0-2-riscv64 https://github.com/zyedidia/micro/issues/1 SMP Debian 5.18.5-1 (2022-06-16) riscv64 GNU/Linux

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/phil/Library/Caches/go-build"
GOENV="/Users/phil/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/phil/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/phil/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.18.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/phil/riscv64-ubuntu/micro/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bl/ncrkxg1d6r130_5pb8mn3y3m0000gn/T/go-build3611809763=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

1. After cloning the repo, the build command is GOOS=linux GOARCH=riscv64 CGO_ENABLED=0 make build.
2. The resulting binary is called micro and I scp that to the Debian guest

What did you expect to see?

I expect to run the micro editor

What did you see instead?

The micro editor crashes on launch with the output below, apparently showing a crash in the Go GC. I opened an issue in the micro repo, but the author suggests that the root cause is inside Go.

Interestingly, the same micro binary runs on an Ubuntu guest Linux ubuntu 5.13.0-1026-generic #29~20.04.1-Ubuntu SMP Fri Jun 3 11:55:52 UTC 2022 riscv64 riscv64 riscv64 GNU/Linux but crashes on launch on the Debian guest.

stack trace stack trace

runtime: lfstack.push invalid packing: node=0xffffff8cc2b980 cnt=0x1 packed=0xffff8cc2b9800001 -> node=0xffff8cc2b980

fatal error: lfstack.push
runtime stack:

runtime.throw({0x5966ba, 0xc})

runtime/panic.go:992 +0x58

runtime.(*lfstack).push(0xaf62e8, 0xffffff8cc2b980)

runtime/lfstack.go:30 +0x144

runtime.(*spanSetBlockAlloc).free(...)

runtime/mspanset.go:292

runtime.(*spanSet).reset(0xaef898)

runtime/mspanset.go:265 +0xac

runtime.finishsweep_m()

runtime/mgcsweep.go:260 +0xc8

runtime.gcStart.func1()

runtime/mgc.go:664 +0x1c

runtime.systemstack()

runtime/asm_riscv64.s:133 +0x50
goroutine 1 [running]:

runtime.systemstack_switch()

runtime/asm_riscv64.s:96 +0x8 fp=0xc0004b3720 sp=0xc0004b3718 pc=0x74098

runtime.gcStart({0x0, 0x0, 0x0})

runtime/mgc.go:663 +0x500 fp=0xc0004b3798 sp=0xc0004b3720 pc=0x29188

runtime.mallocgc(0x30, 0x5576e0, 0x1)

runtime/malloc.go:1205 +0x71c fp=0xc0004b3810 sp=0xc0004b3798 pc=0x1bdb4

runtime.growslice(0x5576e0, {0x0, 0x0, 0x0}, 0x1)

runtime/slice.go:278 +0x4e0 fp=0xc0004b3860 sp=0xc0004b3810 pc=0x5e4b8

regexp/syntax.(*compiler).inst(...)

regexp/syntax/compile.go:164

regexp/syntax.(*compiler).init(...)

regexp/syntax/compile.go:83

regexp/syntax.Compile(0xc000131810)

regexp/syntax/compile.go:73 +0x9c fp=0xc0004b3920 sp=0xc0004b3860 pc=0x151214

regexp.compile({0x59c8fd, 0x14}, 0xd4, 0x0)

regexp/regexp.go:180 +0xa4 fp=0xc0004b39b8 sp=0xc0004b3920 pc=0x167c94

regexp.Compile(...)

regexp/regexp.go:135

regexp.MustCompile({0x59c8fd, 0x14})

regexp/regexp.go:315 +0x3c fp=0xc0004b3a38 sp=0xc0004b39b8 pc=0x1688ac

main.LoadInput({0xc00001c310, 0x0, 0x0})

github.com/zyedidia/micro/v2/cmd/micro/micro.go:166 +0x14c fp=0xc0004b3bb8 sp=0xc0004b3a38 pc=0x4af5d4

main.main()

github.com/zyedidia/micro/v2/cmd/micro/micro.go:319 +0x748 fp=0xc0004b3f80 sp=0xc0004b3bb8 pc=0x4b0988

runtime.main()

runtime/proc.go:250 +0x228 fp=0xc0004b3fd8 sp=0xc0004b3f80 pc=0x48480

runtime.goexit()

runtime/asm_riscv64.s:497 +0x4 fp=0xc0004b3fd8 sp=0xc0004b3fd8 pc=0x75ec4
goroutine 6 [sleep]:

time.Sleep(0x1dcd65000)

runtime/time.go:194 +0x150

github.com/zyedidia/micro/v2/internal/buffer.backupThread()

github.com/zyedidia/micro/v2/internal/buffer/backup.go:37 +0x28

created by github.com/zyedidia/micro/v2/internal/buffer.init.0

github.com/zyedidia/micro/v2/internal/buffer/backup.go:52 +0x68
goroutine 8 [syscall]:

os/signal.signal_recv()

runtime/sigqueue.go:151 +0x38

os/signal.loop()

os/signal/signal_unix.go:23 +0x1c

created by os/signal.Notify.func1.1

os/signal/signal.go:151 +0x2c
goroutine 9 [select]:

github.com/zyedidia/tcell/v2.(*tScreen).mainLoop(0xc000190000)

github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:1580 +0xe0

created by github.com/zyedidia/tcell/v2.(*tScreen).Init

github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:210 +0x6cc
goroutine 10 [IO wait]:

internal/poll.runtime_pollWait(0xffffff8cc35198, 0x72)

runtime/netpoll.go:302 +0xe0

internal/poll.(*pollDesc).wait(0xc000058e58, 0x72, 0x1)

internal/poll/fd_poll_runtime.go:83 +0x34

internal/poll.(*pollDesc).waitRead(...)

internal/poll/fd_poll_runtime.go:88

internal/poll.(*FD).Read(0xc000058e40, {0xc0003fb000, 0x1000, 0x1000})

internal/poll/fd_unix.go:167 +0x1f0

os.(*File).read(...)

os/file_posix.go:31

os.(*File).Read(0xc00000c780, {0xc0003fb000, 0x1000, 0x1000})

os/file.go:119 +0x70

github.com/zyedidia/tcell/v2.(*tScreen).inputLoop(0xc000190000)

github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:1633 +0x94

created by github.com/zyedidia/tcell/v2.(*tScreen).Init

github.com/zyedidia/tcell/v2@v2.0.9/tscreen.go:211 +0x724

[mengzhuo]

It will be fixed after this CL been merged.
https://go-review.googlesource.com/c/go/+/409055

[phpeterson-usf]

Thanks. I would be thrilled if this could make the 1.19 release train to support my teaching objectives.

[cherrymui]

Can this reproduce on a real hardware?

For QEMU, I guess it would be possible to configure it to not enable SV57 mode?

[phpeterson-usf]

I believe SV57 is enabled in the 5.18 kernel which is distributed with Debian/riscv64. I have looked unsuccessfully for runtime configuration for that feature. If there's a way to change that in QEMU, I don't know what it is.

I don't have an Unmatched board, and SiFive discontinued them in favor of a pre-release collaboration with Intel.

If it's too late for 1.19, the two possible workarounds I see are:

1. Downgrade the kernel to pre-SV57
2. Build Go from source, patching in Ian's change myself

[phpeterson-usf]

FWIW, I built from go1.19rc2 + Ian's change (e9f157e.diff) and cross-compiled micro to avoid bootstrapping Go on riscv64. The resulting micro binary seems to work fine on my Debian/riscv64 guest.

[mengzhuo]

Can this reproduce on a real hardware?

For QEMU, I guess it would be possible to configure it to not enable SV57 mode?

Most of existed real hardwares (that I can get) using sv39

• Sifive unmatched(Go build farm): sv39
• Starfive v1: sv39
• Allwinner D1: sv39

Qemu configured with -cpu=sifive-u54 to disable SV57.

[yuzibo]

Hi,
I just have Unmatched boards by hand and I tested it easily:

vimer@unmatched:~/go/proj/bin$ uname -a
Linux unmatched 5.18.0-2-riscv64 #1 SMP Debian 5.18.5-1 (2022-06-16) riscv64 GNU/Linux
vimer@unmatched:~/go/proj/bin$ go env
GO111MODULE=""
GOARCH="riscv64"
GOBIN=""
GOCACHE="/home/vimer/.cache/go-build"
GOENV="/home/vimer/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="riscv64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/vimer/go/proj/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/vimer/go/proj"
GOPRIVATE=""
GOPROXY="https://proxy.golang.com.cn"
GOROOT="/home/vimer/go/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/vimer/go/go/pkg/tool/linux_riscv64"
GOVCS=""
GOVERSION="go1.18.4"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2096220389=/tmp/go-build -gno-record-gcc-switches"

vimer@unmatched:~/go/proj/bin$ ./micro --version
Version: 2.0.11-dev.100
Commit hash: 225927b9
Compiled on July 29, 2022

vimer@unmatched:~/go/proj/bin$ ip a | ./micro

The micro editor runs fine in my view also.
Please let me know if there are any issues.