runtime: panics and/or deadlocks on FreeBSD 12.2-RELEASE

[problame]

What version of Go are you using (go version)?

$ go version
go version go1.15.6 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/cs/.cache/go-build"
GOENV="/home/cs/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/cs/development/golang/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/cs/development/golang"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/cs/go1.15.6"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/cs/go1.15.6/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build804757202=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I maintain zrepl, a tool for ZFS replication written in Go.
Some users have been hitting Go runtime panics and/or lock-ups on FreeBSD.

• go runtime panics and/or zrepl process hangs on FreeBSD 12 zrepl/zrepl#343 (comment)
  fatal error: bad g->status in ready
• go runtime panics and/or zrepl process hangs on FreeBSD 12 zrepl/zrepl#343 (comment)
  fatal error: wirep: invalid p state
• FreeBSD >= 12.2, < 13.0: Go runtime deadlocks and/or panics on multicore systems zrepl/zrepl#411 (comment)
  (hung in runtime.newobject)

The problems started with reports in July 2020 with a user on a development version of FreeBSD (12-STABLE, it must have been somewhere between 12.1-RELEASE and 12.2-RELEASE).
The problem occurred both with Go 1.13.5 and 14.*.
At the time I requested to reproduce the problem on an official binary release of FreeBSD but that did not happen.

After I updated my personal FreeBSD server to 12.2-RELEASE, I started to encounter similar issues as reported in July 2020.
I have not yet encountered runtime panics.
But several lock-ups (for lack of a better term) of the Go runtime.
The last of the links above contains a stack trace of a goroutine blocked forever on runtime.newobject (stack obtained using dlv).

Summary of my triaging since July:

• We have ruled out that it's due to the one use of unsafe in zrepl by removing the unsafe code path in a test build. The panics / lock-ups still occurred.
• The problems stop reproducibly when limiting the process to one CPU using the OS scheduler (cpuset).
• Most often the problems happen while sockets are being used. It does not happen when the daemon is idle.
• We have ruled out faulty hardware.
• The issue has only occurred on Intel systems on bare metal so far. I was unable to reproduce it in a stress test between two FreeBSD VMs on a Ryzen 1700X.
• I suspect the root cause is one of the following:
  • FreeBSD kernel bug introduced between 12.1-RELEASE and 12.2-RELEASE
  • Go runtime bug (would need to be present in multiple Go versions though)

It would be very helpful to get a quick explanation of what these panics mean so that I can narrow down my audit of the changes between FreeBSD 12.1-RELEASE and 12.2-RELEASE.

Also, I can offer a tmate or similar to a Go / FreeBSD developer to the system with the locked-up daemon.
The lock-up usually occurs after 2-3 days on my system, sometimes sooner, but I can leave it in the locked-up state for a day or two.

Related zrepl issues:

• go runtime panics and/or zrepl process hangs on FreeBSD 12 zrepl/zrepl#343
• FreeBSD >= 12.2, < 13.0: Go runtime deadlocks and/or panics on multicore systems zrepl/zrepl#411

[dmgk]

Just a wild guess, but it look related to 4d27c4c
@problame Can you try reproducing this issue with the recent tip? I can update lang/go-devel to include the above commit for easier testing.

[problame]

@dmgk that seems ... quite concerning, and since zrepl uses pipes like crazy, I wouldn't be surprised if this is the problem.

Is this a regression in Go or something that was always handled incorrectly?
I'm asking because I currently try to 'bisect' Go versions by compiling zrepl with older Go versions ans seeing whether the problem occurs.
The version that is running right now is built with Go 1.13, I would expect a crash to happen within the next 5 days.

Should I abort that run and build with tip + 4d27c4c as you told me to, or should we wait and see whether it crashes with Go 1.13?

(I cross-compile from Linux so I don't need the lang/go-devel port to be updated).

[dmgk]

Well, looking at the commit, it seems that the issue goes back to 2009 - https://github.com/golang/go/blame/master/src/runtime/sys_freebsd_amd64.s#L21, so I'd try reproducing with the tip first.

[problame]

Any idea why this would surface only since FreeBSD 12.2 then?

[mdtancsa]

Just a wild guess, but it look related to 4d27c4c
@problame Can you try reproducing this issue with the recent tip? I can update lang/go-devel to include the above commit for easier testing.

Hi,
I would be happy to try locally if you can update the port with this tip/fix/patch. I was the original reporter. Its fairly easy for me to reproduce. A couple of other details about when I ran into the issue last summer, I added more retention to the server (45K+ snapshots) and more clients replicating. So it would have become more busy as well as moving to RELENG12 at the time on FreeBSD.

[dmgk]

@mdtancsa Thanks, I just updated lang/go-devel to go1.16rc1 a few hours ago, it includes 4d27c4c.

[mdtancsa]

Thanks!! I am running now the same version I have been using for a while (v0.3.0-rc1) but compiled with golang dev from the ports! Thanks for the quick update. cpuset -l 0,1,2,3,4,5 -p 79245 Usually running with more than 2 CPU cores will cause a deadlock within a day or so. It should not take too long to find out.     ---Mike

…

On 1/28/2021 11:06 AM, Dmitri Goutnik wrote: @mdtancsa <https://github.com/mdtancsa> Thanks, I just updated |lang/go-devel| to |go1.16rc1| a few hours ago, it includes 4d27c4c <4d27c4c>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#43873 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGMT4QO7MXBFE5BGJTIJS2DS4GDPXANCNFSM4WP73Z7A>.

[mdtancsa]

So far so good! I am allowing all threads to bind to whatever core of the CPU. Normally, I would have had a few deadlocks by now. So its certainly MUCH improved.

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
10778 root 21 0 739M 83M uwait 7 8:05 4.72% zrepl-jan2021{zrepl-jan2021}
10778 root 21 0 739M 83M wait 10 3:03 3.50% zrepl-jan2021{zrepl-jan2021}
10778 root 23 0 739M 83M kqread 1 1:21 3.00% zrepl-jan2021{zrepl-jan2021}
10778 root 24 0 739M 83M uwait 5 2:44 2.87% zrepl-jan2021{zrepl-jan2021}
10778 root 21 0 739M 83M uwait 5 2:39 2.75% zrepl-jan2021{zrepl-jan2021}
10778 root 21 0 739M 83M uwait 11 3:10 2.27% zrepl-jan2021{zrepl-jan2021}
10778 root 21 0 739M 83M uwait 2 2:05 1.70% zrepl-jan2021{zrepl-jan2021}
10778 root 21 0 739M 83M uwait 9 2:51 1.45% zrepl-jan2021{zrepl-jan2021}
10778 root 21 0 739M 83M uwait 11 3:02 1.27% zrepl-jan2021{zrepl-jan2021}
10778 root 20 0 739M 83M uwait 0 3:02 0.51% zrepl-jan2021{zrepl-jan2021}
10778 root 20 0 739M 83M uwait 8 2:40 0.46% zrepl-jan2021{zrepl-jan2021}
10778 root 52 0 739M 83M uwait 7 0:25 0.34% zrepl-jan2021{zrepl-jan2021}
10778 root 23 0 739M 83M wait 4 2:34 0.28% zrepl-jan2021{zrepl-jan2021}
10778 root 21 0 739M 83M uwait 6 3:05 0.10% zrepl-jan2021{zrepl-jan2021}
10778 root 22 0 739M 83M uwait 7 3:11 0.04% zrepl-jan2021{zrepl-jan2021}
10778 root 20 0 739M 83M uwait 0 2:57 0.03% zrepl-jan2021{zrepl-jan2021}
10778 root 21 0 739M 83M uwait 4 2:56 0.02% zrepl-jan2021{zrepl-jan2021}
10778 root 22 0 739M 83M uwait 3 3:00 0.00% zrepl-jan2021{zrepl-jan2021}
10778 root 27 0 739M 83M uwait 8 1:59 0.00% zrepl-jan2021{zrepl-jan2021}
10778 root 20 0 739M 83M uwait 3 1:41 0.00% zrepl-jan2021{zrepl-jan2021}
10778 root 20 0 739M 83M uwait 1 0:00 0.00% zrepl-jan2021{zrepl-jan2021}
10778 root 20 0 739M 83M uwait 9 0:00 0.00% zrepl-jan2021{zrepl-jan2021}

This is still v0.3.0-rc1. If it survives the weekend without a deadlock, I will check out a newer version. Its a bit of work since I have to update all the clients as well. @problame, is there a particular version from the repo I should use ? Or can I just stick with 0.3.1 ?

[problame]

@mdtancsa Chasing this bug the last few weeks we've found a few places in zrepl that could cause deadlocks / data races. In all of those cases it was easy to rule out that they were related to the bug I have reported here. Those patches are in branch problame/develop. But that branch also contains a bump of a bunch of dependencies, including gRPC, which should not be included in your testing.

[mdtancsa]

I dont remember what patches I have been using on that version I have widely deployed, if any :(  I think at this point, its probably best to bring the server and its clients up to date.  Is the branch problame/develop safe enough to run in production do you think ?     ---Mike

…

On 1/30/2021 11:56 AM, Christian Schwarz wrote: @mdtancsa <https://github.com/mdtancsa> Chasing this bug the last few weeks we've found a few places in zrepl that could cause deadlocks / data races. In all of those cases it was easy to rule out that they were related to the bug I have reported here. Those patches are in branch |problame/develop|. But that branch also contains a bump of a bunch of dependencies, including gRPC, which should not be included in your testing. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#43873 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGMT4QJ3ZP2G3TEEWMIC22TS4Q233ANCNFSM4WP73Z7A>.

[problame]

If you are already running 0.3.1 then I wouldn't worry.

You shouldn't run the Update to protobuf v1.25 and grpc 1.35; bump CI to go1.12 when doing testing for this issue though because a bump in grpc / protobuf might introduce a regression that we could falsely identify as the issue we are debugging here.

So the commit for you is 166e80bb.

If you are not running 0.3.1 yet, just make sure that your grid pruning rules are correct (ref zrepl/zrepl#400 )