Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"fatal error: concurrent map read and map write" in buildkit builder docker image #2111

Closed
qoo2p5 opened this issue May 12, 2021 · 4 comments · Fixed by #2178
Closed

"fatal error: concurrent map read and map write" in buildkit builder docker image #2111

qoo2p5 opened this issue May 12, 2021 · 4 comments · Fixed by #2178
Labels
status/needs-more-info

Comments

@qoo2p5
Copy link

qoo2p5 commented May 12, 2021 •
edited
Loading

We use buildx with kubernetes driver to build our images. Sometimes buildkit pods fail with the following error: fatal error: concurrent map read and map write

Two interesting goroutines have the following stacktraces:

goroutine 4114488 [running]: runtime.throw(0x1300e9d, 0x21) /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc001ff33d8 sp=0xc001ff33a8 pc=0x42f572 runtime.mapaccess1_faststr(0x10fb0e0, 0xc000c83c20, 0x12e9a38, 0xf, 0xc001ff34a8) /usr/local/go/src/runtime/map_faststr.go:21 +0x44f fp=0xc001ff3448 sp=0xc001ff33d8 pc=0x412d3f github.com/moby/buildkit/cache/metadata.(*StorageItem).Get(0xc000bcf130, 0x12e9a38, 0xf, 0xc0cb33) /src/cache/metadata/metadata.go:267 +0x63 fp=0xc001ff3488 sp=0xc001ff3448 pc=0xc0cb33 ...

goroutine 4115673 [runnable]: go.etcd.io/bbolt.(*node).root(0xc0002e33b0, 0x0) /src/vendor/go.etcd.io/bbolt/node.go:24 +0x5c go.etcd.io/bbolt.(*Bucket).spill(0xc002aea100, 0xc001a0c100, 0xc0022539e0) /src/vendor/go.etcd.io/bbolt/bucket.go:573 +0x497 go.etcd.io/bbolt.(*Bucket).spill(0xc002aea080, 0xc001a0c000, 0xc002253bd8) /src/vendor/go.etcd.io/bbolt/bucket.go:537 +0x3d5 go.etcd.io/bbolt.(*Bucket).spill(0xc0001f9898, 0x3baed82819d30, 0x1d19940) /src/vendor/go.etcd.io/bbolt/bucket.go:537 +0x3d5 go.etcd.io/bbolt.(*Tx).Commit(0xc0001f9880, 0x0, 0x0) /src/vendor/go.etcd.io/bbolt/tx.go:160 +0xec go.etcd.io/bbolt.(*DB).Update(0xc0004b4400, 0xc002253e18, 0x0, 0x0) /src/vendor/go.etcd.io/bbolt/db.go:701 +0x106 github.com/moby/buildkit/cache/metadata.(*Store).Update(0xc00000e0f8, 0xc0008d7f41, 0x19, 0xc002253e80, 0x0, 0x0) /src/cache/metadata/metadata.go:167 +0x76 github.com/moby/buildkit/cache/metadata.(*StorageItem).Update(...) /src/cache/metadata/metadata.go:250 github.com/moby/buildkit/cache.updateLastUsed(0xc000bcf130, 0xc000c83c01, 0xc002253f40) /src/cache/metadata.go:503 +0x185 ...

So it's clear that si.SetValue in the second goroutine is called without locking mutex
@tonistiigi
Copy link
Member

Looks like there is a bunch of trace missing from what you have pasted. Eg. where is SetValue called. Please post full output.

@erebe
Copy link

erebe commented Jun 7, 2021

Hello,
I have the same issue (but not quite the same stacktrace) where our jobs inside gitlab-ci get killed due to a panic in buildkit.

runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 time="2021-06-07T16:06:11.872248689Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 fatal error: concurrent map read and map write
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 goroutine 363 [running]:                                                                                                                                                                                                                              [964/1380]
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 runtime.throw(0x1d828d7, 0x21)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/runtime/panic.go:774 +0x74 fp=0xc000aee040 sp=0xc000aee010 pc=0x431b04
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 runtime.mapaccess1_faststr(0x28bc9c0, 0xc000a4d770, 0x1d46b30, 0x7, 0x27b7be0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/runtime/map_faststr.go:21 +0x451 fp=0xc000aee0b0 sp=0xc000aee040 pc=0x414f71
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/textproto.MIMEHeader.Get(0xc000a4d770, 0x1d46b30, 0x7, 0xc001124d20, 0x27b7be0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/textproto/header.go:35 +0x5f fp=0xc000aee0e8 sp=0xc000aee0b0 pc=0x6f88cf
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.Header.Get(...)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/header.go:47
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.http2checkConnHeaders(0xc00081f700, 0xc000aee198, 0x54343a)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/h2_bundle.go:7422 +0x4a fp=0xc000aee158 sp=0xc000aee0e8 pc=0x75d43a
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.(*http2ClientConn).roundTrip(0xc0012b0780, 0xc00081f700, 0x0, 0x0, 0x0, 0x0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/h2_bundle.go:7453 +0x64 fp=0xc000aee498 sp=0xc000aee158 pc=0x75d8c4
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.(*http2Transport).RoundTripOpt(0xc000c4c540, 0xc00081f700, 0xc000aee600, 0x772e77, 0x0, 0xc0008f6240)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/h2_bundle.go:6936 +0x174 fp=0xc000aee5c0 sp=0xc000aee498 pc=0x759d94
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.(*http2Transport).RoundTrip(...)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/h2_bundle.go:6897
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.http2noDialH2RoundTripper.RoundTrip(0xc000c4c540, 0xc00081f700, 0xc000bf0870, 0x5, 0xc0008f62c8)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/h2_bundle.go:9032 +0x40 fp=0xc000aee618 sp=0xc000aee5c0 pc=0x766c90
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.(*Transport).roundTrip(0x3ccca20, 0xc00081f700, 0x20ccd98, 0x0, 0x0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/transport.go:485 +0xdca fp=0xc000aee870 sp=0xc000aee618 pc=0x78e5da
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.(*Transport).RoundTrip(0x3ccca20, 0xc00081f700, 0x20ccd98, 0x0, 0x0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/roundtrip.go:17 +0x37 fp=0xc000aee8a8 sp=0xc000aee870 pc=0x774627
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.(*Transport).RoundTrip(0x3d4f740, 0xc00081f700, 0x0, 0x0, 0x150)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/client.go:124 +0x530 fp=0xc000aee9c8 sp=0xc000aee8a8 pc=0x1575aa0
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/vendor/github.com/moby/buildkit/util/tracing.(*Transport).RoundTrip(0x3cb7c10, 0xc00081f700, 0x3cb7c10, 0x0, 0x0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/tracing/tracing.go:80 +0x2a4 fp=0xc000aeea50 sp=0xc000aee9c8 pc=0x1579ed4
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.send(0xc00081f700, 0x2b70ac0, 0x3cb7c10, 0x0, 0x0, 0x0, 0xc000010288, 0x8, 0x1, 0x0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/client.go:250 +0x445 fp=0xc000aeebb8 sp=0xc000aeea50 pc=0x737645
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.(*Client).send(0x3d56000, 0xc00081f700, 0x0, 0x0, 0x0, 0xc000010288, 0x0, 0x1, 0x120)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/client.go:174 +0xfc fp=0xc000aeec38 sp=0xc000aeebb8 pc=0x73705c
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.(*Client).do(0x3d56000, 0xc00081f700, 0x0, 0x0, 0x0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/client.go:641 +0x3d0 fp=0xc000aeee38 sp=0xc000aeec38 pc=0x738750
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 net/http.(*Client).Do(...)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/net/http/client.go:509
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/vendor/golang.org/x/net/context/ctxhttp.Do(0x2bcb980, 0xc000a4f1d0, 0x3d56000, 0xc00081f600, 0x0, 0x0, 0x2bcb980)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go:27 +0xf1 fp=0xc000aeeea8 sp=0xc000aeee38 pc=0x13a6f21
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/docker.(*dockerBase).doRequest(0xc000aa8240, 0x2bcb980, 0xc000a4d830, 0xc00081f600, 0x0, 0x0, 0x0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/docker/resolver.go:413 +0x483 fp=0xc000aef460 sp=0xc000aeeea8 pc=0x13bbbc3
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/docker.(*dockerBase).doRequestWithRetries(0xc000aa8240, 0x2bcb980, 0xc000a4d830, 0xc00081f600, 0xc0000f4348, 0x1, 0x1, 0xc00081f600, 0x0, 0x0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/docker/resolver.go:425 +0x58 fp=0xc000aef4d8 sp=0xc000aef460 pc=0x13bbfc8
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/docker.(*dockerBase).doRequestWithRetries(0xc000aa8240, 0x2bcb980, 0xc000a4d830, 0xc00081f600, 0xc0000f4348, 0x1, 0x1, 0x0, 0xc000a9e400, 0x0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/docker/resolver.go:438 +0x1a7 fp=0xc000aef550 sp=0xc000aef4d8 pc=0x13bc117
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve(0xc000a2cd00, 0x2bcb980, 0xc000a4d830, 0xc0008f2900, 0x37, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/remotes/docker/resolver.go:235 +0x556 fp=0xc000aef8c0 sp=0xc000aef550 pc=0x13b9db6
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/builder/builder-next/adapters/containerimage.(*cachedResolver).Resolve(0xc000a4d800, 0x2bcb8c0, 0xc000a2cc40, 0xc0008f2900, 0x37, 0x37, 0x0, 0x0, 0x0, 0x0, ...)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/builder/builder-next/adapters/containerimage/pull.go:838 +0xab fp=0xc000aef9b8 sp=0xc000aef8c0 pc=0x19f780b
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/vendor/github.com/moby/buildkit/util/imageutil.Config(0x2bcb8c0, 0xc000a2cc40, 0xc0008f2000, 0x37, 0x2badfc0, 0xc000a4d800, 0x7f8bcc240978, 0xc000a2f9c0, 0x0, 0x0, ...)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/imageutil/config.go:81 +0xa21 fp=0xc000aefd20 sp=0xc000aef9b8 pc=0x19bd8e1
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/builder/builder-next/adapters/containerimage.(*imageSource).resolveRemote.func1(0x2bcb8c0, 0xc000a2cc40, 0x2bcb8c0, 0xc000a2cc40, 0xc000dcabd0, 0x0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/builder/builder-next/adapters/containerimage/pull.go:138 +0x15e fp=0xc000aefe18 sp=0xc000aefd20 pc=0x19f891e
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol.(*call).run(0xc000a7c2d0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol/flightcontrol.go:121 +0xf0 fp=0xc000aefef8 sp=0xc000aefe18 pc=0x19537c0
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol.(*call).run-fm()
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol/flightcontrol.go:117 +0x2c fp=0xc000aeff10 sp=0xc000aefef8 pc=0x195535c
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 sync.(*Once).doSlow(0xc000a7c330, 0xc000dcabc0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/sync/once.go:66 +0xe5 fp=0xc000aeffb0 sp=0xc000aeff10 pc=0x47f3b5
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 sync.(*Once).Do(0xc000a7c330, 0xc000dcabc0)
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/sync/once.go:57 +0x47 fp=0xc000aeffd0 sp=0xc000aeffb0 pc=0x47f2c7
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 runtime.goexit()
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000aeffd8 sp=0xc000aeffd0 pc=0x462381
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0 created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol.(*call).wait
runner-qb6nwxz-project-15049077-concurrent-0ndrkf svc-0         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol/flightcontrol.go:148 +0x216

Our docker version is docker:19.03.13-dind
Let me know if you need more information.

@tonistiigi
Copy link
Member

@erebe This looks like containerd/containerd#4855 . Original report is different.

@erebe
Copy link

erebe commented Jun 7, 2021 •
edited
Loading

Indeed, I just found it by browsing other issues. Thanks for having looked at it :)
Do you want me to delete my comments ?

@tonistiigi tonistiigi mentioned this issue Jun 15, 2021
Merged
This was referenced Sep 6, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status/needs-more-info
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

metadata: hold lock on storageitem update tonistiigi/buildkit
3 participants
@tonistiigi @erebe @qoo2p5