concurrent map writes to cm.records

[weishenmezhememan]

Build image with the version 0.8.3， panic with error as follow：

Apr 28 04:20:05 buildkitd: fatal error: concurrent map writes

Apr 28 04:20:05 buildkitd: goroutine 63818 [running]:

Apr 28 04:20:05 buildkitd: runtime.throw(0x1205b5b, 0x15)

Apr 28 04:20:05 buildkitd: /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc000ba10a8 sp=0xc000ba1078 pc=0x430712

Apr 28 04:20:05 buildkitd: runtime.mapdelete_faststr(0x104ff40, 0xc0000b74d0, 0xc0004324e1, 0x19)

Apr 28 04:20:05 buildkitd: /usr/local/go/src/runtime/map_faststr.go:306 +0x386 fp=0xc000ba1110 sp=0xc000ba10a8 pc=0x414b56

Apr 28 04:20:05 buildkitd: github.com/moby/buildkit/cache.(*cacheRecord).remove(0xc000e5a090, 0x139afe0, 0xc000345940, 0x1b54201, 0x0, 0x0)

Apr 28 04:20:05 buildkitd: /src/cache/refs.go:258 +0x67 fp=0xc000ba11f0 sp=0xc000ba1110 pc=0xb8d9f7

Apr 28 04:20:05 buildkitd: github.com/moby/buildkit/cache.(*cacheManager).prune(0xc00044a080, 0x139afe0, 0xc000345940, 0xc0008143c0, 0x1377d40, 0x1243978, 0x0, 0x1376440, 0xc0009acbc0, 0x0, ...)

Apr 28 04:20:05 buildkitd: /src/cache/manager.go:726 +0xe26 fp=0xc000ba1570 sp=0xc000ba11f0 pc=0xb852c6

Apr 28 04:20:05 buildkitd: github.com/moby/buildkit/cache.(*cacheManager).prune(0xc00044a080, 0x139afe0, 0xc000345940, 0xc0008143c0, 0x1377d40, 0x1243978, 0x0, 0x1376440, 0xc0009acbc0, 0x0, ...)

Apr 28 04:20:05 buildkitd: /src/cache/manager.go:743 +0x1163 fp=0xc000ba18f0 sp=0xc000ba1570 pc=0xb85603

Apr 28 04:20:05 buildkitd: github.com/moby/buildkit/cache.(*cacheManager).prune(0xc00044a080, 0x139afe0, 0xc000345940, 0xc0008143c0, 0x1377d40, 0x1243978, 0x0, 0x1376440, 0xc0009acbc0, 0x0, ...)

Apr 28 04:20:05 buildkitd: /src/cache/manager.go:743 +0x1163 fp=0xc000ba1c70 sp=0xc000ba18f0 pc=0xb85603

Apr 28 04:20:05 buildkitd: github.com/moby/buildkit/cache.(*cacheManager).pruneOnce(0xc00044a080, 0x139afe0, 0xc000345940, 0xc0008143c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2328, ...)

Apr 28 04:20:05 buildkitd: /src/cache/manager.go:562 +0x1c8 fp=0xc000ba1d88 sp=0xc000ba1c70 pc=0xb841c8

Apr 28 04:20:05 buildkitd: github.com/moby/buildkit/cache.(*cacheManager).Prune(0xc00044a080, 0x139afe0, 0xc000345940, 0xc0008143c0, 0xc00044c0c0, 0x4, 0x4, 0x432450, 0x1245fa8)Apr 28 04:20:05 paas-controller-3 buildkitd: /src/cache/manager.go:516 +0x142 fp=0xc000ba1e80 sp=0xc000ba1d88 pc=0xb83ea2

Apr 28 04:20:05 buildkitd: github.com/moby/buildkit/worker/base.(*Worker).Prune(0xc000454120, 0x139afe0, 0xc000345940, 0xc0008143c0, 0xc00044c0c0, 0x4, 0x4, 0xc00071ef30, 0x4429ff)

Apr 28 04:20:05 buildkitd: /src/worker/base

There may be some problems with the method ‘’prune‘’：
func (cm *cacheManager) prune(ctx context.Context, ch chan client.UsageInfo, opt pruneOpt) error {
var toDelete []*deleteRecord
if opt.keepBytes != 0 && opt.totalSize < opt.keepBytes {
return nil
}
cm.mu.Lock()
...
cm.mu.Unlock()
...
for _, cr := range toDelete {
cr.mu.Lock()
//the record object will operate the current cm (cacheManager) map when it is removed, and cm.mu.Lock() should be performed before this operation. At present, there is no such action in the code.
if err1 := cr.remove(ctx, true); err == nil {
err = err1
}
}
...
}

[weishenmezhememan]

Please help confirm this problem, or what more information needs to be provided？