fatal error: concurrent map writes

[kernfeld-cockroach]

Describe the problem

When I run CockroachCloud's end-to-end tests, my local control plane DB appears to always crash with the error fatal error: concurrent map writes

To Reproduce

It's likely infeasible to set up the CockroachCloud development environment needed to reliably reproduce this bug. Even if that were done, the panic may depend on the state of the DB in a way that would be difficult to replicate.

Expected behavior

DB does not panic.

Additional data / screenshots

goroutine 1709 [running]:
runtime.throw(0x8737a25, 0x15)
	/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00411d978 sp=0xc00411d948 pc=0x4038ef2
runtime.mapassign_fast32(0x8002400, 0xc002509590, 0xc00000002c, 0x95a78a0)
	/usr/local/go/src/runtime/map_fast32.go:101 +0x33e fp=0xc00411d9b8 sp=0xc00411d978 pc=0x4013fde
github.com/cockroachdb/cockroach/pkg/sql/opt.(*TableMeta).copyScalars(0xc002abf930, 0xc00411dbf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/table_meta.go:190 +0x165 fp=0xc00411dad0 sp=0xc00411d9b8 pc=0x5f22c25
github.com/cockroachdb/cockroach/pkg/sql/opt.(*Metadata).CopyFrom(0xc002a48a80, 0xc0004e5180, 0xc00411dbf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/metadata.go:227 +0x445 fp=0xc00411dba8 sp=0xc00411dad0 pc=0x5f1c505
github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).CopyAndReplace(0xc002d60598, 0x9694ec0, 0xc001d63310, 0xc001efa820, 0xc00411dc48)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:222 +0xb2 fp=0xc00411dc10 sp=0xc00411dba8 pc=0x60e7f72
github.com/cockroachdb/cockroach/pkg/sql/opt/norm.(*Factory).AssignPlaceholders(0xc002d60598, 0xc0004e5180, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:269 +0x105 fp=0xc00411dc80 sp=0xc00411dc10 pc=0x60e8245
github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).reuseMemo(0xc002d60538, 0xc0004e5180, 0xc002627800, 0x8730105, 0x13)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:436 +0x8c fp=0xc00411dcc0 sp=0xc00411dc80 pc=0x714c70c
github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo(0xc002d60538, 0x9550240, 0xc002627800, 0x2a80, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:469 +0x84a fp=0xc00411dde8 sp=0xc00411dcc0 pc=0x714d02a
github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan(0xc002d5fdf8, 0x9550240, 0xc002627800, 0xc01b6653f8, 0x1b6653f80408c93c)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:194 +0xde fp=0xc00411dea0 sp=0xc00411dde8 pc=0x714b79e
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan(0xc002d5f900, 0x9550240, 0xc002627800, 0xc002d5fdf8, 0x0, 0xc00411dfe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:948 +0x5a fp=0xc00411df48 sp=0xc00411dea0 pc=0x70583ba
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc002d5f900, 0x9550240, 0xc002627800, 0xc002d5fdf8, 0x188b1088, 0xc00323e180, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:829 +0x18a fp=0xc00411e208 sp=0xc00411df48 pc=0x70574ea
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc002d5f900, 0x9550240, 0xc002627800, 0x956e340, 0xc002802be0, 0xc00125da6e, 0xdb, 0xb, 0x1, 0xc0037383c0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:667 +0xfa6 fp=0xc00411eff0 sp=0xc00411e208 pc=0x70546a6
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc002d5f900, 0x9550180, 0xc003878080, 0x956e340, 0xc002802be0, 0xc00125da6e, 0xdb, 0xb, 0x1, 0xc0037383c0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:123 +0xb14 fp=0xc00411f520 sp=0xc00411eff0 pc=0x7052ff4
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal(0xc002d5f900, 0x9550180, 0xc003878080, 0xc0037383c0, 0xc001777ce0, 0xb, 0xb, 0xc001cb8320, 0xb, 0xb, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:210 +0x14e fp=0xc00411f658 sp=0xc00411f520 pc=0x705344e
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2(0xc00125ded0, 0x0, 0x0, 0x1b650fc0, 0xed8c073e3, 0x0, 0xc002d5f900, 0xc00411fa20, 0xc00411fa10, 0x1e, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1596 +0x47b fp=0xc00411f870 sp=0xc00411f658 pc=0x7217b5b
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc002d5f900, 0x9550180, 0xc003878080, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1598 +0x45c fp=0xc00411fcc8 sp=0xc00411f870 pc=0x704749c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc002d5f900, 0x9550180, 0xc002d9fc00, 0xc001928be0, 0x5400, 0x15000, 0xc001928c80, 0xc0028472a0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1450 +0x228 fp=0xc00411fd68 sp=0xc00411fcc8 pc=0x7046dc8
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc000652580, 0x9550180, 0xc002d9fc00, 0xc002d5f900, 0x5400, 0x15000, 0xc001928c80, 0xc0028472a0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:484 +0xce fp=0xc00411fdf0 sp=0xc00411fd68 pc=0x70427ae
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc000df01bf, 0xc0036d2160, 0x9550180, 0xc002d9fc00, 0xc0028472a0, 0xc000652580, 0xc00323db00, 0x95b75c0, 0xc0031e4780, 0xc002762180, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:627 +0x3ea fp=0xc00411ff38 sp=0xc00411fdf0 pc=0x7708eea
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00411ff40 sp=0xc00411ff38 pc=0x4071e21
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:555 +0x185

I will happily provide additional logs!

Environment:

cockroach version
Build Tag:        v21.1.4
Build Time:       2021/06/29 19:09:44
Distribution:     CCL
Platform:         darwin amd64 (x86_64-apple-darwin19)
Go Version:       go1.15.11
C Compiler:       Clang 10.0.0
Build Commit ID:  92e342b8f0032a5771d3e50932df4c6d7d85fe41
Build Type:       release

Additional context

I plan to try to fix this by switching to the version of CRDB available from homebrew, which is currently 21.1.7.

[jordanlewis]

cc @rytaft @RaduBerinde

[jordanlewis]

Oh, sorry for the ping. @kernfeld-cockroach, this should be resolved by #66833, which is contained in 21.1.7. I'll close this but if it reoccurs please re-open. Thanks for filing!

[kernfeld-cockroach]

That did fix it, thanks!