fatal error: concurrent map read and map write in Optimistic Sharding DDL

[GMHDBJD]

What did you do?

fatal error: concurrent map read and map write

goroutine 37885 [running]:
runtime.throw(0x24027ce, 0x21)
	/usr/local/go/src/runtime/panic.go:1117 +0x72 fp=0xc08f804f40 sp=0xc08f804f10 pc=0x438592
runtime.mapaccess2(0x20897e0, 0xc09ce4e2d0, 0xc04d162d40, 0xc04d162d40, 0xc04100c97f)
	/usr/local/go/src/runtime/map.go:469 +0x255 fp=0xc08f804f80 sp=0xc08f804f40 pc=0x40f915
reflect.mapaccess(0x20897e0, 0xc09ce4e2d0, 0xc04d162d40, 0x23e208f)
	/usr/local/go/src/runtime/map.go:1318 +0x3f fp=0xc08f804fb8 sp=0xc08f804f80 pc=0x46bb3f
reflect.Value.MapIndex(0x20897e0, 0xc09ce4e2d0, 0x15, 0x1ffd7a0, 0xc04d162d40, 0x98, 0x20898a0, 0xc09ce4f710, 0x15)
	/usr/local/go/src/reflect/value.go:1189 +0x16e fp=0xc08f805030 sp=0xc08f804fb8 pc=0x4c8d0e
encoding/json.mapEncoder.encode(0xc0722c59c0, 0xc06341c100, 0x20897e0, 0xc09ce4e2d0, 0x15, 0x100)
	/usr/local/go/src/encoding/json/encode.go:813 +0x317 fp=0xc08f8051d8 sp=0xc08f805030 pc=0x52f237
encoding/json.mapEncoder.encode-fm(0xc06341c100, 0x20897e0, 0xc09ce4e2d0, 0x15, 0xc04d160100)
	/usr/local/go/src/encoding/json/encode.go:779 +0x6d fp=0xc08f805218 sp=0xc08f8051d8 pc=0x53b9ed
encoding/json.mapEncoder.encode(0xc0722c59e0, 0xc06341c100, 0x2089780, 0xc09764a560, 0x95, 0x100)
	/usr/local/go/src/encoding/json/encode.go:813 +0x366 fp=0xc08f8053c0 sp=0xc08f805218 pc=0x52f286
encoding/json.mapEncoder.encode-fm(0xc06341c100, 0x2089780, 0xc09764a560, 0x95, 0x100)
	/usr/local/go/src/encoding/json/encode.go:779 +0x6d fp=0xc08f805400 sp=0xc08f8053c0 pc=0x53b9ed
encoding/json.structEncoder.encode(0xc0722e2900, 0x3, 0x4, 0xc0722ff020, 0xc06341c100, 0x221e7c0, 0xc09764a540, 0x99, 0x2210100)
	/usr/local/go/src/encoding/json/encode.go:761 +0x2ab fp=0xc08f805498 sp=0xc08f805400 pc=0x52eccb
encoding/json.structEncoder.encode-fm(0xc06341c100, 0x221e7c0, 0xc09764a540, 0x99, 0x3e10100)
	/usr/local/go/src/encoding/json/encode.go:732 +0x8c fp=0xc08f8054f0 sp=0xc08f805498 pc=0x53b94c
encoding/json.(*encodeState).reflectValue(0xc06341c100, 0x221e7c0, 0xc09764a540, 0x99, 0xc08f800100)
	/usr/local/go/src/encoding/json/encode.go:360 +0x82 fp=0xc08f805528 sp=0xc08f8054f0 pc=0x52c342
encoding/json.(*encodeState).marshal(0xc06341c100, 0x221e7c0, 0xc09764a540, 0x100, 0x0, 0x0)
	/usr/local/go/src/encoding/json/encode.go:332 +0xf9 fp=0xc08f805588 sp=0xc08f805528 pc=0x52bf39
encoding/json.Marshal(0x221e7c0, 0xc09764a540, 0x221e7c0, 0xc09764a540, 0x2cb93c8, 0x76, 0x2cb93c8)
	/usr/local/go/src/encoding/json/encode.go:161 +0x52 fp=0xc08f805600 sp=0xc08f805588 pc=0x52b3b2
github.com/pingcap/dm/pkg/shardddl/optimism.SourceTables.toJSON(...)
	/home/jenkins/agent/workspace/build_dm_multi_branch_v2.0.7/go/src/github.com/pingcap/dm/pkg/shardddl/optimism/table.go:100
github.com/pingcap/dm/pkg/shardddl/optimism.SourceTables.String(0xc080bdd698, 0x8, 0xc06a95ce20, 0x1a, 0xc09ce4e2a0, 0x0, 0x4565f9, 0x3743fb8)
	/home/jenkins/agent/workspace/build_dm_multi_branch_v2.0.7/go/src/github.com/pingcap/dm/pkg/shardddl/optimism/table.go:94 +0x77 fp=0xc08f805678 sp=0xc08f805600 pc=0x1e8ed97
github.com/pingcap/dm/pkg/shardddl/optimism.(*SourceTables).String(0xc09764a510, 0x221e7c0, 0xc09764a510)
	<autogenerated>:1 +0x77 fp=0xc08f8056f8 sp=0xc08f805678 pc=0x1e929f7
go.uber.org/zap/zapcore.encodeStringer(0x23c9577, 0xd, 0x221e7c0, 0xc09764a510, 0x2799658, 0xc0808d89c0, 0x0, 0x0)
	/nfs/cache/mod/go.uber.org/zap@v1.19.0/zapcore/field.go:231 +0xf0 fp=0xc08f805788 sp=0xc08f8056f8 pc=0x550b30
go.uber.org/zap/zapcore.Field.AddTo(0x23c9577, 0xd, 0x19, 0x0, 0x0, 0x0, 0x221e7c0, 0xc09764a510, 0x2799658, 0xc0808d89c0)
	/nfs/cache/mod/go.uber.org/zap@v1.19.0/zapcore/field.go:174 +0xcab fp=0xc08f805860 sp=0xc08f805788 pc=0x55038b
github.com/pingcap/log.(*textEncoder).addFields(0xc0808d89c0, 0xc03b7eb380, 0x3, 0x3)
	/nfs/cache/mod/github.com/pingcap/log@v0.0.0-20210906054005-afc726e70354/zap_text_encoder.go:640 +0x152 fp=0xc08f805928 sp=0xc08f805860 pc=0x87a052
github.com/pingcap/log.(*textEncoder).EncodeEntry(0xc0003d91c0, 0x0, 0xc0648abc03c0e50c, 0x8e27e925ce854, 0x3e17740, 0x0, 0x0, 0x23df159, 0x15, 0x1, ...)
	/nfs/cache/mod/github.com/pingcap/log@v0.0.0-20210906054005-afc726e70354/zap_text_encoder.go:466 +0x5d6 fp=0xc08f805a88 sp=0xc08f805928 pc=0x877bb6
github.com/pingcap/log.(*textIOCore).Write(0xc0004d27e0, 0x0, 0xc0648abc03c0e50c, 0x8e27e925ce854, 0x3e17740, 0x0, 0x0, 0x23df159, 0x15, 0x1, ...)
	/nfs/cache/mod/github.com/pingcap/log@v0.0.0-20210906054005-afc726e70354/zap_text_core.go:64 +0xa9 fp=0xc08f805b78 sp=0xc08f805a88 pc=0x873a29
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc08945c000, 0xc03b7eb380, 0x3, 0x3)
	/nfs/cache/mod/go.uber.org/zap@v1.19.0/zapcore/entry.go:220 +0x12d fp=0xc08f805d20 sp=0xc08f805b78 pc=0x54e7ed
go.uber.org/zap.(*Logger).Info(0xc00040b740, 0x23df159, 0x15, 0xc03b7eb380, 0x3, 0x3)
	/nfs/cache/mod/go.uber.org/zap@v1.19.0/logger.go:192 +0x85 fp=0xc08f805d58 sp=0xc08f805d20 pc=0x76a185
github.com/pingcap/dm/dm/master/shardddl.(*Optimist).handleSourceTables(0xc00081b090, 0x276a310, 0xc09c12f780, 0xc09c138660)
	/home/jenkins/agent/workspace/build_dm_multi_branch_v2.0.7/go/src/github.com/pingcap/dm/dm/master/shardddl/optimist.go:508 +0x75 fp=0xc08f805f70 sp=0xc08f805d58 pc=0x1e98f75
github.com/pingcap/dm/dm/master/shardddl.(*Optimist).watchSourceInfoOperation.func3(0xc08527c320, 0xc00081b090, 0x276a310, 0xc09c12f780, 0xc09c138660)
	/home/jenkins/agent/workspace/build_dm_multi_branch_v2.0.7/go/src/github.com/pingcap/dm/dm/master/shardddl/optimist.go:456 +0x75 fp=0xc08f805fb8 sp=0xc08f805f70 pc=0x1eaaf15
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc08f805fc0 sp=0xc08f805fb8 pc=0x472301
created by github.com/pingcap/dm/dm/master/shardddl.(*Optimist).watchSourceInfoOperation
	/home/jenkins/agent/workspace/build_dm_multi_branch_v2.0.7/go/src/github.com/pingcap/dm/dm/master/shardddl/optimist.go:454 +0x1fb

What did you expect to see?

no panic

What did you see instead?

panic

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

v2.0.7

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)

[GMHDBJD]

Root Cause:
handleSourceTablePut read SourceTable while handleInfoPut write SourceTable
https://github.com/pingcap/ticdc/blob/9721053edae6b146b726bd5daec04a0dc8c57b1a/dm/dm/master/shardddl/optimist.go#L508
https://github.com/pingcap/ticdc/blob/9721053edae6b146b726bd5daec04a0dc8c57b1a/dm/pkg/shardddl/optimism/keeper.go#L243