fatal error: concurrent map writes, HashJoin on temporary table

[tiancaiamao]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Not sure how to reproduce.. here is the panic stack:

fatal error: concurrent map writes

goroutine 16084384107 [running]:
github.com/pingcap/tidb/sessionctx/variable.(*SessionVars).GetTemporaryTable(0x402d991408?, 0x40008fc700)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessionctx/variable/session.go:2298 +0xd4
github.com/pingcap/tidb/table/tables.(*TableCommon).Allocators(0x4001cc4b40, {0x531e3f8, 0x46251bf680})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/table/tables/tables.go:1550 +0x7c
github.com/pingcap/tidb/executor.getAutoIncrementID({0x531e3f8, 0x46251bf680}, 0x4339266a80, 0x40008fc700)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:389 +0xdc
github.com/pingcap/tidb/executor.(*memtableRetriever).setDataFromTables(0x42a5a7baa0, {0x52aca90?, 0x44f1754a50?}, {0x531e3f8, 0x46251bf680?}, {0x45d7eb6800, 0xe, 0x1?})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:689 +0x690
github.com/pingcap/tidb/executor.(*memtableRetriever).retrieve(0x42a5a7baa0, {0x52aca90, 0x44f1754a50}, {0x531e3f8, 0x46251bf680})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:111 +0x1e8
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next(0x405ebfdc20, {0x52aca90, 0x44f1754a50}, 0x4069e0fc20)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/memtable_reader.go:118 +0x210
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b17e0, 0x405ebfdc20}, 0x4069e0fc20)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*SelectionExec).Next(0x5ae73ec460, {0x52aca90, 0x44f1754a50}, 0x4558ba57c0)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1596 +0xbc
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b1c60, 0x5ae73ec460}, 0x4558ba57c0)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*buildWorker).fetchBuildSideRows(0x4069e0f900, {0x52aca90, 0x44f1754a50}, 0x4124a3a240, 0x4124a3a2a0, 0x44fdfb5140)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:332 +0x114
github.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndBuildHashTable.func2()
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:1178 +0xa8
github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover.func1()
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:51 +0x64
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:43 +0x9c

goroutine 1 [chan receive, 68928 minutes]:
github.com/pingcap/tidb/server.(*Server).Run(0x4007395b00)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:402 +0x20c
main.main()

goroutine 110 [select]:
github.com/tikv/client-go/v2/internal/locate.(*RegionCache).cacheGC(0x40014274a0)
        /go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20231121073938-194639470f84/internal/locate/region_cache.go:2013 +0x144
created by github.com/tikv/client-go/v2/internal/locate.NewRegionCache
        /go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20231121073938-194639470f84/internal/locate/region_cache.go:433 +0x350

goroutine 101 [select]:
go.opencensus.io/stats/view.(*worker).start(0x4000686100)
        /go/pkg/mod/go.opencensus.io@v0.23.1-0.20220331163232-052120675fac/stats/view/worker.go:292 +0x88
created by go.opencensus.io/stats/view.init.0
        /go/pkg/mod/go.opencensus.io@v0.23.1-0.20220331163232-052120675fac/stats/view/worker.go:34 +0xa4

goroutine 49 [select]:
github.com/tikv/pd/client.(*baseClient).memberLoop(0x4000c534a0)
        /go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20230904040343-947701a32c05/base_client.go:139 +0x140
created by github.com/tikv/pd/client.(*baseClient).init
        /go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20230904040343-947701a32c05/base_client.go:109 +0x164

goroutine 114 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
        /go/pkg/mod/github.com/golang/glog@v1.0.0/glog.go:882 +0x5c
created by github.com/golang/glog.init.0
        /go/pkg/mod/github.com/golang/glog@v1.0.0/glog.go:410 +0x200

goroutine 321 [chan receive, 822 minutes]:
gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun(0x4000fc3f20)
        /go/pkg/mod/gopkg.in/natefinch/lumberjack.v2@v2.0.0/lumberjack.go:379 +0x44
created by gopkg.in/natefinch/lumberjack%2ev2.(*Logger).mill.func1
        /go/pkg/mod/gopkg.in/natefinch/lumberjack.v2@v2.0.0/lumberjack.go:390 +0xa4

2. What did you expect to see? (Required)

no panic

3. What did you see instead (Required)

panic

Master should have the same bug.

tidb/pkg/sessionctx/variable/session.go

Line 2775 in 240702e

tempTables[tblInfo.ID] = tempTable

4. What is your TiDB version? (Required)

v6.5.6-20240517-4b81451

[lcwangchao]

Seems information_schema.table is joining it self and caused the data race:

goroutine 16084384107 [running]:
github.com/pingcap/tidb/sessionctx/variable.(*SessionVars).GetTemporaryTable(0x402d991408?, 0x40008fc700)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessionctx/variable/session.go:2298 +0xd4
github.com/pingcap/tidb/table/tables.(*TableCommon).Allocators(0x4001cc4b40, {0x531e3f8, 0x46251bf680})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/table/tables/tables.go:1550 +0x7c
github.com/pingcap/tidb/executor.getAutoIncrementID({0x531e3f8, 0x46251bf680}, 0x4339266a80, 0x40008fc700)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:389 +0xdc
github.com/pingcap/tidb/executor.(*memtableRetriever).setDataFromTables(0x42a5a7baa0, {0x52aca90?, 0x44f1754a50?}, {0x531e3f8, 0x46251bf680?}, {0x45d7eb6800, 0xe, 0x1?})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:689 +0x690
github.com/pingcap/tidb/executor.(*memtableRetriever).retrieve(0x42a5a7baa0, {0x52aca90, 0x44f1754a50}, {0x531e3f8, 0x46251bf680})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:111 +0x1e8
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next(0x405ebfdc20, {0x52aca90, 0x44f1754a50}, 0x4069e0fc20)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/memtable_reader.go:118 +0x210
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b17e0, 0x405ebfdc20}, 0x4069e0fc20)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*SelectionExec).Next(0x5ae73ec460, {0x52aca90, 0x44f1754a50}, 0x4558ba57c0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1596 +0xbc
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b1c60, 0x5ae73ec460}, 0x4558ba57c0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*buildWorker).fetchBuildSideRows(0x4069e0f900, {0x52aca90, 0x44f1754a50}, 0x4124a3a240, 0x4124a3a2a0, 0x44fdfb5140)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:332 +0x114
github.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndBuildHashTable.func2()
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:1178 +0xa8
github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover.func1()
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:51 +0x64
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:43 +0x9c



goroutine 16084384202 [runnable]:
github.com/pingcap/tidb/statistics.PseudoTable(0x40008fcc40)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/table.go:1117 +0x50
github.com/pingcap/tidb/table/tables.TempTableFromMeta(0x40008fcc40)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/table/tables/tables.go:2025 +0x20
github.com/pingcap/tidb/sessionctx/variable.(*SessionVars).GetTemporaryTable(0x404bb6d438?, 0x40008fcc40)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessionctx/variable/session.go:2297 +0xb4
github.com/pingcap/tidb/table/tables.(*TableCommon).Allocators(0x4001cc4ea0, {0x531e3f8, 0x46251bf680})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/table/tables/tables.go:1550 +0x7c
github.com/pingcap/tidb/executor.getAutoIncrementID({0x531e3f8, 0x46251bf680}, 0x4339266a80, 0x40008fcc40)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:389 +0xdc
github.com/pingcap/tidb/executor.(*memtableRetriever).setDataFromTables(0x42a5a7ba40, {0x52aca90?, 0x44f1754a50?}, {0x531e3f8, 0x46251bf680?}, {0x444393d880, 0xe, 0x0?})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:689 +0x690
github.com/pingcap/tidb/executor.(*memtableRetriever).retrieve(0x42a5a7ba40, {0x52aca90, 0x44f1754a50}, {0x531e3f8, 0x46251bf680})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:111 +0x1e8
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next(0x405ebfdb80, {0x52aca90, 0x44f1754a50}, 0x4069e0fbd0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/memtable_reader.go:118 +0x210
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b17e0, 0x405ebfdb80}, 0x4069e0fbd0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*SelectionExec).Next(0x5ae73ec380, {0x52aca90, 0x44f1754a50}, 0x40bb626820)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1596 +0xbc
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b1c60, 0x5ae73ec380}, 0x40bb626820)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*probeSideTupleFetcher).fetchProbeSideChunks(0x43468aa940, {0x52aca90, 0x44f1754a50}, 0x400)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:253 +0x134
github.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndProbeHashTable.func1()
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:390 +0x84
github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover.func1()
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:51 +0x64
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:43 +0x9c