tidb connection panic after fault recover from one of tidb network partition

[CabinfeverB]

What did you do?
1、run tpcc
2、inject one of tidb network partition last for 10mins and recover

What did you expect to see?
no panic

What did you see instead?
tidb connection panic after fault recover

[2024/03/16 12:02:50.582 +08:00] [ERROR] [conn.go:1150] ["connection running loop panic"] [keyspaceName=keyspace_a] [keyspaceID=1] [conn=4069787314631476933] [gw_conn=] [lastSQL="update district set d_ytd = d_ytd + ? where d_w_id = ? and d_id = ?"] [err="inconsistent label cardinality: expected 2 label values but got 1 in []string{"default"}"] [stack="github.com/pingcap/tidb/server.(*clientConn).Run.func1\n\t/go/src/github.com/pingcap/tidb/server/conn.go:1153\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec.func1\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:463\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\ngithub.com/prometheus/client_golang/prometheus.(*CounterVec).WithLabelValues\n\t/go/cache/github.com/prometheus/client_golang@v1.18.0/prometheus/counter.go:284\ngithub.com/tikv/pd/client/resource_group/controller.(*ResourceGroupsController).IsBackgroundRequest\n\t/go/cache/github.com/tikv/pd/client@v0.0.0-20240314040809-c2c5d849cef5/resource_group/controller/controller.go:567\ngithub.com/tikv/client-go/v2/internal/client.buildResourceControlInterceptor.func1.1\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/client/client_interceptor.go:104\ngithub.com/tikv/client-go/v2/internal/client.interceptedClient.SendRequest\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/client/client_interceptor.go:58\ngithub.com/tikv/client-go/v2/internal/client.reqCollapse.SendRequest\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/client/client_collapse.go:74\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).sendReqToRegion\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/locate/region_request.go:1712\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReqCtx\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/locate/region_request.go:1470\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionRequestSender).SendReq\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/internal/locate/region_request.go:252\ngithub.com/tikv/client-go/v2/txnkv/transaction.actionPessimisticLock.handleSingleBatch\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/pessimistic.go:189\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnBatches\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/2pc.go:1084\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnGroupMutations\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/2pc.go:989\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).doActionOnMutations\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/2pc.go:808\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).pessimisticLockMutations\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/pessimistic.go:601\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).lockKeys\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/txn.go:1107\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).LockKeysFunc\n\t/go/cache/github.com/tikv/client-go/v2@v2.0.8-0.20240314054243-d5fcee418f95/txnkv/transaction/txn.go:919\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).LockKeysFunc\n\t/go/src/github.com/pingcap/tidb/store/driver/txn/txn_driver.go:91\ngithub.com/pingcap/tidb/session.(*LazyTxn).LockKeysFunc\n\t/go/src/github.com/pingcap/tidb/session/txn.go:477\ngithub.com/pingcap/tidb/session.(*LazyTxn).LockKeys\n\t/go/src/github.com/pingcap/tidb/session/txn.go:452\ngithub.com/pingcap/tidb/executor.doLockKeys\n\t/go/src/github.com/pingcap/tidb/executor/executor.go:1369\ngithub.com/pingcap/tidb/executor.(*PointGetExecutor).lockKeyBase\n\t/go/src/github.com/pingcap/tidb/executor/point_get.go:409\ngithub.com/pingcap/tidb/executor.(*PointGetExecutor).lockKeyIfNeeded\n\t/go/src/github.com/pingcap/tidb/executor/point_get.go:384\ngithub.com/pingcap/tidb/executor.(*PointGetExecutor).Next\n\t/go/src/github.com/pingcap/tidb/executor/point_get.go:256\ngithub.com/pingcap/tidb/executor.Next\n\t/go/src/github.com/pingcap/tidb/executor/executor.go:327\ngithub.com/pingcap/tidb/executor.(*UpdateExec).updateRows\n\t/go/src/github.com/pingcap/tidb/executor/update.go:265\ngithub.com/pingcap/tidb/executor.(*UpdateExec).Next\n\t/go/src/github.com/pingcap/tidb/executor/update.go:239\ngithub.com/pingcap/tidb/executor.Next\n\t/go/src/github.com/pingcap/tidb/executor/executor.go:327\ngithub.com/pingcap/tidb/executor.(*ExecStmt).next\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:1202\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:947\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handlePessimisticDML\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:1008\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:770\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/go/src/github.com/pingcap/tidb/executor/adapter.go:568\ngithub.com/pingcap/tidb/session.runStmt\n\t/go/src/github.com/pingcap/tidb/session/session.go:2413\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/go/src/github.com/pingcap/tidb/session/session.go:2262\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/go/src/github.com/pingcap/tidb/server/driver_tidb.go:294\ngithub.com/pingcap/tidb/server.(*clientConn).executePreparedStmtAndWriteResult\n\t/go/src/github.com/pingcap/tidb/server/conn_stmt.go:308\ngithub.com/pingcap/tidb/server.(*clientConn).executePlanCacheStmt\n\t/go/src/github.com/pingcap/tidb/server/conn_stmt.go:236\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmtExecute\n\t/go/src/github.com/pingcap/tidb/server/conn_stmt.go:227\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/go/src/github.com/pingcap/tidb/server/conn.go:1483\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/go/src/github.com/pingcap/tidb/server/conn.go:1234\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/go/src/github.com/pingcap/tidb/server/server.go:755"]
[2024/03/16 12:02:50.582 +08:00] [INFO] [server.go:733] ["connection closed"] [keyspaceName=keyspace_a] [keyspaceID=1] [conn=4069787314631476933] [gw_conn=] [remoteAddr=10.233.103.73:50984] [connectionID=4069787314631476933]

What version of PD are you using (pd-server -V)?
./pd-server -V
Release Version: v7.1.1-serverless
Edition: Community
Git Commit Hash: https://github.com/tidbcloud/pd-cse/commit/a502745f1d93745e03cab261619d1c3d3c8d4e22
Git Branch: release-7.1-keyspace
UTC Build Time: 2024-03-12 07:25:44
2024-03-16T11:42:40.444+0800 INFO k8s/client.go:135

[CabinfeverB]

ref https://github.com/tidbcloud/pd-cse/issues/275

[seiya-annie]

/report customer