too many Information schema is out of date: schema failed to update in 1 lease in the test

[hawkingrei]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

[2023/02/11 14:39:51.605 +08:00] [WARN] [session.go:988] ["can not retry txn"] [label=internal] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"] [errorVerbose="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).checkSchemaValid\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1746\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).execute\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1585\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).Commit\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/txn.go:496\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:95\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.cleanMDLInfo\n\tddl/ddl_worker.go:461\ngithub.com/pingcap/tidb/ddl.(*ddl).delivery2worker.func1\n\tddl/job_table.go:295\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594\ngithub.com/pingcap/errors.AddStack\n\texternal/com_github_pingcap_errors/errors.go:174\ngithub.com/pingcap/errors.Trace\n\texternal/com_github_pingcap_errors/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/driver/error.ToTiDBErr\n\tstore/driver/error/error.go:166\ngithub.com/pingcap/tidb/store/driver/txn.extractKeyErr\n\tstore/driver/txn/error.go:162\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).extractKeyErr\n\tstore/driver/txn/txn_driver.go:308\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:96\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.cleanMDLInfo\n\tddl/ddl_worker.go:461\ngithub.com/pingcap/tidb/ddl.(*ddl).delivery2worker.func1\n\tddl/job_table.go:295\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594"] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=true] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2023/02/11 14:39:51.606 +08:00] [WARN] [session.go:1004] ["commit failed"] ["finished txn"="Txn{state=invalid}"] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"] [errorVerbose="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).checkSchemaValid\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1746\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).execute\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1585\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).Commit\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/txn.go:496\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:95\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.cleanMDLInfo\n\tddl/ddl_worker.go:461\ngithub.com/pingcap/tidb/ddl.(*ddl).delivery2worker.func1\n\tddl/job_table.go:295\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594\ngithub.com/pingcap/errors.AddStack\n\texternal/com_github_pingcap_errors/errors.go:174\ngithub.com/pingcap/errors.Trace\n\texternal/com_github_pingcap_errors/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/driver/error.ToTiDBErr\n\tstore/driver/error/error.go:166\ngithub.com/pingcap/tidb/store/driver/txn.extractKeyErr\n\tstore/driver/txn/error.go:162\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).extractKeyErr\n\tstore/driver/txn/txn_driver.go:308\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:96\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.cleanMDLInfo\n\tddl/ddl_worker.go:461\ngithub.com/pingcap/tidb/ddl.(*ddl).delivery2worker.func1\n\tddl/job_table.go:295\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594"]
[2023/02/11 14:39:51.608 +08:00] [WARN] [adapter.go:1672] ["# Txn_start_ts: 439378925698154496\n# Query_time: 5.17800344\n# Parse_time: 0\n# Compile_time: 0.00485737\n# Rewrite_time: 0\n# Optimize_time: 0\n# Wait_TS: 0.00012195\n# Prewrite_time: 0.000495006 Get_commit_ts_time: 0.000024441 Slowest_prewrite_rpc_detail: {total:0.000s, region_id: 2, store: store1, } Write_keys: 1 Write_size: 19 Prewrite_region: 1\n# Is_internal: true\n# Digest: bdab55af0288282745b06ae511bf5258d90e46fca67dd2e2ef0faca225d2d599\n# Num_cop_tasks: 0\n# Mem_max: 8587\n# Prepared: false\n# Plan_from_cache: false\n# Plan_from_binding: false\n# Has_more_results: false\n# KV_total: 0\n# PD_total: 0\n# Backoff_total: 0\n# Write_sql_response_total: 0\n# Result_rows: 0\n# Succ: false\n# IsExplicitTxn: false\n# IsSyncStatsFailed: false\n# Plan: tidb_decode_plan('zgLwQDAJMjlfMgkwCTAJTi9BCTAJdGltZTo5MzAuNcK1cywgbG9vcHM6MSwgY29tbWl0X3R4bjoge3ByZXdyaXRlOjQ5CScMZ2V0XxEiFHM6MjQuNAU+IHNsb3dlc3RfcA0w0F9ycGM6IHt0b3RhbDogMC4wMDBzLCByZWdpb25faWQ6IDIsIHN0b3JlOiBzdG9yZTEsIH0sESAIbnVtAYkFeRBfa2V5cxkO8ENieXRlOjE5fQkzOTUgQnl0ZXMJTi9BCjEJMzZfMQkwCTEJdGFibGU6dGlkYl9tZGxfaW5mbywgaGFuZGxlOjQ3CTEJdAH4CDQ1MTL4ACgyLCBHZXQ6e251bQW7CDEsIAW8AF8lJkQxMzkuM8K1c30JTi9BCU4vQQo=')\n# Plan_digest: ca2b8c61a8ea82f17bacb18dbe38cd4b008fc18663d28963f5ced78ed80210d7\ndelete from mysql.tidb_mdl_info where job_id = 47;"]
[2023/02/11 14:39:51.608 +08:00] [WARN] [session.go:2211] ["run statement failed"] [schemaVersion=27] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"] [errorVerbose="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).checkSchemaValid\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1746\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).execute\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1585\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).Commit\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/txn.go:496\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:95\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.cleanMDLInfo\n\tddl/ddl_worker.go:461\ngithub.com/pingcap/tidb/ddl.(*ddl).delivery2worker.func1\n\tddl/job_table.go:295\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594\ngithub.com/pingcap/errors.AddStack\n\texternal/com_github_pingcap_errors/errors.go:174\ngithub.com/pingcap/errors.Trace\n\texternal/com_github_pingcap_errors/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/driver/error.ToTiDBErr\n\tstore/driver/error/error.go:166\ngithub.com/pingcap/tidb/store/driver/txn.extractKeyErr\n\tstore/driver/txn/error.go:162\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).extractKeyErr\n\tstore/driver/txn/txn_driver.go:308\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:96\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.cleanMDLInfo\n\tddl/ddl_worker.go:461\ngithub.com/pingcap/tidb/ddl.(*ddl).delivery2worker.func1\n\tddl/job_table.go:295\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594"] [session="{\n  \"currDBName\": \"\",\n  \"id\": 0,\n  \"status\": 2,\n  \"strictMode\": true,\n  \"user\": null\n}"]
[2023/02/11 14:39:51.610 +08:00] [WARN] [ddl_worker.go:463] ["unexpected error when clean mdl info"] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"] [errorVerbose="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).checkSchemaValid\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1746\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).execute\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1585\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).Commit\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/txn.go:496\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:95\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.cleanMDLInfo\n\tddl/ddl_worker.go:461\ngithub.com/pingcap/tidb/ddl.(*ddl).delivery2worker.func1\n\tddl/job_table.go:295\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594\ngithub.com/pingcap/errors.AddStack\n\texternal/com_github_pingcap_errors/errors.go:174\ngithub.com/pingcap/errors.Trace\n\texternal/com_github_pingcap_errors/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/driver/error.ToTiDBErr\n\tstore/driver/error/error.go:166\ngithub.com/pingcap/tidb/store/driver/txn.extractKeyErr\n\tstore/driver/txn/error.go:162\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).extractKeyErr\n\tstore/driver/txn/txn_driver.go:308\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:96\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.cleanMDLInfo\n\tddl/ddl_worker.go:461\ngithub.com/pingcap/tidb/ddl.(*ddl).delivery2worker.func1\n\tddl/job_table.go:295\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594"]
[2023/02/11 14:39:51.639 +08:00] [WARN] [session.go:988] ["can not retry txn"] [label=internal] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"] [errorVerbose="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).checkSchemaValid\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1746\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).execute\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1585\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).Commit\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/txn.go:496\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:95\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.insertDDLJobs2Table\n\tddl/job_table.go:462\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs2Table\n\tddl/ddl_worker.go:379\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs\n\tddl/ddl_worker.go:218\ngithub.com/pingcap/tidb/ddl.(*ddl).limitDDLJobs\n\tddl/ddl_worker.go:204\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594\ngithub.com/pingcap/errors.AddStack\n\texternal/com_github_pingcap_errors/errors.go:174\ngithub.com/pingcap/errors.Trace\n\texternal/com_github_pingcap_errors/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/driver/error.ToTiDBErr\n\tstore/driver/error/error.go:166\ngithub.com/pingcap/tidb/store/driver/txn.extractKeyErr\n\tstore/driver/txn/error.go:162\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).extractKeyErr\n\tstore/driver/txn/txn_driver.go:308\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:96\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.insertDDLJobs2Table\n\tddl/job_table.go:462\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs2Table\n\tddl/ddl_worker.go:379\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs\n\tddl/ddl_worker.go:218\ngithub.com/pingcap/tidb/ddl.(*ddl).limitDDLJobs\n\tddl/ddl_worker.go:204\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594"] [IsBatchInsert=false] [IsPessimistic=false] [InRestrictedSQL=true] [tidb_retry_limit=10] [tidb_disable_txn_auto_retry=true]
[2023/02/11 14:39:51.640 +08:00] [WARN] [session.go:1004] ["commit failed"] ["finished txn"="Txn{state=invalid}"] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"] [errorVerbose="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).checkSchemaValid\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1746\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).execute\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1585\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).Commit\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/txn.go:496\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:95\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.insertDDLJobs2Table\n\tddl/job_table.go:462\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs2Table\n\tddl/ddl_worker.go:379\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs\n\tddl/ddl_worker.go:218\ngithub.com/pingcap/tidb/ddl.(*ddl).limitDDLJobs\n\tddl/ddl_worker.go:204\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594\ngithub.com/pingcap/errors.AddStack\n\texternal/com_github_pingcap_errors/errors.go:174\ngithub.com/pingcap/errors.Trace\n\texternal/com_github_pingcap_errors/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/driver/error.ToTiDBErr\n\tstore/driver/error/error.go:166\ngithub.com/pingcap/tidb/store/driver/txn.extractKeyErr\n\tstore/driver/txn/error.go:162\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).extractKeyErr\n\tstore/driver/txn/txn_driver.go:308\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:96\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.insertDDLJobs2Table\n\tddl/job_table.go:462\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs2Table\n\tddl/ddl_worker.go:379\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs\n\tddl/ddl_worker.go:218\ngithub.com/pingcap/tidb/ddl.(*ddl).limitDDLJobs\n\tddl/ddl_worker.go:204\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594"]
[2023/02/11 14:39:51.643 +08:00] [WARN] [adapter.go:1672] ["# Txn_start_ts: 439378925735116800\n# Query_time: 5.086321539\n# Parse_time: 0\n# Compile_time: 0.020112661\n# Rewrite_time: 0.00027956\n# Optimize_time: 0.001770133\n# Wait_TS: 0.000148824\n# Prewrite_time: 0.000740735 Get_commit_ts_time: 0.000021666 Slowest_prewrite_rpc_detail: {total:0.001s, region_id: 2, store: store1, } Write_keys: 1 Write_size: 4172 Prewrite_region: 1\n# Is_internal: true\n# Digest: 1b73a00c73a03d7dc9dd08a1ea99474b2523e5f72366f740be5156fa244256c4\n# Num_cop_tasks: 0\n# Mem_max: 16920\n# Prepared: false\n# Plan_from_cache: false\n# Plan_from_binding: false\n# Has_more_results: false\n# KV_total: 0\n# PD_total: 0\n# Backoff_total: 0\n# Write_sql_response_total: 0\n# Result_rows: 0\n# Succ: false\n# IsExplicitTxn: false\n# IsSyncStatsFailed: false\n# Plan: tidb_decode_plan('gwLYMAkyN18xCTAJMAlOL0EJMAl0aW1lOjkyNC42wrVzLCBsb29wczoxLCBwcmVwYXJlOiA0NTMuOAUcJGluc2VydDo0NzANEWhjb21taXRfdHhuOiB7cHJld3JpdGU6NzQwLjcFMQxnZXRfESQMczoyMQ0XHHNsb3dlc3RfETLQX3JwYzoge3RvdGFsOiAwLjAwMXMsIHJlZ2lvbl9pZDogMiwgc3RvcmU6IHN0b3JlMSwgfSwRIBhudW06MSwgBXsQX2tleXMZDlhieXRlOjQxNzJ9CTQuNTIgS0IJTi9BCg==')\n\"insert into mysql.tidb_ddl_job(job_id, reorg, schema_ids, table_ids, job_meta, type, processing) values(49, false, \\\"1\\\", \\\"48\\\", 0x7b226964223a34392c2274797065223a332c22736368656d615f6964223a312c227461626c655f6964223a34382c22736368656d615f6e616d65223a226d7973716c222c227461626c655f6e616d65223a22736368656d615f696e6465785f7573616765222c227374617465223a382c227761726e696e67223a6e756c6c2c22657272223a6e756c6c2c226572725f636f756e74223a302c22726f775f636f756e74223a302c227261775f61726773223a5b7b226964223a34382c226e616d65223a7b224f223a22736368656d615f696e6465785f7573616765222c224c223a22736368656d615f696e6465785f7573616765227d2c2263686172736574223a22757466386d6234222c22636f6c6c617465223a22757466386d62345f62696e222c22636f6c73223a5b7b226964223a312c226e616d65223a7b224f223a225441424c455f4944222c224c223a227461626c655f6964227d2c226f6666736574223a302c226f726967696e5f64656661756c74223a6e756c6c2c226f726967696e5f64656661756c745f626974223a6e756c6c2c2264656661756c74223a6e756c6c2c2264656661756c745f626974223a6e756c6c2c2264656661756c745f69735f65787072223a66616c73652c2267656e6572617465645f657870725f737472696e67223a22222c2267656e6572617465645f73746f726564223a66616c73652c22646570656e64656e636573223a6e756c6c2c2274797065223a7b225470223a382c22466c6167223a343039392c22466c656e223a36342c22446563696d616c223a302c2243686172736574223a2262696e617279222c22436f6c6c617465223a2262696e617279222c22456c656d73223a6e756c6c2c22456c656d73497342696e6172794c6974223a6e756c6c2c224172726179223a66616c73657d2c227374617465223a352c22636f6d6d656e74223a22222c2268696464656e223a66616c73652c226368616e67655f73746174655f696e666f223a6e756c6c2c2276657273696f6e223a327d2c7b226964223a322c226e616d65223a7b224f223a22494e4445585f4944222c224c223a22696e6465785f6964227d2c226f6666736574223a312c226f726967696e5f64656661756c74223a6e756c6c2c226f726967696e5f64656661756c745f626974223a6e756c6c2c2264656661756c74223a6e756c6c2c2264656661756c745f626974223a6e756c6c2c2264656661756c745f69735f65787072223a66616c73652c2267656e6572617465645f657870725f737472696e67223a22222c2267656e6572617465645f73746f726564223a66616c73652c22646570656e64656e636573223a6e756c6c2c2274797065223a7b225470223a382c22466c6167223a343039392c22466c656e223a32312c22446563696d616c223a302c2243686172736574223a2262696e617279222c22436f6c6c617465223a2262696e617279222c22456c656d73223a6e756c6c2c22456c656d73497342696e6172794c6974223a6e756c6c2c224172726179223a66616c73657d2c227374617465223a352c22636f6d6d656e74223a22222c2268696464656e223a66616c73652c226368616e67655f73746174655f696e666f223a6e756c6c2c2276657273696f6e223a327d2c7b226964223a332c226e616d65223a7b224f223a2251554552595f434f554e54222c224c223a2271756572795f636f756e74227d2c226f6666736574223a322c226f726967696e5f64656661756c74223a6e756c6c2c226f726967696e5f64656661756c745f626974223a6e756c6c2c2264656661756c74223a6e756c6c2c2264656661756c745f626974223a6e756c6c2c2264656661756c745f69735f65787072223a66616c73652c2267656e6572617465645f657870725f737472696e67223a22222c2267656e6572617465645f73746f726564223a66616c73652c22646570656e64656e636573223a6e756c6c2c2274797065223a7b225470223a382c22466c6167223a302c22466c656e223a36342c22446563696d616c223a302c2243686172736574223a2262696e617279222c22436f6c6c617465223a2262696e617279222c22456c656d73223a6e756c6c2c22456c656d73497342696e6172794c6974223a6e756c6c2c224172726179223a66616c73657d2c227374617465223a352c22636f6d6d656e74223a22222c2268696464656e223a66616c73652c226368616e67655f73746174655f696e666f223a6e756c6c2c2276657273696f6e223a327d2c7b226964223a342c226e616d65223a7b224f223a22524f57535f53454c4543544544222c224c223a22726f77735f73656c6563746564227d2c226f6666736574223a332c226f726967696e5f64656661756c74223a6e756c6c2c226f726967696e5f64656661756c745f626974223a6e756c6c2c2264656661756c74223a6e756c6c2c2264656661756c745f626974223a6e756c6c2c2264656661756c745f69735f65787072223a66616c73652c2267656e6572617465645f657870725f737472696e67223a22222c2267656e6572617465645f73746f726564223a66616c73652c22646570656e64656e636573223a6e756c6c2c2274797065223a7b225470223a382c22466c6167223a302c22466c656e223a36342c22446563696d616c223a302c2243686172736574223a2262696e617279222c22436f6c6c617465223a2262696e617279222c22456c65\"(len:8389);"]
[2023/02/11 14:39:51.644 +08:00] [WARN] [session.go:2211] ["run statement failed"] [schemaVersion=27] [error="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV"] [errorVerbose="[domain:8027]Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).checkSchemaValid\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1746\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*twoPhaseCommitter).execute\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/2pc.go:1585\ngithub.com/tikv/client-go/v2/txnkv/transaction.(*KVTxn).Commit\n\texternal/com_github_tikv_client_go_v2/txnkv/transaction/txn.go:496\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:95\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.insertDDLJobs2Table\n\tddl/job_table.go:462\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs2Table\n\tddl/ddl_worker.go:379\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs\n\tddl/ddl_worker.go:218\ngithub.com/pingcap/tidb/ddl.(*ddl).limitDDLJobs\n\tddl/ddl_worker.go:204\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594\ngithub.com/pingcap/errors.AddStack\n\texternal/com_github_pingcap_errors/errors.go:174\ngithub.com/pingcap/errors.Trace\n\texternal/com_github_pingcap_errors/juju_adaptor.go:15\ngithub.com/pingcap/tidb/store/driver/error.ToTiDBErr\n\tstore/driver/error/error.go:166\ngithub.com/pingcap/tidb/store/driver/txn.extractKeyErr\n\tstore/driver/txn/error.go:162\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).extractKeyErr\n\tstore/driver/txn/txn_driver.go:308\ngithub.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Commit\n\tstore/driver/txn/txn_driver.go:96\ngithub.com/pingcap/tidb/session.(*LazyTxn).Commit\n\tsession/txn.go:425\ngithub.com/pingcap/tidb/session.(*session).commitTxnWithTemporaryData\n\tsession/session.go:832\ngithub.com/pingcap/tidb/session.(*session).doCommit\n\tsession/session.go:718\ngithub.com/pingcap/tidb/session.(*session).doCommitWithRetry\n\tsession/session.go:962\ngithub.com/pingcap/tidb/session.(*session).CommitTxn\n\tsession/session.go:1089\ngithub.com/pingcap/tidb/session.autoCommitAfterStmt\n\tsession/tidb.go:283\ngithub.com/pingcap/tidb/session.finishStmt\n\tsession/tidb.go:249\ngithub.com/pingcap/tidb/session.runStmt\n\tsession/session.go:2361\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\tsession/session.go:2199\ngithub.com/pingcap/tidb/session.(*session).ExecuteInternal\n\tsession/session.go:1659\ngithub.com/pingcap/tidb/ddl.(*session).execute\n\tddl/ddl.go:1660\ngithub.com/pingcap/tidb/ddl.insertDDLJobs2Table\n\tddl/job_table.go:462\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs2Table\n\tddl/ddl_worker.go:379\ngithub.com/pingcap/tidb/ddl.(*ddl).addBatchDDLJobs\n\tddl/ddl_worker.go:218\ngithub.com/pingcap/tidb/ddl.(*ddl).limitDDLJobs\n\tddl/ddl_worker.go:204\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\tutil/wait_group_wrapper.go:154\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1594"] [session="{\n  \"currDBName\": \"\",\n  \"id\": 0,\n  \"status\": 2,\n  \"strictMode\": true,\n  \"user\": null\n}"]

4. What is your TiDB version? (Required)

[hawkingrei]

https://do.pingcap.net/jenkins/blue/rest/organizations/jenkins/pipelines/pingcap/pipelines/tidb/pipelines/ghpr_unit_test/runs/16640/nodes/56/steps/61/log/?start=0

[hawkingrei]

the root reason is the high use usage in the CI. so I have to reduce the concurrency in the bazel's config.