Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

flaky test TestRenameTableAutoIDs #53765

Closed
Tracked by #41316
hawkingrei opened this issue Jun 3, 2024 · 1 comment · Fixed by #54695
Closed
Tracked by #41316

flaky test TestRenameTableAutoIDs #53765

hawkingrei opened this issue Jun 3, 2024 · 1 comment · Fixed by #54695
Assignees
@tiancaiamao
Labels
component/ddl This issue is related to DDL of TiDB. component/test feature/developing the related feature is in development severity/major type/bug The issue is confirmed as a bug.

Comments

@hawkingrei
Copy link
Member

hawkingrei commented Jun 3, 2024 •
edited
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

     testkit.go:157: 
        	Error Trace:	pkg/testkit/testkit.go:157
        	            				pkg/testkit/testkit.go:150
        	            				pkg/ddl/schema_test.go:398
        	Error:      	Received unexpected error:
        	            	[schema:1146]Table 'renametableautoids2.t2' doesn't exist
        	            	github.com/pingcap/errors.AddStack
        	            		external/com_github_pingcap_errors/errors.go:178
        	            	github.com/pingcap/errors.Trace
        	            		external/com_github_pingcap_errors/juju_adaptor.go:15
        	            	github.com/pingcap/tidb/pkg/planner/core.Preprocess
        	            		pkg/planner/core/preprocess.go:141
        	            	github.com/pingcap/tidb/pkg/executor.(*Compiler).Compile
        	            		pkg/executor/compiler.go:67
        	            	github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt
        	            		pkg/session/session.go:2094
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext
        	            		pkg/testkit/testkit.go:383
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExecWithContext
        	            		pkg/testkit/testkit.go:155
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExec
        	            		pkg/testkit/testkit.go:150
        	            	github.com/pingcap/tidb/pkg/ddl_test.TestRenameTableAutoIDs
        	            		pkg/ddl/schema_test.go:398
        	            	testing.tRunner
        	            		GOROOT/src/testing/testing.go:1595
        	            	runtime.goexit
        	            		src/runtime/asm_amd64.s:1650
        	Test:       	TestRenameTableAutoIDs
        	Messages:   	sql:insert into RenameTableAutoIDs2.t2 values (50, 5), [], error stack [schema:1146]Table 'renametableautoids2.t2' doesn't exist
        	            	github.com/pingcap/errors.AddStack
        	            		external/com_github_pingcap_errors/errors.go:178
        	            	github.com/pingcap/errors.Trace
        	            		external/com_github_pingcap_errors/juju_adaptor.go:15
        	            	github.com/pingcap/tidb/pkg/planner/core.Preprocess
        	            		pkg/planner/core/preprocess.go:141
        	            	github.com/pingcap/tidb/pkg/executor.(*Compiler).Compile
        	            		pkg/executor/compiler.go:67
        	            	github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt
        	            		pkg/session/session.go:2094
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext
        	            		pkg/testkit/testkit.go:383
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExecWithContext
        	            		pkg/testkit/testkit.go:155
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExec
        	            		pkg/testkit/testkit.go:150
        	            	github.com/pingcap/tidb/pkg/ddl_test.TestRenameTableAutoIDs
        	            		pkg/ddl/schema_test.go:398
        	            	testing.tRunner
        	            		GOROOT/src/testing/testing.go:1595
        	            	runtime.goexit
        	            		src/runtime/asm_amd64.s:1650

https://tiprow.hawkingrei.com/view/gs/pingcapprow/pr-logs/pull/pingcap_tidb/53762/fast_test_tiprow/1797589167828373504

4. What is your TiDB version? (Required)
@hawkingrei hawkingrei added the type/bug The issue is confirmed as a bug. label Jun 3, 2024
@hawkingrei hawkingrei mentioned this issue Jun 3, 2024
Open
@jebter jebter added component/test severity/major component/ddl This issue is related to DDL of TiDB. labels Jun 4, 2024
@tangenta tangenta self-assigned this Jun 7, 2024
@River2000i
Copy link
Contributor

River2000i commented Jul 17, 2024 •
edited
Loading

disable infoschemav2 in default TestKit and enable infoschemav2 when running ddl can reproduce stably

func (tk *TestKit) RefreshSession() {
	tk.session = NewSession(tk.t, tk.store)
	if intest.InTest {
		seed := uint64(time.Now().UnixNano())
		tk.t.Logf("RefreshSession rand seed: %d", seed)
		rng := rand.New(rand.NewSource(int64(seed)))
		// disable infoschema v2 in default TestKit
		if rng.Intn(10) >= 11 { 
			fmt.Println("RefreshSession set infoschema v2")
			tk.MustExec("set @@global.tidb_schema_cache_size = 1024 * 1024 * 1024")
		}
	}

	// enforce sysvar cache loading, ref loadCommonGlobalVariableIfNeeded
	tk.MustExec("select 3")
}
waitFor := func(col int, tableName, s string) {
		for {
			tk4 := testkit.NewTestKit(t, store)
                         // enable infoschemav2 in new testkit
			tk4.MustExec("set @@global.tidb_schema_cache_size = 1024 * 1024 * 1024")
			tk4.MustExec(`use test`)
			sql := `admin show ddl jobs where db_name like '` + strings.ToLower(dbName) + `%' and table_name like '` + tableName + `%' and job_type = 'rename table'`
			res := tk4.MustQuery(sql).Rows()
			if len(res) == 1 && res[0][col] == s {
				break
			}
			logutil.DDLLogger().Info("Could not find match", zap.String("tableName", tableName), zap.String("s", s), zap.Int("colNum", col))

			for i := range res {
				strs := make([]string, 0, len(res[i]))
				for j := range res[i] {
					strs = append(strs, res[i][j].(string))
				}
				logutil.DDLLogger().Info("ddl jobs", zap.Strings("jobs", strs))
			}
			time.Sleep(10 * time.Millisecond)
		}
	}
[2024/07/17 15:15:11.586 +08:00] [INFO] [ddl.go:1209] ["start DDL job"] [category=ddl] [job="ID:110, Type:rename table, State:queueing, SchemaState:none, SchemaID:106, TableID:108, RowCount:0, ArgLen:3, start time: 2024-07-17 15:15:11.585 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"] [query="rename table t to RenameTableAutoIDs2.t2"]
[2024/07/17 15:15:11.588 +08:00] [INFO] [ddl_worker.go:1195] ["run DDL job"] [worker="worker 84, tp general"] [category=ddl] [jobID=110] [conn=50] [category=ddl] [job="ID:110, Type:rename table, State:queueing, SchemaState:none, SchemaID:106, TableID:108, RowCount:0, ArgLen:0, start time: 2024-07-17 15:15:11.585 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"]
[2024/07/17 15:15:11.590 +08:00] [INFO] [set.go:170] ["set global var"] [conn=53] [name=tidb_schema_cache_size] [val=1073741824]
[2024/07/17 15:15:11.592 +08:00] [ERROR] [domain.go:346] ["failed to load schema diff"] [error="builder's (v2=true) infoschema mismatch, return error to trigger full reload"] [errorVerbose="builder's (v2=true) infoschema mismatch, return error to trigger full reload\ngithub.com/pingcap/tidb/pkg/infoschema.(*Builder).InitWithOldInfoSchema\n\t/Users/lx/PingCAP/tidb/pkg/infoschema/builder.go:800\ngithub.com/pingcap/tidb/pkg/domain.(*Domain).tryLoadSchemaDiffs\n\t/Users/lx/PingCAP/tidb/pkg/domain/domain.go:531\ngithub.com/pingcap/tidb/pkg/domain.(*Domain).loadInfoSchema\n\t/Users/lx/PingCAP/tidb/pkg/domain/domain.go:329\ngithub.com/pingcap/tidb/pkg/domain.(*Domain).Reload\n\t/Users/lx/PingCAP/tidb/pkg/domain/domain.go:679\ngithub.com/pingcap/tidb/pkg/domain.(*Domain).loadSchemaInLoop\n\t/Users/lx/PingCAP/tidb/pkg/domain/domain.go:996\ngithub.com/pingcap/tidb/pkg/domain.(*Domain).Init.func3\n\t/Users/lx/PingCAP/tidb/pkg/domain/domain.go:1367\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupEnhancedWrapper).Run.func1\n\t/Users/lx/PingCAP/tidb/pkg/util/wait_group_wrapper.go:99\nruntime.goexit\n\t/Users/lx/go/go1.21.0/src/runtime/asm_arm64.s:1197"] [stack="github.com/pingcap/tidb/pkg/domain.(*Domain).loadInfoSchema\n\t/Users/lx/PingCAP/tidb/pkg/domain/domain.go:346\ngithub.com/pingcap/tidb/pkg/domain.(*Domain).Reload\n\t/Users/lx/PingCAP/tidb/pkg/domain/domain.go:679\ngithub.com/pingcap/tidb/pkg/domain.(*Domain).loadSchemaInLoop\n\t/Users/lx/PingCAP/tidb/pkg/domain/domain.go:996\ngithub.com/pingcap/tidb/pkg/domain.(*Domain).Init.func3\n\t/Users/lx/PingCAP/tidb/pkg/domain/domain.go:1367\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupEnhancedWrapper).Run.func1\n\t/Users/lx/PingCAP/tidb/pkg/util/wait_group_wrapper.go:99"]
[2024/07/17 15:15:11.602 +08:00] [INFO] [schema_test.go:406] ["Could not find match"] [category=ddl] [tableName=t] [s=running] [colNum=11]
[2024/07/17 15:15:11.602 +08:00] [INFO] [schema_test.go:413] ["ddl jobs"] [category=ddl] [jobs="[110,renametableautoids2,t,\"rename table\",none,106,108,0,\"2024-07-17 15:15:11.585000\",<nil>,<nil>,queueing]"]
[2024/07/17 15:15:11.619 +08:00] [INFO] [set.go:170] ["set global var"] [conn=54] [name=tidb_schema_cache_size] [val=1073741824]
[2024/07/17 15:15:11.635 +08:00] [INFO] [schema_test.go:402] ["ddl jobs done"] [category=ddl] [jobs="[110,renametableautoids2,t,\"rename table\",public,106,108,0,\"2024-07-17 15:15:11.585000\",\"2024-07-17 15:15:11.588000\",<nil>,running]"]
[2024/07/17 15:15:11.642 +08:00] [INFO] [set.go:170] ["set global var"] [conn=55] [name=tidb_schema_cache_size] [val=1073741824]
[2024/07/17 15:15:11.655 +08:00] [INFO] [schema_test.go:402] ["ddl jobs done"] [category=ddl] [jobs="[110,renametableautoids2,t,\"rename table\",public,106,108,0,\"2024-07-17 15:15:11.585000\",\"2024-07-17 15:15:11.588000\",<nil>,running]"]
[2024/07/17 15:15:11.656 +08:00] [WARN] [session.go:2120] ["compile SQL failed"] [error="[schema:1146]Table 'renametableautoids2.t2' doesn't exist"] [errorVerbose="[schema:1146]Table 'renametableautoids2.t2' doesn't exist\ngithub.com/pingcap/errors.AddStack\n\t/Users/lx/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/errors.go:178\ngithub.com/pingcap/errors.Trace\n\t/Users/lx/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/planner/core.Preprocess\n\t/Users/lx/PingCAP/tidb/pkg/planner/core/preprocess.go:144\ngithub.com/pingcap/tidb/pkg/executor.(*Compiler).Compile\n\t/Users/lx/PingCAP/tidb/pkg/executor/compiler.go:67\ngithub.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt\n\t/Users/lx/PingCAP/tidb/pkg/session/session.go:2098\ngithub.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext\n\t/Users/lx/PingCAP/tidb/pkg/testkit/testkit.go:386\ngithub.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExecWithContext\n\t/Users/lx/PingCAP/tidb/pkg/testkit/testkit.go:158\ngithub.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExec\n\t/Users/lx/PingCAP/tidb/pkg/testkit/testkit.go:153\ngithub.com/pingcap/tidb/pkg/ddl_test.TestRenameTableAutoIDs\n\t/Users/lx/PingCAP/tidb/pkg/ddl/schema_test.go:431\ntesting.tRunner\n\t/Users/lx/go/go1.21.0/src/testing/testing.go:1595\nruntime.goexit\n\t/Users/lx/go/go1.21.0/src/runtime/asm_arm64.s:1197"] [SQL="insert into RenameTableAutoIDs2.t2 values (50, 5)"]

@tiancaiamao tiancaiamao assigned tiancaiamao and unassigned tangenta Jul 17, 2024
@tiancaiamao tiancaiamao added feature/developing the related feature is in development and removed may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 labels Jul 17, 2024
@tiancaiamao tiancaiamao mentioned this issue Jul 17, 2024
Merged
13 tasks
@ti-chi-bot ti-chi-bot bot closed this as completed in 1c078a0 Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tiancaiamao tiancaiamao

Labels
component/ddl This issue is related to DDL of TiDB. component/test feature/developing the related feature is in development severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ddl: make test TestRenameTableAutoIDs more stable tiancaiamao/tidb
5 participants
@tiancaiamao @hawkingrei @jebter @tangenta @River2000i