Tracking issue for unstable tests

[tisonkun]

Timeout test

• Timeout test db_test.go:1287: testDBSuite5.TestAddIndex5 #27438
• Timeout test db_partition_test.go:632: testIntegrationSuite1.TestCreateTableWithListPartition #27439
• Timeout test db_test.go:573: testDBSuite7.TestCancelAddIndex #27440
• Timeout test db_integration_test.go:1352: testIntegrationSuite6.TestCreateTableTooLarge #27441
• Timeout test db_partition_test.go:2681: testIntegrationSuite4.TestPartitionCancelAddIndex #27443
• Timeout test builtin_string_vec_test.go:529: testVectorizeSuite1.TestVectorizedBuiltinStringEvalOneVec #27444
• Timeout test db_test.go:557: testDBSuite8.TestCancelAddPrimaryKey #27445
• Unstable test builtin_vectorized_test.go:119: testVectorizeSuite2.TestMockVecPlusIntParallel #27446
• Timeout test builtin_time_vec_generated_test.go:6727: testVectorizeSuite1.TestVectorizedBuiltinTimeEvalOneVecGenerated #27447
• Timeout test `builtin_time_vec_test.go:579: testVectorizeSuite2.TestVectorizedBuiltinTimeFunc #27449
• Timeout test tidb_test.go:878: tidbTestSerialSuite.TestTLSAuto #27450
• Timeout test testDDLTableSplitSuite.TestTableSplit #27620
• Timeout test builtin_time_vec_test.go:575: testVectorizeSuite2.TestVectorizedBuiltinTimeEvalOneVec #28017
• Timeout test builtin_cast_vec_test.go:156: testEvaluatorSuite.TestVectorizedBuiltinCastFunc #28019

Integration tests

• randgen test break by #26545 #27124

race

• session_test.go has incorrect parallel tests cause data race #27979
• race detected on master #25003
• data race in executor_test #25013
• data race in release-4.0 #25073
• data race in github.com/pingcap/tidb/store/mockstore/unistore.(*RPCClient).SendRequest() #25189
• data race in buildSamplingStats #25270
• data race in TestClusteredIndexSyntax #25393
• data race in tidb/domain/domain.go #25643
• data race in table/tables/tables_test.go #25691
• data race in TestFastAnalyze #25782
• data race in tidb/domain/domain.go:1336 #25898
• data race in github.com/pingcap/parser/types.(*FieldType).EvalType() #26539
• data race in github.com/pingcap/parser/ast.(*node).SetOriginTextPosition() #26538
• data race in telemetry_test.TestReport #26718
• Data race in sessionctx/variable.(*SessionVars).InitStatementContext #27114
• Data race in TestParquetVariousTypes #27307
• data race in br/pkg/lightning/mydump.setDatumByInt() #27473
• data race in util/collate/collate.go #27474
• data race in executor/executor_test.go #27476
• data race in executor_test testSerialSuite2).TestCharsetFeature #27485
• data race in tikv.(*KVStore).getTimestampWithRetry() #27821
• data race in executor_test.(*mockSessionManager2).Kill() #27822
• Data race in github.com/pingcap/tidb/util/chunk.(*RowContainer).ActionSpill() #27876
• Unstable test store_test.go fatal error: concurrent map writes #28318
• data race in github.com/pingcap/tidb/bindinfo_test.TestGCBindRecord() #28777
• data race in expression/builtin_cast_vec_test.go #29164
• Data race in github.com/pingcap/tidb/br/pkg/lightning.TestRunServer #29721
• Data race in github.com/pingcap/tidb/br/pkg/lightning.TestGetDeleteTask #29726
• data race in github.com/pingcap/tidb/br/pkg/mock.Test #29727
• data race in github.com/pingcap/tidb/ddl.TestT #29728
• data race in TestDumpPlanReplayerAPI #29731

leak

• Leak test inspectionSuite.TearDownSuite #25271
• Leak test analyze_test.go:55: testSuite1.TestAnalyzePartition #25392
• Leak in client-go/v2/txnkv/transaction.keepAlive #26997
• Leak test testRestoreSchemaSuite.TearDownSuite #26999
• Leak tests pkg executor github.com/pingcap/tidb/executor.(*slowQueryRetriever).parseSlowLog #27437
• Leak in executor.(*slowQueryRetriever).parseSlowLog #27475
• Leak test pdutil.TestScheduler #28816

hang

• tests of executor package hang #27966
• CI check_dev_2 hangs for 10 minutes #25770
• Test session with tikv hang more then 10 minutes #25894
• CI check2 hang more then 10 minutes #25895
• Check dev hang for 10 minutes #26186
• Mysql test hangs more then 10 minutes #26458
• bindinfo hang for 9 minutes #27410
• bindinfo hang #27420

logical error

• unstable test testEvaluatorSuite.TestExprOnlyPushDownToFlash #25584
• unstable test TestModifyOneByte/encrypted #28299

Timeout

• Unit test take longer then 3 seconds in CI #26717

unknown

• unstable test analyze_test.go:577: testSuite1.TestAnalyzeIncremental #25010
• unstable test FAIL: prepare_test.go:448: testPrepareSerialSuite.TestPrepareCacheForPartition #25014
• Bring some tests back online before 2021-06-18 #25157
• unstable test analyze_test.go:584: testSuite1.TestAnalyzeIncremental #25187
• unstable test analyze_test.go:561: testSuite1.TestAnalyzeIndex #25188
• unstable test point_get_plan_test.go:58: testPointGetSuite.TearDownSuite #25190
• unstable test partition_table_test.go:2686: testSuiteWithData.TestRangePartitionBoundariesBetweenS #25269
• unstable test testSuite5.TestSelectGlobalVar #25272
• unstable test testSuite1.TestAnalyzeFullSamplingOnIndexWithVirtualColumnOrPrefixColumn #25274
• untable test testSuiteWithData.TestRangePartitionBoundariesEq #25395
• Unstable test testPlanSerialSuite.TestPlanCacheHitInfo #25585
• unstable test testDBSuite5.TestAddPrimaryKeyRollback1 #25634
• Unstable test testPrepareSerialSuite.TestPrepareOverMaxPreparedStmtCount #25589
• unstable test testPrepareSerialSuite.TestConstPropAndPPDWithCache #25640
• unstable test testPlanSerialSuite.TestPartitionWithVariedDatasources #25642
• unstable test testPlanSerialSuite.TestPrepareCacheDeferredFunction #25690
• unstable test in testPrepareSerialSuite.TestPrepareCache #25692
• Unstable test testUtilSuite.TearDownSuite #25780
• Unstable test testSuite5.TestValidateSetVar #25781
• Unstable test testPrepareSerialSuite.TestPrepareCacheNow #25783
• Unstable test testSysVarSuite.TestInstanceScopedVars #25893
• Unstable test testInfoschemaTableSuite.TestForAnalyzeStatus #25896
• Unstable test testStaleTxnSerialSuite.TestSelectAsOf #25897
• TIDB exit again and again in ci #26009
• Integration test break by #25766 #26018
• unstable test: trackerrecorder_test.go:63: trackRecorderSuite.TestHeapProfileRecorder #26099
• Unstable integration mysql test #26138
• Unstable test testTxnStateSerialSuite.TestCommitting #26185
• Unstable test testPessimisticSuite.TestAmendForUniqueIndex and testStatsSuite.TestAnalyzeGlobalStatsWithOpts2 #25986
• conn_test.go:ConnTestSuite.TestConnExecutionTimeout failed #15862
• Unstable test testSuite.TestIterator #26235
• prepare_test.go:testPlanSerialSuite.TestIssue18066 failed #18945
• Unstable test testTopSQLReporter.TestCollectInternal #26321
• unstable test TestGlobalKillSuite.TestLostConnection #26409
• unstable test testSerialSuite1.TestMergeJoinInDisk #26410
• unstable test case TestAnalyzeGlobalStatsWithOpts #24010
• TestAnalyzeGlobalStatsWithOpts2 test failed #24679
• unstable test ConnTestSuite.TearDownSuite #25458
• Unstable test testPrepareSerialSuite.TestPrepareCacheNow #25783
• unstable unit test: testShowStatsSuite.TestShowAnalyzeStatus #14931
• Unstable test analyze_test.go:950: testSerialSuite2.TestIssue20874 #26684
• unstable test analyze_test.go:584: testSuite1.TestAnalyzeIncremental #25187
• flaky test: runTestInfoschemaClientErrors #23977
• unstable test testIntegrationPartitionSerialSuite.TestListPartitionOrderLimit #26796
• Unstable test testPlanSerialSuite.TestPlanCacheUnsignedHandleOverflow #26934
• Unstable test explainfor_test.go: testSerialSuite.TestExplainFor #22212
• unstable test tidbTestTopSQLSuite.TestTopSQLCPUProfile #25122
• tidbTestSerialSuite.TestTLS takes more then 3 seconds #26947
• Unstable test testSerialDBSuite1.TestCreateTableWithRangeColumnPartition #26986
• Unstable test testCTCSerialSuiteWrapper.TestChangingColOriginDefaultValue #27115
• Unstable test testTxnStateSerialSuite.TestBlocked #27116
• unstable test testSerialSuite.TestTemporaryTableNoNetwork #27448
• Flaky test: TestShowGrantsForCurrentUserUsingRole #27651
• AutoTLS UT fails #27711
• Unstable test lightningServerSuite.TestGetDeleteTask #27820
• Unstable test testDBSuite1.TestDropPrimaryKey #27888
• Unstable test testIntegrationSuite4.TestTruncatePartitionAndDropTable #27889
• Unstable test testDBSuite8.TestRecoverTable #27892
• Unstable attributes_sql_test.go:237: testDBSuite8.TestFlashbackTable #28020
• Unstable test session_test.go:5663: testTiDBAsLibrary.TestMemoryLeak #28049
• Unstable test github.com/pingcap/tidb/plugin.Shutdown #28054
• Unstable test explainfor_test.go:79: testSerialSuite.TestExplainFor #28348
• Unstable test rule_result_reorder_test.go:165: testRuleReorderResults.TestOrderedResultModeOnOtherOperators #28349
• Unstable test column_type_change_test.go:1596: testColumnTypeChangeSuite.TestChangingColOriginDefaultValue #28350
• Unstable test session_test.go:5721: testTiDBAsLibrary.TestMemoryLeak #28351
• Unstable test db_partition_test.go: testIntegrationSuite5.TestDropSchemaWithPartitionTable #28553
• Unstable tests TestUpdateHistogram #28676
• Possible unstable tests in executor package #28646
• Unstable test: trackerrecorder_test.go:47: TestHeapProfileRecorder #29264
• Unstable test: builtin_string_test.go:2634 TestWeightString #29283
• Unstable test case testDBSuite8.TestCreateWithSameName #29734
• Unstable test TestCreateUniqueExpressionIndex #29989
• Unstable test testDDLSuite.TestParallelDDL #31590

[tisonkun]

@LittleFall thanks for fixing #25584 ! I'd like to know whether or not its cause fits one of entries listed above? Or it seems like a logical errors.

[LittleFall]

it's a logical error (of test framework)

[karuppiah7890]

@tisonkun We can add #28017 to the list I think

[tisonkun]

@karuppiah7890 updated. Thank you!

[karuppiah7890]

We can also add #28019 and #28020 @tisonkun . Sorry, I just created them, should have pinged after creating all of them

[karuppiah7890]

I think we will have to add #28206 to the list

[feitian124]

i believe #28553 is unstable, please check and add to list.

[tisonkun]

@feitian124 added. Thanks for your report.

[karuppiah7890]

We could add #28646 to the list

[jyz0309]

We could add #28706 to the list.

[tisonkun]

We could add #28706 to the list.

Thanks for your reporting. This is duplicate of #28676 and thus I've closed #28706 . Please update your PR to master to resolve it.

[tisonkun]

Hi audience of this thread,

I'm going to close this issue as it grows too long and we actually overcome most of the items.

Fighting with test stability should take some time and I'd prefer tag all related issue with component/test + type/bug instead of such a tracking issue. At least we can bump the tracking issue to a new (clean) one.