TestLabelerRuleTTL is flaky

[okJiang]

Flaky Test

Which jobs are failing

[FAIL]  pkg/schedule/labeler TestLabelerRuleTTL
=== RUN   TestLabelerRuleTTL
==================
WARNING: DATA RACE
Read at 0x00c000433d10 by goroutine [135](https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685#step:4:136):
  runtime.mapaccess1_faststr()
      /opt/hostedtoolcache/go/1.23.2/x64/src/runtime/map_faststr.go:13 +0x0
  github.com/tikv/pd/pkg/schedule/labeler.checkRuleInMemoryAndStorage()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:405 +0x90
  github.com/tikv/pd/pkg/schedule/labeler.TestLabelerRuleTTL()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:393 +0x1891
  github.com/pingcap/failpoint.parseTerm()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:149 +0x233
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x174a
  github.com/tikv/pd/pkg/schedule/labeler.TestLabelerRuleTTL()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:383 +0x174b
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1743 +0x44

Previous write at 0x00c000433d10 by goroutine [137](https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685#step:4:138):
  runtime.mapdelete_faststr()
      /opt/hostedtoolcache/go/1.23.2/x64/src/runtime/map_faststr.go:321 +0x0
  github.com/tikv/pd/pkg/schedule/labeler.(*RegionLabeler).DeleteLabelRuleLocked()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:265 +0x1e8
  github.com/tikv/pd/pkg/schedule/labeler.(*RegionLabeler).checkAndClearExpiredLabels()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:92 +0x417
  github.com/tikv/pd/pkg/schedule/labeler.(*RegionLabeler).doGC()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:67 +0x165
  github.com/tikv/pd/pkg/schedule/labeler.NewRegionLabeler.gowrap1()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:55 +0x44

Goroutine 135 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1743 +0x825
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:2168 +0x85
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1690 +0x226
  testing.runTests()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:2034 +0xf17
  main.main()
      _testmain.go:73 +0x164

Goroutine 137 (running) created at:
  github.com/tikv/pd/pkg/schedule/labeler.NewRegionLabeler()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:55 +0x297
  github.com/tikv/pd/pkg/schedule/labeler.TestLabelerRuleTTL()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:344 +0x48e
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1743 +0x44
==================
    labeler_test.go:405: 
        	Error Trace:	/home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:405
        	            				/home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:393
        	Error:      	Not equal: 
        	            	expected: true
        	            	actual  : false
        	Test:       	TestLabelerRuleTTL
    testing.go:[139](https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685#step:4:140)9: race detected during execution of test
--- FAIL: TestLabelerRuleTTL (0.10s)

CI link

https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685

Reason for failure (if possible)

Anything else

[okJiang]

https://github.com/tikv/pd/actions/runs/11271449522/job/31344283482?pr=8685

[okJiang]

https://github.com/tikv/pd/actions/runs/11285918824/job/31389457320?pr=8685

[okJiang]

https://github.com/tikv/pd/actions/runs/11341558936/job/31540093117?pr=8707

[okJiang]

pd/pkg/schedule/labeler/labeler_test.go

Line 344 in b70107e

labeler, err := NewRegionLabeler(context.Background(), store, time.Millisecond*10)

gcInterval(10ms) is so short that it has been GC before L393(check the status) and after L387(update its ttl).