mcs: data race about scheduler controller

[lhy1024]

Bug Report

What did you do?

run test

What did you expect to see?

What did you see instead?

2023-09-18T08:11:45.8089852Z ==================
2023-09-18T08:11:45.8089957Z WARNING: DATA RACE
2023-09-18T08:11:45.8090106Z Write at 0x00c002fc2b10 by goroutine 65006:
2023-09-18T08:11:45.8090234Z   runtime.racewrite()
2023-09-18T08:11:45.8090397Z       <autogenerated>:1 +0x1e
2023-09-18T08:11:45.8090616Z   github.com/tikv/pd/pkg/schedule/schedulers.(*Controller).Wait()
2023-09-18T08:11:45.8090998Z       /home/runner/work/pd/pd/pkg/schedule/schedulers/scheduler_controller.go:70 +0x1a4
2023-09-18T08:11:45.8091229Z   github.com/tikv/pd/pkg/schedule.(*Coordinator).RunUntilStop()
2023-09-18T08:11:45.8091552Z       /home/runner/work/pd/pd/pkg/schedule/coordinator.go:374 +0x13e
2023-09-18T08:11:45.8091812Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startCluster.func2()
2023-09-18T08:11:45.8092142Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:447 +0x33
2023-09-18T08:11:45.8092154Z 
2023-09-18T08:11:45.8092323Z Previous read at 0x00c002fc2b10 by goroutine 65004:
2023-09-18T08:11:45.8092444Z   runtime.raceread()
2023-09-18T08:11:45.8092608Z       <autogenerated>:1 +0x1e
2023-09-18T08:11:45.8092854Z   github.com/tikv/pd/pkg/schedule/schedulers.(*Controller).AddScheduler()
2023-09-18T08:11:45.8093230Z       /home/runner/work/pd/pd/pkg/schedule/schedulers/scheduler_controller.go:187 +0x2dd
2023-09-18T08:11:45.8093479Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Cluster).updateScheduler()
2023-09-18T08:11:45.8093940Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/cluster.go:237 +0xec4
2023-09-18T08:11:45.8094224Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Cluster).StartBackgroundJobs.func1()
2023-09-18T08:11:45.8094557Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/cluster.go:382 +0x33
2023-09-18T08:11:45.8094567Z 
2023-09-18T08:11:45.8094794Z Goroutine 65006 (running) created at:
2023-09-18T08:11:45.8095032Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startCluster()
2023-09-18T08:11:45.8095374Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:447 +0xeec
2023-09-18T08:11:45.8095809Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startCluster-fm()
2023-09-18T08:11:45.8095978Z       <autogenerated>:1 +0x47
2023-09-18T08:11:45.8096223Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).campaignLeader()
2023-09-18T08:11:45.8096556Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:254 +0xc1e
2023-09-18T08:11:45.8096821Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).primaryElectionLoop()
2023-09-18T08:11:45.8097162Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:222 +0x3a4
2023-09-18T08:11:45.8097429Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startServerLoop.func1()
2023-09-18T08:11:45.8097760Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:143 +0x33
2023-09-18T08:11:45.8097769Z 
2023-09-18T08:11:45.8097905Z Goroutine 65004 (running) created at:
2023-09-18T08:11:45.8098166Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Cluster).StartBackgroundJobs()
2023-09-18T08:11:45.8098510Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/cluster.go:382 +0xfc
2023-09-18T08:11:45.8098749Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startCluster()
2023-09-18T08:11:45.8099079Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:446 +0xd3c
2023-09-18T08:11:45.8099407Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startCluster-fm()
2023-09-18T08:11:45.8099574Z       <autogenerated>:1 +0x47
2023-09-18T08:11:45.8099815Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).campaignLeader()
2023-09-18T08:11:45.8100146Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:254 +0xc1e
2023-09-18T08:11:45.8100398Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).primaryElectionLoop()
2023-09-18T08:11:45.8100729Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:222 +0x3a4
2023-09-18T08:11:45.8100998Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startServerLoop.func1()
2023-09-18T08:11:45.8101327Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:143 +0x33
2023-09-18T08:11:45.8101417Z ==================

What version of PD are you using (pd-server -V)?

[lhy1024]

2023-09-18T14:39:50.3717637Z panic: sync: WaitGroup is reused before previous Wait has returned
2023-09-18T14:39:50.3717742Z 
2023-09-18T14:39:50.3717839Z goroutine 65534 [running]:
2023-09-18T14:39:50.3717960Z sync.(*WaitGroup).Wait(0xc00daa39e8)
2023-09-18T14:39:50.3718160Z 	/opt/hostedtoolcache/go/1.21.1/x64/src/sync/waitgroup.go:118 +0x105
2023-09-18T14:39:50.3718340Z github.com/tikv/pd/pkg/schedule/schedulers.(*Controller).Wait(...)
2023-09-18T14:39:50.3718559Z 	/home/runner/work/pd/pd/pkg/schedule/schedulers/scheduler_controller.go:70
2023-09-18T14:39:50.3718746Z github.com/tikv/pd/pkg/schedule.(*Coordinator).RunUntilStop(0xc00de123c0)
2023-09-18T14:39:50.3718944Z 	/home/runner/work/pd/pd/pkg/schedule/coordinator.go:374 +0x1a5
2023-09-18T14:39:50.3719176Z created by github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startCluster in goroutine 63431
2023-09-18T14:39:50.3719370Z 	/home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:447 +0xeed
2023-09-18T14:39:50.3719547Z FAIL	github.com/tikv/pd/tests/integrations/mcs/scheduling	23.721s

https://github.com/lhy1024/pd/actions/runs/6222253685/job/16891330890

[lhy1024]

https://github.com/tikv/pd/actions/runs/6231989872/job/16914381623

[HuSharp]

https://github.com/tikv/pd/actions/runs/6244307966/job/16951048366?pr=7087