TestScheduleTestSuite/TestDisable is unstable

[lhy1024]

Flaky Test

Which jobs are failing

2023-11-29T01:42:11.6662377Z --- FAIL: TestScheduleTestSuite (103.50s)
2023-11-29T01:42:11.6662663Z     --- FAIL: TestScheduleTestSuite/TestDisable (26.11s)
2023-11-29T01:42:11.6662936Z         testutil.go:265: start to run test in pd mode
2023-11-29T01:42:11.6663205Z         testutil.go:269: finish to run test in pd mode
2023-11-29T01:42:11.6663459Z         testutil.go:274: start to run test in api mode
2023-11-29T01:42:11.6663618Z         testutil.go:64: 
2023-11-29T01:42:11.6664122Z             	Error Trace:	/home/runner/work/pd/pd/pkg/utils/testutil/testutil.go:64
2023-11-29T01:42:11.6664822Z             	            				/home/runner/work/pd/pd/tests/server/api/scheduler_test.go:686
2023-11-29T01:42:11.6665527Z             	            				/home/runner/work/pd/pd/tests/server/api/scheduler_test.go:604
2023-11-29T01:42:11.6666067Z             	            				/home/runner/work/pd/pd/tests/testutil.go:279
2023-11-29T01:42:11.6666613Z             	            				/home/runner/work/pd/pd/tests/testutil.go:260
2023-11-29T01:42:11.6667373Z             	            				/home/runner/work/pd/pd/tests/server/api/scheduler_test.go:568
2023-11-29T01:42:11.6667633Z             	Error:      	Condition never satisfied
2023-11-29T01:42:11.6667935Z             	Test:       	TestScheduleTestSuite/TestDisable
2023-11-29T01:42:11.6668017Z FAIL

CI link

https://github.com/tikv/pd/actions/runs/7026828089/job/19120231295?pr=7465

Reason for failure (if possible)

Anything else

[lhy1024]

It failed in

pd/tests/server/api/scheduler_test.go

Line 604 in 4356aeb

suite.assertNoScheduler(re, urlPrefix, name)

pd/tests/server/api/scheduler_test.go

Lines 684 to 690 in 4356aeb

	func (suite *scheduleTestSuite) assertNoScheduler(re *require.Assertions, urlPrefix string, scheduler string) {
	var schedulers []string
	tu.Eventually(re, func() bool {
	tu.ReadGetJSON(suite.Require(), testDialClient, urlPrefix, &schedulers)
	return !slice.Contains(schedulers, scheduler)
	})
	}

This is api

pd/pkg/schedule/handler/handler.go

Line 862 in 4356aeb

disabled, err := sc.IsSchedulerDisabled(scheduler)

We judge it according to the scheduling config

pd/pkg/mcs/scheduling/server/config/config.go

Line 650 in 4356aeb

schedulers := o.GetScheduleConfig().Schedulers

And log shows that scheduling server received the change of config: false->true

2023-11-29T01:42:11.6103855Z [2023/11/29 01:41:40.719 +00:00] [INFO] [watcher.go:131] ["update scheduling config"] [new="{\"cluster-version\":\"2.0.0\",\"schedule\":{\"max-snapshot-count\":64,\"max-pending-peer-count\":64,\"max-merge-region-size\":20,\"max-merge-region-keys\":200000,\"split-merge-interval\":\"1h0m0s\",\"switch-witness-interval\":\"1h0m0s\",\"enable-one-way-merge\":\"false\",\"enable-cross-table-merge\":\"true\",\"patrol-region-interval\":\"10ms\",\"max-store-down-time\":\"30m0s\",\"max-store-preparing-time\":\"48h0m0s\",\"leader-schedule-limit\":4,\"leader-schedule-policy\":\"count\",\"region-schedule-limit\":2048,\"witness-schedule-limit\":4,\"replica-schedule-limit\":64,\"merge-schedule-limit\":8,\"hot-region-schedule-limit\":4,\"hot-region-cache-hits-threshold\":3,\"store-limit\":{\"1\":{\"add-peer\":15,\"remove-peer\":15},\"2\":{\"add-peer\":15,\"remove-peer\":15},\"3\":{\"add-peer\":15,\"remove-peer\":15},\"4\":{\"add-peer\":15,\"remove-peer\":15}},\"tolerant-size-ratio\":0,\"low-space-ratio\":0.8,\"high-space-ratio\":0.7,\"region-score-formula-version\":\"v2\",\"scheduler-max-waiting-operator\":5,\"enable-remove-down-replica\":\"true\",\"enable-replace-offline-replica\":\"true\",\"enable-make-up-replica\":\"true\",\"enable-remove-extra-replica\":\"true\",\"enable-location-replacement\":\"true\",\"enable-debug-metrics\":\"false\",\"enable-joint-consensus\":\"true\",\"enable-tikv-split-region\":\"true\",\"schedulers-v2\":[{\"type\":\"shuffle-leader\",\"args\":null,\"disable\":true,\"args-payload\":\"\"}],\"schedulers-payload\":null,\"hot-regions-write-interval\":\"10m0s\",\"hot-regions-reserved-days\":7,\"max-movable-hot-peer-size\":512,\"enable-diagnostic\":\"true\",\"enable-witness\":\"false\",\"slow-store-evicting-affected-store-ratio-threshold\":0.3,\"store-limit-version\":\"v1\"},\"replication\":{\"max-replicas\":3,\"location-labels\":\"\",\"strictly-match-label\":\"false\",\"enable-placement-rules\":\"true\",\"enable-placement-rules-cache\":\"false\",\"isolation-level\":\"\"},\"store\":{\"coprocessor\":{\"region-max-size\":\"\",\"region-split-size\":\"\",\"region-max-keys\":0,\"region-split-keys\":0,\"enable-region-bucket\":false,\"region-bucket-size\":\"\"},\"storage\":{\"engine\":\"\"}}}"]

but PD server config updated, and overwrite scheduler config: true->false

2023-11-29T01:42:11.6130130Z [2023/11/29 01:41:40.720 +00:00] [INFO] [server.go:1258] ["PD server config is updated"] [new="{\"use-region-storage\":\"true\",\"max-gap-reset-ts\":\"24h0m0s\",\"key-type\":\"table\",\"runtime-services\":\"\",\"metric-storage\":\"\",\"dashboard-address\":\"http://127.0.0.1:42231\",\"trace-region-flow\":\"true\",\"flow-round-by-digit\":3,\"min-resolved-ts-persistence-interval\":\"1s\",\"server-memory-limit\":0,\"server-memory-limit-gc-trigger\":0.7,\"enable-gogc-tuner\":\"false\",\"gc-tuner-threshold\":0.6,\"block-safe-point-v1\":\"false\"}"] [old="{\"use-region-storage\":\"true\",\"max-gap-reset-ts\":\"24h0m0s\",\"key-type\":\"table\",\"runtime-services\":\"\",\"metric-storage\":\"\",\"dashboard-address\":\"auto\",\"trace-region-flow\":\"true\",\"flow-round-by-digit\":3,\"min-resolved-ts-persistence-interval\":\"1s\",\"server-memory-limit\":0,\"server-memory-limit-gc-trigger\":0.7,\"enable-gogc-tuner\":\"false\",\"gc-tuner-threshold\":0.6,\"block-safe-point-v1\":\"false\"}"]

2023-11-29T01:42:11.6144963Z [2023/11/29 01:41:40.721 +00:00] [INFO] [watcher.go:131] ["update scheduling config"] [new="{\"cluster-version\":\"2.0.0\",\"schedule\":{\"max-snapshot-count\":64,\"max-pending-peer-count\":64,\"max-merge-region-size\":20,\"max-merge-region-keys\":0,\"split-merge-interval\":\"1h0m0s\",\"switch-witness-interval\":\"1h0m0s\",\"enable-one-way-merge\":\"false\",\"enable-cross-table-merge\":\"true\",\"patrol-region-interval\":\"10ms\",\"max-store-down-time\":\"30m0s\",\"max-store-preparing-time\":\"48h0m0s\",\"leader-schedule-limit\":4,\"leader-schedule-policy\":\"count\",\"region-schedule-limit\":2048,\"witness-schedule-limit\":4,\"replica-schedule-limit\":64,\"merge-schedule-limit\":8,\"hot-region-schedule-limit\":4,\"hot-region-cache-hits-threshold\":3,\"store-limit\":{\"1\":{\"add-peer\":15,\"remove-peer\":15},\"2\":{\"add-peer\":15,\"remove-peer\":15},\"3\":{\"add-peer\":15,\"remove-peer\":15},\"4\":{\"add-peer\":15,\"remove-peer\":15}},\"tolerant-size-ratio\":0,\"low-space-ratio\":0.8,\"high-space-ratio\":0.7,\"region-score-formula-version\":\"v2\",\"scheduler-max-waiting-operator\":5,\"enable-remove-down-replica\":\"true\",\"enable-replace-offline-replica\":\"true\",\"enable-make-up-replica\":\"true\",\"enable-remove-extra-replica\":\"true\",\"enable-location-replacement\":\"true\",\"enable-debug-metrics\":\"false\",\"enable-joint-consensus\":\"true\",\"enable-tikv-split-region\":\"true\",\"schedulers-v2\":[{\"type\":\"balance-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"balance-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"balance-witness\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"hot-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"transfer-witness-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"shuffle-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"}],\"schedulers-payload\":null,\"hot-regions-write-interval\":\"10m0s\",\"hot-regions-reserved-days\":7,\"max-movable-hot-peer-size\":512,\"enable-diagnostic\":\"true\",\"enable-witness\":\"false\",\"slow-store-evicting-affected-store-ratio-threshold\":0.3,\"store-limit-version\":\"v1\"},\"replication\":{\"max-replicas\":3,\"location-labels\":\"\",\"strictly-match-label\":\"false\",\"enable-placement-rules\":\"true\",\"enable-placement-rules-cache\":\"false\",\"isolation-level\":\"\"},\"store\":{\"coprocessor\":{\"region-max-size\":\"\",\"region-split-size\":\"\",\"region-max-keys\":0,\"region-split-keys\":0,\"enable-region-bucket\":false,\"region-bucket-size\":\"\"},\"storage\":{\"engine\":\"\"}}}"]

[lhy1024]

it is the same with #7440 Tracking Issue: #7464

[HuSharp]

reopen https://github.com/tikv/pd/actions/runs/7136511509/job/19435054648?pr=7493

[rleungx]

The issue might be stale, close it for now.