roachtest: rebalance/by-load/replicas/mixed-version failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on release-25.2 @ 3d3e3e1ddbd63b744d3d91ed1cb909f037e01586:

(mixedversion.go:810).Run: mixed-version test failure while running step 57 (run "rebalance load run"): CPU not evenly balanced after timeout: within bounds mean=80.6 tolerance=20.0% (±16.1) bounds=[64.5, 96.7]
	stores=[s1: 77 (-3.9%), s2: 78 (-2.3%), s3: 75 (-6.6%), s4: 96 (+19.8%), s5: 76 (-4.5%), s6: 78 (-2.5%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• fs=ext4
• localSSD=true
• metamorphicBufferedSender=true
• mvtDeploymentMode=shared-process
• mvtVersions=v24.1.20 → v24.2.10 → v24.3.15 → v25.1.8 → release-25.2
• runtimeAssertionsBuild=true
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-52506

[arulajmani]

s4 seems to be the straggler:

2025/07/12 12:25:59 rebalance_load.go:307: cpu outside bounds mean=81.0 tolerance=20.0% (±16.2) bounds=[64.8, 97.2]
	below  = []
	within = [s1: 76 (-5.6%), s2: 78 (-3.0%), s3: 75 (-6.4%), s5: 79 (-1.7%), s6: 78 (-3.6%)]
	above  = [s4: 97 (+20.4%)]

Looking at the distribution logs, I see:

I250712 12:16:22.949556 482 13@kv/kvserver/store_rebalancer.go:499 ⋮ [T1,Vsystem,n4,s4,store-rebalancer,obj=‹cpu›] 2089  considering load-based lease transfers for s4 with (queries-per-second=3791.1 cpu-per-second=503ms) load, mean=(queries-per-second=4059.5 cpu-per-second=433ms), upperThreshold=(queries-per-second=4059.5 cpu-per-second=483ms)
I250712 12:16:22.950785 482 13@kv/kvserver/store_rebalancer.go:639 ⋮ [T1,Vsystem,n4,s4,store-rebalancer,obj=‹cpu›] 2090  ran out of leases worth transferring and load (queries-per-second=3791.1 cpu-per-second=503ms) is still above desired threshold (queries-per-second=4059.5 cpu-per-second=483ms); considering load-based replica rebalances
I250712 12:16:22.952103 482 13@kv/kvserver/store_rebalancer.go:654 ⋮ [T1,Vsystem,n4,s4,store-rebalancer,obj=‹cpu›] 2091  ran out of replicas worth transferring and load (queries-per-second=3791.1 cpu-per-second=503ms) is still above desired threshold (queries-per-second=4059.5 cpu-per-second=483ms); will check again soon

[arulajmani]

Looking at the CPU profiles, the majority of the regression seems to be coming from SQL land -- we see a 67% (14.1/21.1) regression. n4 is the only node that's running on v24.3; all other nodes are running on v25.1.

Given we don't attribute this SQL usage to any replica, it's understandable that we can't shed load effectively from s4 to bring us within bounds. Given this test doesn't fail too much, and we now have access to CPU profiles thanks to #148658, I'm going to close this issue out without taking action with breadcrumbs on how to triage it if this fails again. However, if we find that this fails more frequently, we can consider relaxing the 20% bounds to be a bit lax when all nodes aren't on the same version.