server: TestClusterVersionUpgrade failed

[cockroach-teamcity]

server.TestClusterVersionUpgrade failed with artifacts on master @ 506d129f5f187134c35e2f71860490e044fde989:

=== RUN   TestClusterVersionUpgrade
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade1761995881
    test_log_scope.go:80: use -show-logs to present logs inline
    version_cluster_test.go:273: condition failed to evaluate within 45s: 0: v21.2-22 active=true (wanted false)
        goroutine 406074 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x526a0f8, 0xc005898b60}, 0x0, 0x3c26240)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x526a0f8, 0xc005898b60}, 0xc0019f5190)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/server_test.TestClusterVersionUpgrade(0xc005898b60)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/version_cluster_test.go:273 +0x8f2
        testing.tRunner(0xc005898b60, 0x4433e30)
        	/usr/local/go/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x35a
    panic.go:642: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade1761995881
--- FAIL: TestClusterVersionUpgrade (46.75s)

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)Parameters in this failure:

• GOFLAGS=-parallel=4

/cc @cockroachdb/server

This test on roachdash | Improve this report!

[jtsiros]

@rauchenstein to triage.

[cockroach-teamcity]

server.TestClusterVersionUpgrade failed with artifacts on master @ 406e0fe1e44cabc8a3384703c642469567c6915e:

=== RUN   TestClusterVersionUpgrade
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade2820236275
    test_log_scope.go:80: use -show-logs to present logs inline
    version_cluster_test.go:273: condition failed to evaluate within 45s: 0: v21.2-26 active=true (wanted false)
        goroutine 416850 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x5294b58, 0xc00fe4f1e0}, 0x0, 0x3c40d80)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x5294b58, 0xc00fe4f1e0}, 0xc00f5797e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/server_test.TestClusterVersionUpgrade(0xc00fe4f1e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/version_cluster_test.go:273 +0x8f6
        testing.tRunner(0xc00fe4f1e0, 0x4450c98)
        	/usr/local/go/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x35a
    panic.go:642: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade2820236275
--- FAIL: TestClusterVersionUpgrade (46.85s)

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)Parameters in this failure:

• GOFLAGS=-parallel=4

This test on roachdash | Improve this report!

[cameronnunez]

@andreimatei any insight on what could be going wrong here?

[andreimatei]

No... How did my name come up? I've looked at the logs quickly and nothing seems to be screaming.
It fails for me under stress in 90s (out of which 45s is the SucceedsSoon). Did you try bisecting?

[cameronnunez]

I noticed you touched this test a while back, so just wondered if anything screamed at you here. I'll try bisecting thanks

[cameronnunez]

@erikgrinaker anything stick out to you here? a bisection pointed me to this commit.

[erikgrinaker]

Can't say it does, no. That particular commit simply adds a trivial cluster version, which isn't even used by anything yet, and otherwise doesn't touch anything related to upgrades.

But does this still reproduce for you with the current master? I've done over 5000 stress runs with a 10-second timeout for SucceedsSoon, no failures yet.

[cameronnunez]

It was failing yesterday, but not anymore after rebasing today. I'll close this.