server: TestClusterVersionUpgrade failed

[cockroach-teamcity]

server.TestClusterVersionUpgrade failed with artifacts on master @ d66c14663e027a5d9699b8a15e3d6f5821877ef5:

=== RUN   TestClusterVersionUpgrade
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/ab5462d2cb3989e2a33018db970f23de/logTestClusterVersionUpgrade615506918
    test_log_scope.go:80: use -show-logs to present logs inline
    version_cluster_test.go:273: condition failed to evaluate within 45s: 0: v21.2-36 active=true (wanted false)
        goroutine 32856908 [running]:
        runtime/debug.Stack()
        	GOROOT/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x4e7c7b8, 0xc01357e340}, 0x0, 0x3bb2040)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x4e7c7b8, 0xc01357e340}, 0xc0105ad288)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/server_test.TestClusterVersionUpgrade(0xc01357e340)
        	github.com/cockroachdb/cockroach/pkg/server_test/version_cluster_test.go:273 +0x8f6
        testing.tRunner(0xc01357e340, 0x3f84298)
        	GOROOT/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	GOROOT/src/testing/testing.go:1306 +0x35a
    panic.go:642: -- test log scope end --
--- FAIL: TestClusterVersionUpgrade (48.59s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• TAGS=bazel,gss,deadlock

/cc @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-12163

[cockroach-teamcity]

server.TestClusterVersionUpgrade failed with artifacts on master @ 5ad21e3896ee809e9c3ebc28bb22166f1275acca:

=== RUN   TestClusterVersionUpgrade
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade192890838
    test_log_scope.go:80: use -show-logs to present logs inline
    version_cluster_test.go:273: condition failed to evaluate within 3m45s: 0: v21.2-36 active=true (wanted false)
        goroutine 679043 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x72
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x8aac8d8, 0xc00682e4e0}, 0x89fe620, 0x5a0dfc0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x7d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x8aac8d8, 0xc00682e4e0}, 0xc006278d8c)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/server_test.TestClusterVersionUpgrade(0xc00682e4e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/version_cluster_test.go:273 +0xae6
        testing.tRunner(0xc00682e4e0, 0x5de7e58)
        	/usr/local/go/src/testing/testing.go:1259 +0x230
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x727
    panic.go:661: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade192890838
--- FAIL: TestClusterVersionUpgrade (234.85s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• GOFLAGS=-race -parallel=4

This test on roachdash | Improve this report!

[cockroach-teamcity]

server.TestClusterVersionUpgrade failed with artifacts on master @ 912964e02ddd951c77d4f71981ae18b3894e9084:

=== RUN   TestClusterVersionUpgrade
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade2151831843
    test_log_scope.go:80: use -show-logs to present logs inline
    version_cluster_test.go:273: condition failed to evaluate within 45s: 0: v21.2-42 active=true (wanted false)
        goroutine 39263657 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x4fbb778, 0xc00866eea0}, 0x0, 0x3cc1280)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x4fbb778, 0xc00866eea0}, 0xc00e9845b8)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/server_test.TestClusterVersionUpgrade(0xc00866eea0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/version_cluster_test.go:273 +0x8f6
        testing.tRunner(0xc00866eea0, 0x409b1a0)
        	/usr/local/go/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x35a
    panic.go:642: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade2151831843
--- FAIL: TestClusterVersionUpgrade (49.71s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• TAGS=deadlock

• GOFLAGS=-parallel=4

This test on roachdash | Improve this report!

[cockroach-teamcity]

server.TestClusterVersionUpgrade failed with artifacts on master @ d1eafdf56b42ca09a169c199916917a144c2c412:

=== RUN   TestClusterVersionUpgrade
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade1710923491
    test_log_scope.go:80: use -show-logs to present logs inline
    version_cluster_test.go:273: condition failed to evaluate within 3m45s: 0: v21.2-42 active=true (wanted false)
        goroutine 695653 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x72
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x8adf5b8, 0xc008edc4e0}, 0x8a30b60, 0x5a36da0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x7d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x8adf5b8, 0xc008edc4e0}, 0xc00249c7bc)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/server_test.TestClusterVersionUpgrade(0xc008edc4e0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/version_cluster_test.go:273 +0xae6
        testing.tRunner(0xc008edc4e0, 0x5e12d10)
        	/usr/local/go/src/testing/testing.go:1259 +0x230
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x727
    panic.go:661: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade1710923491
--- FAIL: TestClusterVersionUpgrade (241.98s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• GOFLAGS=-json

This test on roachdash | Improve this report!

[andreimatei]

just happened to me too on CI

[cockroach-teamcity]

server.TestClusterVersionUpgrade failed with artifacts on master @ 8cd28089c6c7333615ba3201e841839001d2f0e1:

=== RUN   TestClusterVersionUpgrade
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade2677744765
    test_log_scope.go:80: use -show-logs to present logs inline
    version_cluster_test.go:273: condition failed to evaluate within 45s: 0: v21.2-48 active=true (wanted false)
        goroutine 42023011 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x50063f8, 0xc005dc8000}, 0x0, 0x3cedf40)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x50063f8, 0xc005dc8000}, 0xc006c725d0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/server_test.TestClusterVersionUpgrade(0xc005dc8000)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/version_cluster_test.go:273 +0x8f6
        testing.tRunner(0xc005dc8000, 0x40cdba0)
        	/usr/local/go/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x35a
    panic.go:642: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade2677744765
--- FAIL: TestClusterVersionUpgrade (50.12s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• TAGS=deadlock

• GOFLAGS=-parallel=4

This test on roachdash | Improve this report!

[cockroach-teamcity]

server.TestClusterVersionUpgrade failed with artifacts on master @ ce6d32ad93283556cb88bb69eae595c4febaa6c4:

=== RUN   TestClusterVersionUpgrade
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade338561729
    test_log_scope.go:80: use -show-logs to present logs inline
    version_cluster_test.go:273: condition failed to evaluate within 3m45s: 0: v21.2-50 active=true (wanted false)
        goroutine 529114 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x72
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x8bc1cb8, 0xc00fff8000}, 0x8b127e0, 0x5a90c20)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x7d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x8bc1cb8, 0xc00fff8000}, 0xc000a0347c)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/server_test.TestClusterVersionUpgrade(0xc00fff8000)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/version_cluster_test.go:273 +0xae6
        testing.tRunner(0xc00fff8000, 0x5e74010)
        	/usr/local/go/src/testing/testing.go:1259 +0x230
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x727
    panic.go:661: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade338561729
--- FAIL: TestClusterVersionUpgrade (254.79s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• GOFLAGS=-json

This test on roachdash | Improve this report!

[cockroach-teamcity]

server.TestClusterVersionUpgrade failed with artifacts on master @ 5c0d5132ade6d2e30a9d414163851272750d8afa:

=== RUN   TestClusterVersionUpgrade
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade3877796030
    test_log_scope.go:80: use -show-logs to present logs inline
    version_cluster_test.go:273: condition failed to evaluate within 45s: 0: v21.2-52 active=true (wanted false)
        goroutine 39611831 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x509a198, 0xc00fe5a9c0}, 0x0, 0x3d3e7a0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x509a198, 0xc00fe5a9c0}, 0xc00ada75e8)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/server_test.TestClusterVersionUpgrade(0xc00fe5a9c0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/version_cluster_test.go:273 +0x8f6
        testing.tRunner(0xc00fe5a9c0, 0x41268e0)
        	/usr/local/go/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x35a
    panic.go:642: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade3877796030
--- FAIL: TestClusterVersionUpgrade (60.68s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• TAGS=deadlock

• GOFLAGS=-parallel=4

This test on roachdash | Improve this report!

[cameronnunez]

Can confirm this test is still failing. Unsure why yet.

[cockroach-teamcity]

server.TestClusterVersionUpgrade failed with artifacts on master @ 89198dd89a0248a6344296138431c4f84dd09913:

=== RUN   TestClusterVersionUpgrade
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade2998515523
    test_log_scope.go:80: use -show-logs to present logs inline
    version_cluster_test.go:271: condition failed to evaluate within 3m45s: 0: v21.2-68 active=true (wanted false)
        goroutine 574656 [running]:
        runtime/debug.Stack()
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x72
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x8d8cb98, 0xc0022a2000}, 0x8cdc4e0, 0x5b3f540)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x7d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x8d8cb98, 0xc0022a2000}, 0xc00555364c)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x5b
        github.com/cockroachdb/cockroach/pkg/server_test.TestClusterVersionUpgrade(0xc0022a2000)
        	/go/src/github.com/cockroachdb/cockroach/pkg/server/version_cluster_test.go:271 +0xaa7
        testing.tRunner(0xc0022a2000, 0x5f2f420)
        	/usr/local/go/src/testing/testing.go:1259 +0x230
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:1306 +0x727
    panic.go:661: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestClusterVersionUpgrade2998515523
--- FAIL: TestClusterVersionUpgrade (243.91s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• GOFLAGS=-json

This test on roachdash | Improve this report!

[cockroach-teamcity]

server.TestClusterVersionUpgrade failed with artifacts on master @ c518952e739509c8cfe8e3a93c2b6ffb3002b866:

=== RUN   TestClusterVersionUpgrade
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/ab5462d2cb3989e2a33018db970f23de/logTestClusterVersionUpgrade3840330042
    test_log_scope.go:80: use -show-logs to present logs inline
    version_cluster_test.go:271: condition failed to evaluate within 45s: 0: v21.2-80 active=true (wanted false)
        goroutine 43155277 [running]:
        runtime/debug.Stack()
        	GOROOT/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x5338df8, 0xc003cc2ea0}, 0x0, 0x3f57fc0)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x5338df8, 0xc003cc2ea0}, 0xc0130cb2b0)
        	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/server_test.TestClusterVersionUpgrade(0xc003cc2ea0)
        	github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/version_cluster_test.go:271 +0x896
        testing.tRunner(0xc003cc2ea0, 0x43699d8)
        	GOROOT/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	GOROOT/src/testing/testing.go:1306 +0x35a
    panic.go:642: -- test log scope end --
--- FAIL: TestClusterVersionUpgrade (51.32s)

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• TAGS=bazel,gss,deadlock

This test on roachdash | Improve this report!