roachtest: schemachange/bulkingest failed [connection refused but no CRDB error]

[cockroach-teamcity]

roachtest.schemachange/bulkingest failed with artifacts on release-23.2 @ d48b4f943e00d8f10b3b8aaf868da4c560bdae57:

(test_runner.go:1113).runTest: test timed out (2h0m0s)
(schemachange.go:421).2: dial tcp 34.73.1.236:26257: connect: connection refused
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/schemachange/bulkingest/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-33533

[rafiss]

34.73.1.236 appears to be n1. (side note: i thought the node ID would appear in this error message, but we only get the IP.)

The code that timed out during connection is:

				t.L().Printf("Creating index")
				before := timeutil.Now()
				if _, err := db.Exec(`CREATE INDEX payload_a ON bulkingest.bulkingest (payload, a)`); err != nil {
					t.Fatal(err)
				}

But it seems like the cluster was healthy and according to run_141746.338675167_n5_workload-run-bulking.log, it was handling the workload run bulkingest command fine.

I made PR #114551 to improve the logging. But I don't have any leads why the test would fail to connect to CRDB.

[blathers-crl]

cc @cockroachdb/test-eng

[rafiss]

I will assign to TestEng in case they know of a way to diagnose a timeout like this. If not, feel free to close this.

[renatolabs]

The error message definitely could use some improvement; I'm pretty sure the connection refused part is a red herring and the actual reason the test failed is because it timed out:

(test_runner.go:1113).runTest: test timed out (2h0m0s)

I think the sequence of events here is:

• test times out, test context is canceled;
• cockroach processes are killed when context is canceled;
• test observes connection refused error because the process is now dead.

So we have two errors happening before the test's Run function returns. I believe this would have been a little less confusing if we were using ExecContext() in the test, as we would likely see a context canceled error instead, which would point to some other error being the root cause, bringing more attention to the "test timed out" message.

For Test Eng more concretely, I think a takeaway is that we should not show any other errors if the test timed out. That is always going to be the root cause for the failure.

In any case, the ROACHTEST_metamorphicBuild=true tag here is suggestive. Seems like another case of a test that might need a longer timeout with metamorphic builds.

[cockroach-teamcity]

roachtest.schemachange/bulkingest failed with artifacts on release-23.2 @ 0456c0d4e4dca39bde1b4bc146bf7158db3c0f30:

(test_runner.go:1114).runTest: test timed out (2h0m0s)
(schemachange.go:421).2: dial tcp 34.74.177.221:26257: connect: connection refused
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/schemachange/bulkingest/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[renatolabs]

Test times out when using metamorphic cockroach builds. These runs were disabled (#114618) until we have a stable release. We'll re-assess test failures when this feature is enabled again.