roachtest: kv/quiescence/nodes=3 failed

[cockroach-teamcity]

roachtest.kv/quiescence/nodes=3 failed with artifacts on master @ a0ab818e89508ca0b65926a4faac4c563d114acf:

test artifacts and logs in: /artifacts/kv/quiescence/nodes=3/run_1
(cluster.go:1867).Start: parallel execution failure: ~ ./cockroach sql --url 'postgres://root@localhost:26257?sslmode=disable' "-e
CREATE SCHEDULE IF NOT EXISTS test_only_backup FOR BACKUP INTO 'gs://cockroachdb-backup-testing/roachprod-scheduled-backups/teamcity-8725593-1676528162-07-n4cpu4/1676530610503793144?AUTH=implicit' RECURRING '*/15 * * * *'
FULL BACKUP '@hourly'
WITH SCHEDULE OPTIONS first_run = 'now'"
ERROR: cannot dial server.
Is the server running?
If the server is running, check --host client-side and --advertise server-side.
timeout: context deadline exceeded
Failed running "sql": COMMAND_PROBLEM: ssh verbose log retained in ssh_065650.503891105_n3_run-sql.log: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-24581

[erikgrinaker]

This test verifies that ranges still quiesce after killing a node. However, at the end of the test it restarts the node to appease the dead node monitor:

cockroach/pkg/cmd/roachtest/tests/kv.go

Line 449 in eec5a47

c.Start(ctx, t.L(), option.DefaultStartOpts(), install.MakeClusterSettings(), c.Node(nodes)) // satisfy dead node detector, even if test fails below

It appears that this runs into a race with the scheduled backup injector that registers scheduled backups on cluster/node startup:

cockroach/pkg/roachprod/install/cockroach.go

Lines 798 to 806 in 399a278

	// createFixedBackupSchedule creates a cluster backup schedule which, by
	// default, runs an incremental every 15 minutes and a full every hour. On
	// `roachprod create`, the user can provide a different recurrence using the
	// 'schedule-backup-args' flag. If roachprod is local, the backups get stored in
	// nodelocal, and otherwise in 'gs://cockroachdb-backup-testing'.
	// This cmd also ensures that only one schedule will be created for the cluster.
	func (c *SyncedCluster) createFixedBackupSchedule(
	ctx context.Context, l *logger.Logger, scheduledBackupArgs string,
	) error {

Races aside, we don't really need to be restarting the node here, we can use Monitor.ExpectDeath() instead. Will submit a PR.

[erikgrinaker]

@msbutler The scheduled backup injection needs to be a bit more robust. In this case it seemed like it tried connecting to the restarted node too quickly and timed out, the node likely took more than 10 seconds to fully start back up (the default statement timeout).

[msbutler]

@erikgrinaker thanks for the initial triage on this. I've seen this failure mode before and I am puzzled by it. Just asked test-eng a couple questions here.

[msbutler]

resolved by #97495