roachtest: backup/KMS/AWS/n3cpu4 failed

[cockroach-teamcity]

roachtest.backup/KMS/AWS/n3cpu4 failed with artifacts on master @ 90a1f850ccfc9a123c9dcaba317296151ae32a8c:

(monitor.go:153).Wait: monitor failure: dial tcp 18.119.116.114:26257: connect: connection refused
test artifacts and logs in: /artifacts/backup/KMS/AWS/n3cpu4/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=4 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_metamorphicBuild=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-33215

[msbutler]

The backup/restore roundtrip on this test completed and the test failed because node 1 crashed during sql fingerprinting. I have not yet figured out why this occured. It could be disk stalling, given the following in logs in cockroach-stderr:

*
* WARNING: disk slowness detected: unable to sync log files within 10s
*
*
* WARNING: disk slowness detected: unable to sync log files within 10s

[msbutler]

Aha, I wonder if SHOW EXPERIMENTAL FINGERPRINT bumped against the memory monitor added by the runtime assertions. This test allows for 50% memory usage and the last heap profile shows we're above the utilization (about 4.5 GB) for a c6id.xlarge vm (capacity of 8GB):

[msbutler]

I'm going to close this issue out. If an OOM caused the test to fail, then this is a duplicate to #113816.

If an OOM didn't take n1 down, I suspect the vm on n1 was quite unhealthy as well:

• After the n1 cockroach process crashed, the roachtest driver could not gather the dmesg file from n1
• we detect disk slowness right before the test fails.

[msbutler]

@DarrylWong reran this on TC and confirmed this test failed due to an OOm'd node. closing this out as a duplicate of #113816