roachtest: tpccbench/nodes=6/cpu=16/multi-az failed

[cockroach-teamcity]

roachtest.tpccbench/nodes=6/cpu=16/multi-az failed with artifacts on ricky-test-upgrade-roachprod @ 6d809e6a4377d4d9952cd0e0ed7b4bf6c9f8b3b7:

The test failed on branch=ricky-test-upgrade-roachprod, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1
	cluster.go:1667,context.go:140,cluster.go:1656,test_runner.go:849: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2878303-1618337346-59-n7cpu16-geo --oneshot --ignore-empty-nodes: exit status 1 7: skipped
		1: dead
		4: 1590
		2: 1563
		6: 1608
		5: 1595
		3: 1580
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1159
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:283
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2054
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh tpccbench/nodes=6/cpu=16/multi-az

Same failure on other branches

• roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #62339 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] [C-test-failure O-roachtest O-robot branch-release-21.1]
• roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #61974 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] [C-test-failure O-roachtest O-robot branch-master]
• roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #59044 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [C-test-failure O-roachtest O-robot branch-release-20.1]

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

[tbg]

mystery node crash, but also this is on Ricky's test branch so closing as we are already tracking this on master.