roachtest: kv/splits/nodes=3 failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/9f191cdad64747f7f19ec4990e559cbb19e6d372

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=kv/splits/nodes=3 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=928248&tab=buildLog

The test failed on master:
	test.go:498,cluster.go:890,kv.go:185,cluster.go:1199,errgroup.go:58: /home/agent/work/.go/bin/roachprod run teamcity-928248-kv-splits-nodes-3:4 -- ./workload run kv --init --max-ops=1 --concurrency=192 --splits=500000 {pgurl:1-3} returned:
		stderr:
		
		stdout:
		57.577058 426 workload/workload.go:456  finished 322000 of 500000 splits
		I180927 12:05:07.027744 426 workload/workload.go:456  finished 323000 of 500000 splits
		I180927 12:05:19.083255 426 workload/workload.go:456  finished 324000 of 500000 splits
		I180927 12:05:26.807815 426 workload/workload.go:456  finished 325000 of 500000 splits
		I180927 12:05:36.605375 426 workload/workload.go:456  finished 326000 of 500000 splits
		I180927 12:05:46.005447 426 workload/workload.go:456  finished 327000 of 500000 splits
		I180927 12:05:55.196671 426 workload/workload.go:456  finished 328000 of 500000 splits
		I180927 12:06:06.590026 426 workload/workload.go:456  finished 329000 of 500000 splits
		I180927 12:06:15.281723 426 workload/workload.go:456  finished 330000 of 500000 splits
		I180927 12:06:23.601290 426 workload/workload.go:456  finished 331000 of 500000 splits
		Error: ALTER TABLE kv SPLIT AT VALUES (-5961625273234979840): pq: split at key /Table/53/1/-5961625273234979840 failed: context deadline exceeded
		Error:  exit status 1
		: exit status 1
	test.go:498,cluster.go:1220,kv.go:188: Goexit() was called

[nvanbenschoten]

I can see from the logs that this ran for over 10 hours, but the artifact isn't present on TeamCity. I'll give it a few hours and see if it pops up.

[nvanbenschoten]

Yeah, still no artifacts. Probably related to TeamCity hitting a storage limit as this was running. Closing since there's not much to debug.