roachtest: version/mixedWith=v2.0.5/nodes=5 failed

[cockroach-teamcity]

SHA: https://github.com/cockroachdb/cockroach/commits/cff3e8870a9f2f84feb37415f223ce6922487504

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=version/mixedWith=v2.0.5/nodes=5 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1085650&tab=buildLog

The test failed on master:
	test.go:696,cluster.go:1511,version.go:223,version.go:233: unexpected node event: 1: dead

[tbg]

Happened again:

F190109 16:36:06.083008 23 server/server.go:1617  [n1] failed to run migration "create default databases": create-default-db: unsupported request
goroutine 23 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc4204da200, 0xc4204da2a0, 0x4765b00, 0x10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1005 +0xcf
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x4edc400, 0xc400000004, 0x4765b42, 0x10, 0x651, 0xc4252171a0, 0x5f)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:872 +0x939
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x35f5520, 0xc420987140, 0x4, 0x2, 0x0, 0x0, 0xc424f70be8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:85 +0x2e5
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x35f5520, 0xc420987140, 0x1, 0xc400000004, 0x0, 0x0, 0xc424f70be8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:71 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x35f5520, 0xc420987140, 0xc424f70be8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:191 +0x6c
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start(0xc42098a700, 0x35f5520, 0xc420987140, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1617 +0x307d
github.com/cockroachdb/cockroach/pkg/cli.runStart.func3.2(0xc4201de3f0, 0xc4204dc0d0, 0xc4202ac650, 0x35f5520, 0xc420236540, 0x495a505, 0xed3c819ed, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:605 +0x109
github.com/cockroachdb/cockroach/pkg/cli.runStart.func3(0xc4204dc0d0, 0x35f5520, 0xc420236540, 0x36287e0, 0xc4205f8000, 0xc4201de3f0, 0xc4202ac650, 0x495a505, 0xed3c819ed, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:710 +0x11c
created by github.com/cockroachdb/cockroach/pkg/cli.runStart
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:561 +0x6f8

#33578