roachtest: tpccbench/nodes=3/cpu=16 failed #37488

cockroach-teamcity · 2019-05-13T01:44:26Z

SHA: https://github.com/cockroachdb/cockroach/commits/c25518b4e9a723d8de0dba30a95ce0ade7963aed

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1286188&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: unexpected node event: 3: dead

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2019-05-15T01:50:19Z

SHA: https://github.com/cockroachdb/cockroach/commits/ba5c092a726134b73e789c2047f7ec151be7c1a1

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1288263&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: unexpected node event: 2: dead

cockroach-teamcity · 2019-05-17T01:50:13Z

SHA: https://github.com/cockroachdb/cockroach/commits/e4806ba5d174c7aeb88ebbefb64d29747803cefa

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1292134&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: unexpected node event: 2: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1833,tpcc.go:853,tpcc.go:554,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1292134-tpccbench-nodes-3-cpu-16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		2: dead
		1: 4318
		Error:  3: dead, 2: dead

nvanbenschoten · 2019-05-20T05:43:52Z

The failures over the past 3 days are because of #37590.

nvanbenschoten · 2019-05-21T20:14:45Z

Previous three issues addressed by #37701.

nvanbenschoten · 2019-05-21T21:30:45Z

The first few failures here saw a few node deaths, but none of them have logs or output from any of the nodes. It looks like they all got stuck for ~10 hours on the line:

16:35:46 cluster.go:1056: running (fast) consistency checks

It looks like the consistency check blocked until the test's context was canceled. Once the context was canceled, everything else (i.e. grabbing logs, etc.) failed quickly with the error:

stderr:

stdout:
: context canceled

I think this will be fixed by @tbg's recent change to find a live node before running a consistency check: b2904de#diff-cbbad82b1777096ad1f0f0b84e28b7dcR1091. We should let that change bake for a day and see if a failure like this still gets stuck.

tbg · 2019-05-21T21:33:54Z

How would the above diff fix that? Wouldn't it get stuck on the SELECT 1 for 11 hours?

nvanbenschoten · 2019-05-22T19:32:43Z

How would the above diff fix that? Wouldn't it get stuck on the SELECT 1 for 11 hours?

Good point. I just opened #37742 to address this.

The last failure is addressed by #37726.

cockroach-teamcity · 2019-05-23T12:06:45Z

SHA: https://github.com/cockroachdb/cockroach/commits/c9301cf71ea69da451fe5e5ba2c3074a4fe53831

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1303703&tab=buildLog

The test failed on branch=master, cloud=aws:
	cluster.go:1875,tpcc.go:853,tpcc.go:554,test.go:1251: unexpected node event: 1: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1875,tpcc.go:853,tpcc.go:554,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1303703-tpccbench-nodes-3-cpu-16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: 19929
		1: dead
		3: 19559
		Error:  1: dead

nvanbenschoten · 2019-05-23T16:23:43Z

F190523 12:06:14.956091 22565 kv/txn_coord_sender.go:913  [n1,client=10.12.32.99:40188,user=root] unexpected txn state: "sql txn" id=2fb6a2e8 key=/Table/60/1/1875/0 rw=true pri=0.00955542 stat=COMMITTED epo=1 ts=1558613168.062142550,0 orig=1558613168.062142550,0 max=1558613168.562142550,0 wto=false seq=7 int=6
goroutine 22565 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc000056301, 0xc000056300, 0x5567300, 0x16)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1020 +0xd4
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x5cf4540, 0xc000000004, 0x55673cc, 0x16, 0x391, 0xc01cb24500, 0xfd)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:878 +0x93d
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x3b5d920, 0xc01f07ef90, 0x4, 0x2, 0x33fa4c9, 0x18, 0xc0217df0d8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:85 +0x2d8
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x3b5d920, 0xc01f07ef90, 0x1, 0x4, 0x33fa4c9, 0x18, 0xc0217df0d8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:71 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(0x3b5d920, 0xc01f07ef90, 0x33fa4c9, 0x18, 0xc0217df0d8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:182 +0x7e
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).maybeRejectClientLocked(0xc04a72e000, 0x3b5d920, 0xc01f07ef90, 0xc0487bd000, 0x20)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:913 +0x180
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc04a72e000, 0x3b5d920, 0xc01f07ef90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:748 +0x107
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc000716300, 0x3b5d920, 0xc01f07ef90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:648 +0x119
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc02c04f8c0, 0x3b5d920, 0xc01f07ef90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:793 +0x13c

nvanbenschoten · 2019-05-23T16:25:20Z

So tc.mu.txn.Status equals COMMITTED but tc.mu.txnState == txnPending. This might be fallout from parallel commits.

cockroach-teamcity · 2019-05-23T19:45:41Z

SHA: https://github.com/cockroachdb/cockroach/commits/c9301cf71ea69da451fe5e5ba2c3074a4fe53831

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1303699&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1875,tpcc.go:853,tpcc.go:554,test.go:1251: unexpected node event: 3: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1875,tpcc.go:853,tpcc.go:554,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1303699-tpccbench-nodes-3-cpu-16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		2: 19126
		1: 19269
		Error:  3: dead

cockroach-teamcity · 2019-05-24T07:28:11Z

SHA: https://github.com/cockroachdb/cockroach/commits/699f675c73f8420802f92e46f65e6dce52abc12f

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1306272&tab=buildLog

The test failed on branch=release-19.1, cloud=aws:
	cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1306272-tpccbench-nodes-3-cpu-16:4 -- ./workload fixtures import tpcc --warehouses=2500 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		I190524 07:27:46.748678 1 ccl/workloadccl/cliccl/fixtures.go:331  starting import of 9 tables
		Error: importing fixture: importing table item: pq: expected tpcc version "2.1.0" but got "2.0.1"
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_18.191.197.13_2019-05-24T07:27:45Z: exit status 1
		: exit status 1

cockroach-teamcity · 2019-05-24T12:26:37Z

SHA: https://github.com/cockroachdb/cockroach/commits/630a6e9cb3771912cd138f9aa3bea1f0ca9fa7c9

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1306254&tab=buildLog

The test failed on branch=master, cloud=aws:
	cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: unexpected node event: 1: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1306254-tpccbench-nodes-3-cpu-16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		3: 24115
		2: 24539
		Error:  1: dead

cockroach-teamcity · 2019-05-25T07:45:23Z

SHA: https://github.com/cockroachdb/cockroach/commits/db98d5fb943e0a45b3878bdf042838408e9aee40

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1308285&tab=buildLog

The test failed on branch=release-19.1, cloud=aws:
	cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1308285-tpccbench-nodes-3-cpu-16:4 -- ./workload fixtures import tpcc --warehouses=2500 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		I190525 07:45:00.876702 1 ccl/workloadccl/cliccl/fixtures.go:331  starting import of 9 tables
		Error: importing fixture: importing table order_line: pq: expected tpcc version "2.1.0" but got "2.0.1"
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_3.17.204.140_2019-05-25T07:44:59Z: exit status 1
		: exit status 1

cockroach-teamcity · 2019-05-25T15:18:44Z

SHA: https://github.com/cockroachdb/cockroach/commits/fc7e48295cd05f94fd2883498d96d91ad538e559

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1308267&tab=buildLog

The test failed on branch=master, cloud=aws:
	cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: unexpected node event: 1: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1308267-tpccbench-nodes-3-cpu-16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		3: 17141
		2: 17585
		Error:  1: dead

cockroach-teamcity · 2019-05-26T10:56:57Z

SHA: https://github.com/cockroachdb/cockroach/commits/db9c1217a6967fcac2d135cf0f24a4265dc76d77

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1310300&tab=buildLog

The test failed on branch=master, cloud=aws:
	cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: unexpected node event: 1: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1310300-tpccbench-nodes-3-cpu-16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		3: 15805
		2: 15736
		Error:  1: dead

See cockroachdb#37866. This has been failing in roachtests like cockroachdb#37488. This will make it easier to debug until the issue is fixed. Release note: None

37867: kv: replace unexpected txn state assertion with error r=nvanbenschoten a=nvanbenschoten See #37866. This has been failing in roachtests like #37488. This will make it easier to debug until the issue is fixed. Release note: None Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>

cockroach-teamcity · 2019-05-29T09:24:40Z

SHA: https://github.com/cockroachdb/cockroach/commits/61715f0f96f519d599eec6541bbee7394d63209a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1312956&tab=buildLog

The test failed on branch=release-19.1, cloud=aws:
	cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1312956-tpccbench-nodes-3-cpu-16:4 -- ./workload fixtures import tpcc --warehouses=2500 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		ockroach/pkg/ccl/storageccl/export_storage.go:230
		github.com/cockroachdb/cockroach/pkg/ccl/importccl.readInputFiles
			/go/src/github.com/cockroachdb/cockroach/pkg/ccl/importccl/read_import_proc.go:75
		github.com/cockroachdb/cockroach/pkg/ccl/importccl.(*csvInputReader).readFiles
			/go/src/github.com/cockroachdb/cockroach/pkg/ccl/importccl/read_import_csv.go:80
		github.com/cockroachdb/cockroach/pkg/ccl/importccl.(*readImportDataProcessor).doRun.func1
			/go/src/github.com/cockroachdb/cockroach/pkg/ccl/importccl/read_import_proc.go:488
		github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1
			/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:170
		github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
			/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1333
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_3.14.11.86_2019-05-29T09:24:18Z: exit status 1
		: exit status 1

cockroach-teamcity · 2019-05-30T07:46:11Z

SHA: https://github.com/cockroachdb/cockroach/commits/f49f211f8fb2c2aa51182054192ebfcb9c0355f0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1315184&tab=buildLog

The test failed on branch=release-19.1, cloud=aws:
	cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1315184-tpccbench-nodes-3-cpu-16:4 -- ./workload fixtures import tpcc --warehouses=2500 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		I190530 07:45:48.702114 1 ccl/workloadccl/cliccl/fixtures.go:331  starting import of 9 tables
		Error: importing fixture: importing table new_order: pq: expected tpcc version "2.1.0" but got "2.0.1"
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_18.219.97.58_2019-05-30T07:45:47Z: exit status 1
		: exit status 1

cockroach-teamcity · 2019-06-01T07:09:50Z

SHA: https://github.com/cockroachdb/cockroach/commits/83e62d69214aaa0f7b976f764b97b0e21a41cde3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1318707&tab=buildLog

The test failed on branch=release-19.1, cloud=aws:
	cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1318707-tpccbench-nodes-3-cpu-16:4 -- ./workload fixtures import tpcc --warehouses=2500 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		I190601 07:09:26.906037 1 ccl/workloadccl/cliccl/fixtures.go:331  starting import of 9 tables
		Error: importing fixture: importing table item: pq: expected tpcc version "2.1.0" but got "2.0.1"
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_3.14.71.8_2019-06-01T07:09:25Z: exit status 1
		: exit status 1

cockroach-teamcity · 2019-06-04T07:43:24Z

SHA: https://github.com/cockroachdb/cockroach/commits/2ac886fb419af583e7727b3143392585e57b6658

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1321615&tab=buildLog

The test failed on branch=release-19.1, cloud=aws:
	cluster.go:1875,tpcc.go:828,tpcc.go:554,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1321615-tpccbench-nodes-3-cpu-16:4 -- ./workload fixtures import tpcc --warehouses=2500 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		I190604 07:43:03.359563 1 ccl/workloadccl/cliccl/fixtures.go:331  starting import of 9 tables
		Error: importing fixture: importing table history: pq: expected tpcc version "2.1.0" but got "2.0.1"
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_18.220.180.121_2019-06-04T07:43:02Z: exit status 1
		: exit status 1

cockroach-teamcity · 2019-06-05T08:08:38Z

SHA: https://github.com/cockroachdb/cockroach/commits/5a88de2233e1405c0553f2d5380fd24218fac3d2

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1324173&tab=buildLog

The test failed on branch=release-19.1, cloud=aws:
	cluster.go:1872,tpcc.go:825,tpcc.go:551,test.go:1248: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1324173-tpccbench-nodes-3-cpu-16:4 -- ./workload fixtures import tpcc --warehouses=2500 --split --scatter --checks=false {pgurl:1} returned:
		stderr:
		
		stdout:
		I190605 08:08:16.897740 1 ccl/workloadccl/cliccl/fixtures.go:324  starting import of 9 tables
		Error: importing fixture: importing table customer: pq: expected tpcc version "2.1.0" but got "2.0.1"
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_18.221.121.167_2019-06-05T08:08:15Z: exit status 1
		: exit status 1

This was missed during cockroachdb#37726. Closes cockroachdb#37488. Touches cockroachdb#37163. Release note: None

38022: roachtest: use 'fixtures import' from 'cockroach workload' r=danhhz a=tbg This was missed during #37726. Closes #37488. Touches #37163. Release note: None Co-authored-by: Tobias Schottdorf <tobias.schottdorf@gmail.com>

cockroach-teamcity added this to the 19.2 milestone May 13, 2019

cockroach-teamcity assigned nvanbenschoten May 13, 2019

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels May 13, 2019

This comment has been minimized.

Sign in to view

nvanbenschoten mentioned this issue May 20, 2019

workload/tpcc: duplicate FK detection during initialization broken #37590

Closed

This comment has been minimized.

Sign in to view

nvanbenschoten mentioned this issue May 23, 2019

roachtest: schemachange/index/tpcc/w=1000 failed #36094

Closed

nvanbenschoten mentioned this issue May 26, 2019

kv: replace unexpected txn state assertion with error #37867

Merged

tbg mentioned this issue Jun 4, 2019

roachtest: use 'fixtures import' from 'cockroach workload' #38022

Merged

tbg added a commit to tbg/cockroach that referenced this issue Jun 5, 2019

roachtest: use 'fixtures import' from 'cockroach workload'

95fa2a2

This was missed during cockroachdb#37726. Closes cockroachdb#37488. Touches cockroachdb#37163. Release note: None

craig bot closed this as completed in #38022 Jun 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: tpccbench/nodes=3/cpu=16 failed #37488

roachtest: tpccbench/nodes=3/cpu=16 failed #37488

cockroach-teamcity commented May 13, 2019

cockroach-teamcity commented May 15, 2019

cockroach-teamcity commented May 17, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nvanbenschoten commented May 20, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nvanbenschoten commented May 21, 2019

nvanbenschoten commented May 21, 2019

tbg commented May 21, 2019

This comment has been minimized.

nvanbenschoten commented May 22, 2019

cockroach-teamcity commented May 23, 2019

nvanbenschoten commented May 23, 2019

nvanbenschoten commented May 23, 2019

cockroach-teamcity commented May 23, 2019

cockroach-teamcity commented May 24, 2019

cockroach-teamcity commented May 24, 2019

cockroach-teamcity commented May 25, 2019

cockroach-teamcity commented May 25, 2019

cockroach-teamcity commented May 26, 2019

cockroach-teamcity commented May 29, 2019

cockroach-teamcity commented May 30, 2019

cockroach-teamcity commented Jun 1, 2019

cockroach-teamcity commented Jun 4, 2019

cockroach-teamcity commented Jun 5, 2019

roachtest: tpccbench/nodes=3/cpu=16 failed #37488

roachtest: tpccbench/nodes=3/cpu=16 failed #37488

Comments

cockroach-teamcity commented May 13, 2019

cockroach-teamcity commented May 15, 2019

cockroach-teamcity commented May 17, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nvanbenschoten commented May 20, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nvanbenschoten commented May 21, 2019

nvanbenschoten commented May 21, 2019

tbg commented May 21, 2019

This comment has been minimized.

nvanbenschoten commented May 22, 2019

cockroach-teamcity commented May 23, 2019

nvanbenschoten commented May 23, 2019

nvanbenschoten commented May 23, 2019

cockroach-teamcity commented May 23, 2019

cockroach-teamcity commented May 24, 2019

cockroach-teamcity commented May 24, 2019

cockroach-teamcity commented May 25, 2019

cockroach-teamcity commented May 25, 2019

cockroach-teamcity commented May 26, 2019

cockroach-teamcity commented May 29, 2019

cockroach-teamcity commented May 30, 2019

cockroach-teamcity commented Jun 1, 2019

cockroach-teamcity commented Jun 4, 2019

cockroach-teamcity commented Jun 5, 2019