roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #41553

cockroach-teamcity · 2019-10-12T18:34:32Z

SHA: https://github.com/cockroachdb/cockroach/commits/2859f9b35f1ed6d1a4e960a106e951d81de21287

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1535775&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191012-1535775/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:704: test timed out (10h0m0s)

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2019-10-15T17:51:18Z

SHA: https://github.com/cockroachdb/cockroach/commits/a40cefbb5bd5e9d34f5db930334a385d0934d2d0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1538863&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191015-1538863/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:704: test timed out (10h0m0s)

cockroach-teamcity · 2019-10-17T18:00:59Z

SHA: https://github.com/cockroachdb/cockroach/commits/06106c6209709b9ef3f1d194a70e71ddf812c4ce

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1542489&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191017-1542489/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:704: test timed out (10h0m0s)

cockroach-teamcity · 2019-10-22T17:48:54Z

SHA: https://github.com/cockroachdb/cockroach/commits/9c58b4b7661d8a50e8c5403a62596438c52538cc

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1551245&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191022-1551245/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:704: test timed out (10h0m0s)

cockroach-teamcity · 2019-10-23T17:58:06Z

SHA: https://github.com/cockroachdb/cockroach/commits/5612fc5f44e34cf10f60e63ed5a53b6dfa867190

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1554254&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191023-1554254/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:704: test timed out (10h0m0s)

cockroach-teamcity · 2019-10-25T20:35:52Z

SHA: https://github.com/cockroachdb/cockroach/commits/1a940ddc06876a1d6511e614391fcffcbe42f664

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1557715&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191025-1557715/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:704: test timed out (10h0m0s)

cockroach-teamcity · 2019-10-29T20:19:50Z

SHA: https://github.com/cockroachdb/cockroach/commits/a57647381a4714b48f6ec6dec0bf766eaa6746dd

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1561660&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191029-1561660/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-10-30T19:27:44Z

SHA: https://github.com/cockroachdb/cockroach/commits/a4d88c2c5ab6131878d2b4552446d94fd93b1553

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1563612&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191030-1563612/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-03T19:46:40Z

SHA: https://github.com/cockroachdb/cockroach/commits/b9dc16b9d6cbd4627bac4b6329e87e29e52401b9

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1571295&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191103-1571295/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-06T05:22:16Z

SHA: https://github.com/cockroachdb/cockroach/commits/239513342a2d23f683bbc1d386f87ff59cc78d10

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1575479&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=provisional_201910141814_v19.2.0-rc.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191106-1575479/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:2163,tpcc.go:720,tpcc.go:561,test_runner.go:697: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1575479-1573004318-27-n10cpu4:10 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 {pgurl:1} returned:
		stderr:
		
		stdout:
		0000 rows, 120000000 index entries, 14 MiB)
		I191106 04:56:56.648476 31 ccl/workloadccl/fixture.go:547  loaded 34 GiB table customer in 21m54.061060937s (60000000 rows, 60000000 index entries, 27 MiB)
		I191106 05:11:24.644166 85 ccl/workloadccl/fixture.go:547  loaded 46 GiB table order_line in 36m22.057046179s (600008449 rows, 600008449 index entries, 22 MiB)
		I191106 05:21:31.372208 84 ccl/workloadccl/fixture.go:547  loaded 63 GiB table stock in 46m28.784862513s (200000000 rows, 200000000 index entries, 23 MiB)
		I191106 05:21:31.626896 1 ccl/workloadccl/cliccl/fixtures.go:286  restored 155 GiB bytes in 9 tables (took 46m29.051627359s, 56.78 MiB/s)
		Error: Could not postload: could not partition tables: Couldn't exec "\n\t\t\tALTER INDEX item@replicated_idx_0\n\t\t\tCONFIGURE ZONE USING num_replicas = COPY FROM PARENT, constraints = '{\"+rack=0\":1}', lease_preferences = '[[+rack=0]]'": pq: could not validate zone config: when per-replica constraints are set, num_replicas must be set as well
		Error:  exit status 1
		: exit status 1

cockroach-teamcity · 2019-11-06T14:42:50Z

SHA: https://github.com/cockroachdb/cockroach/commits/2535f0363b5d9b1e466ca3144fe5bcf7ff17e63c

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1575491&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=provisional_201911052027_v19.2.0-rc.4, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191106-1575491/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-07T00:14:39Z

SHA: https://github.com/cockroachdb/cockroach/commits/2535f0363b5d9b1e466ca3144fe5bcf7ff17e63c

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1577529&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=provisional_201911052027_v19.2.0-rc.4, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191106-1577529/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-07T20:07:52Z

SHA: https://github.com/cockroachdb/cockroach/commits/dcddfec777c9c799ed5717e8f14899cfdb03b6e6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1578929&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191107-1578929/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-08T11:09:12Z

SHA: https://github.com/cockroachdb/cockroach/commits/e206eec2aaac3620d1d27873b830bc5f781c94aa

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1580926&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191108-1580926/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:2163,tpcc.go:720,tpcc.go:561,test_runner.go:697: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1580926-1573197102-27-n10cpu4:10 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 {pgurl:1} returned:
		stderr:
		
		stdout:
		able history in 12m47.548866551s (60000000 rows, 120000000 index entries, 11 MiB)
		I191108 10:59:26.870895 29 ccl/workloadccl/fixture.go:547  loaded 34 GiB table customer in 26m32.072134499s (60000000 rows, 60000000 index entries, 22 MiB)
		I191108 11:02:41.902323 83 ccl/workloadccl/fixture.go:547  loaded 46 GiB table order_line in 29m47.104809048s (600008449 rows, 600008449 index entries, 26 MiB)
		I191108 11:08:37.554080 82 ccl/workloadccl/fixture.go:547  loaded 63 GiB table stock in 35m42.756210628s (200000000 rows, 200000000 index entries, 30 MiB)
		I191108 11:08:37.932321 1 ccl/workloadccl/cliccl/fixtures.go:286  restored 155 GiB bytes in 9 tables (took 35m43.148568903s, 73.90 MiB/s)
		Error: Could not postload: could not partition tables: Couldn't exec "\n\t\t\tALTER INDEX item@replicated_idx_2\n\t\t\tCONFIGURE ZONE USING num_replicas = COPY FROM PARENT, constraints = '{\"+rack=2\":1}', lease_preferences = '[[+rack=2]]'": pq: could not validate zone config: at least one replica is required
		Error:  exit status 1
		: exit status 1

cockroach-teamcity · 2019-11-08T11:40:43Z

SHA: https://github.com/cockroachdb/cockroach/commits/ca3615568aa509606410d36bc7cd70eec0484c26

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1580962&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=release-2.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191108-1580962/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:2163,tpcc.go:720,tpcc.go:561,test_runner.go:697: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1580962-1573197016-64-n10cpu4:10 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 {pgurl:1} returned:
		stderr:
		
		stdout:
		ixture.go:547  loaded 8.5 GiB table history in 19m50.077875118s (60000000 rows, 120000000 index entries, 7.4 MiB)
		I191108 11:18:19.439617 95 ccl/workloadccl/fixture.go:547  loaded 34 GiB table customer in 20m40.002233539s (60000000 rows, 60000000 index entries, 28 MiB)
		I191108 11:25:29.805875 100 ccl/workloadccl/fixture.go:547  loaded 63 GiB table stock in 27m50.368083172s (200000000 rows, 200000000 index entries, 38 MiB)
		I191108 11:40:11.534256 101 ccl/workloadccl/fixture.go:547  loaded 46 GiB table order_line in 42m32.096738501s (600008449 rows, 600008449 index entries, 18 MiB)
		I191108 11:40:11.981222 1 ccl/workloadccl/cliccl/fixtures.go:286  restored 155 GiB bytes in 9 tables (took 42m32.544309082s, 62.04 MiB/s)
		Error: Could not postload: could not partition tables: Couldn't exec "\n\t\t\tALTER INDEX item@replicated_idx_0\n\t\t\tCONFIGURE ZONE USING num_replicas = COPY FROM PARENT, constraints = '{\"+rack=0\":1}', lease_preferences = '[[+rack=0]]'": pq: syntax error at or near "from"
		Error:  exit status 1
		: exit status 1

cockroach-teamcity · 2019-11-08T20:16:02Z

SHA: https://github.com/cockroachdb/cockroach/commits/ae07b57a07428f3fb9b1d26d98b8204766091dad

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1580908&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191108-1580908/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-12T19:25:29Z

SHA: https://github.com/cockroachdb/cockroach/commits/64162f27783bd59799531ff977f0fd1d0fd5ae86

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1587139&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191112-1587139/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-12T20:06:44Z

SHA: https://github.com/cockroachdb/cockroach/commits/0e9dd73f803247cdcfd06f51ce6b23396af1b9f5

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1587121&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191112-1587121/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-13T19:58:07Z

SHA: https://github.com/cockroachdb/cockroach/commits/07b5c74deee727e9e74705f4a989e0f6923d99d6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1589096&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191113-1589096/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-14T20:08:47Z

SHA: https://github.com/cockroachdb/cockroach/commits/8622fad01478fb4a4f05b5579eb0b8561c02e491

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1591079&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191114-1591079/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-20T19:49:17Z

SHA: https://github.com/cockroachdb/cockroach/commits/647db0fecd18d8de760443131c6edb33828a619a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1599851&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191120-1599851/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-20T19:51:12Z

SHA: https://github.com/cockroachdb/cockroach/commits/1c3fce73cd25fa69e5a8c05ce8e215d66b4f49e4

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1599833&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191120-1599833/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-21T11:01:52Z

SHA: https://github.com/cockroachdb/cockroach/commits/66290e232d6af34c4d09b2f90f71d520ed47dbe8

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1602090&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191121-1602090/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:2163,tpcc.go:720,tpcc.go:561,test_runner.go:697: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1602090-1574319796-25-n10cpu4:10 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 {pgurl:1} returned:
		stderr:
		
		stdout:
		ble history in 22m46.905922001s (60000000 rows, 120000000 index entries, 6.4 MiB)
		I191121 10:43:10.147923 75 ccl/workloadccl/fixture.go:547  loaded 63 GiB table stock in 28m27.492643664s (200000000 rows, 200000000 index entries, 38 MiB)
		I191121 10:44:27.435943 70 ccl/workloadccl/fixture.go:547  loaded 34 GiB table customer in 29m44.780611889s (60000000 rows, 60000000 index entries, 20 MiB)
		I191121 11:01:17.327592 76 ccl/workloadccl/fixture.go:547  loaded 46 GiB table order_line in 46m34.672598679s (600008449 rows, 600008449 index entries, 17 MiB)
		I191121 11:01:17.648973 1 ccl/workloadccl/cliccl/fixtures.go:286  restored 155 GiB bytes in 9 tables (took 46m35.008421026s, 56.66 MiB/s)
		Error: Could not postload: could not partition tables: Couldn't exec "\n\t\t\tALTER INDEX item@replicated_idx_2\n\t\t\tCONFIGURE ZONE USING num_replicas = COPY FROM PARENT, constraints = '{\"+rack=2\":1}', lease_preferences = '[[+rack=2]]'": pq: could not validate zone config: at least one replica is required
		Error:  exit status 1
		: exit status 1

cockroach-teamcity · 2019-11-21T19:52:58Z

SHA: https://github.com/cockroachdb/cockroach/commits/db252551659b8c404be584e39f2af642d01caee3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1602108&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191121-1602108/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

cockroach-teamcity · 2019-11-21T20:14:53Z

SHA: https://github.com/cockroachdb/cockroach/commits/205330426e50b966ede8a4644d77570d8265fe66

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1602072&tab=artifacts#/tpccbench/nodes=9/cpu=4/chaos/partition

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191121-1602072/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:712: test timed out (10h0m0s)

andreimatei · 2020-01-02T15:01:17Z

Btw, the required version I think is 19.1 according to commits like this 72513bd

cockroach-teamcity · 2020-01-02T20:27:23Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on master@a99740bb34c03ab3af7558129edce9c17c88d4f9:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200102-1670273/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:731: test timed out (10h0m0s)

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-03T13:25:20Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.1@5b73cd419b514452492226c0ea1b19639356ebb8:

		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:561
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2233
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1835
		    main.loadTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:628
		    main.runTPCCBench.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:718
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1671751-1578035659-84-n10cpu4:10 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 {pgurl:1} returned:
		    stderr:
		    
		    stdout:
		    ble history in 17m44.195153476s (60000000 rows, 120000000 index entries, 8.2 MiB)
		    I200103 12:55:41.918149 49 ccl/workloadccl/fixture.go:547  loaded 34 GiB table customer in 18m16.030096693s (60000000 rows, 60000000 index entries, 32 MiB)
		    I200103 13:14:03.406672 86 ccl/workloadccl/fixture.go:547  loaded 63 GiB table stock in 36m37.518647384s (200000000 rows, 200000000 index entries, 29 MiB)
		    I200103 13:24:43.987937 87 ccl/workloadccl/fixture.go:547  loaded 46 GiB table order_line in 47m18.099980282s (600008449 rows, 600008449 index entries, 17 MiB)
		    I200103 13:24:44.426767 1 ccl/workloadccl/cliccl/fixtures.go:286  restored 155 GiB bytes in 9 tables (took 47m18.605197466s, 55.79 MiB/s)
		    Error: Could not postload: could not partition tables: Couldn't exec "\n\t\t\tALTER INDEX item@replicated_idx_2\n\t\t\tCONFIGURE ZONE USING num_replicas = COPY FROM PARENT, constraints = '{\"+rack=2\":1}', lease_preferences = '[[+rack=2]]'": pq: could not validate zone config: at least one replica is required
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-07T06:05:08Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on 43008@b555d16b4455f7122f4211082512820742f8116f:

		        github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		        	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		        runtime.goexit
		        	/usr/local/go/src/runtime/asm_amd64.s:1357
		      - error with embedded safe details: %s returned:
		        stderr:
		        %s
		        stdout:
		        %s
		        -- arg 1: <string>
		        -- arg 2: <string>
		        -- arg 3: <string>
		      - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1676232-1578345768-26-n10cpu4:10 -- ./workload run tpcc --warehouses=2000 --active-warehouses=1050 --tolerate-errors --scatter --ramp=5m0s --duration=10m0s --partitions=3 {pgurl:10} --histograms=perf/warehouses=1050/stats.json returned:
		        stderr:
		        
		        stdout:
		        , d_next_o_id: EOF
		        Initializing 2100 connections...
		        Initializing 10500 workers and preparing statements...
		        I200107 06:00:06.149682 1 workload/cli/run.go:361  retrying after error while creating load: preparing 
		        		UPDATE district
		        		SET d_next_o_id = d_next_o_id + 1
		        		WHERE d_w_id = $1 AND d_id = $2
		        		RETURNING d_tax, d_next_o_id: EOF
		        Initializing 2100 connections...
		        Initializing 10500 workers and preparing statements...
		        I200107 06:01:09.717119 1 workload/cli/run.go:361  retrying after error while creating load: preparing 
		        		UPDATE district
		        		SET d_next_o_id = d_next_o_id + 1
		        		WHERE d_w_id = $1 AND d_id = $2
		        		RETURNING d_tax, d_next_o_id: EOF
		        Initializing 2100 connections...
		        Initializing 10500 workers and preparing statements...
		        I200107 06:02:13.862275 1 workload/cli/run.go:361  retrying after error while creating load: preparing 
		        		UPDATE district
		        		SET d_next_o_id = d_next_o_id + 1
		        		WHERE d_w_id = $1 AND d_id = $2
		        		RETURNING d_tax, d_next_o_id: EOF
		        Initializing 2100 connections...
		        Error:  exit status 255:
		      - exit status 1
		    error running tpcc load generator
		    main.runTPCCBench.func3.1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:809
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-07T11:19:37Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on master@52261c39c65aecf960e3c8d54089a66d378af610:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200107-1676715/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:2189,tpcc.go:720,tpcc.go:561,test_runner.go:716: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2188
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:720
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:561
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 7: dead

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-07T19:55:34Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@f3bcbd305602c89d52cb7ea4cb114f362c5aee23:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200107-1676733/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:731: test timed out (10h0m0s)

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-14T19:34:54Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@7f2413b36fe1ea31a08c3ddd0d1e4b1cd006db37:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200114-1688037/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:731: test timed out (10h0m0s)

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-15T11:16:08Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.1@67be21c787f727df04e77135ad19d134b8e5c762:

		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1847
		    main.loadTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:628
		    main.runTPCCBench.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:718
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1690444-1579071920-25-n10cpu4:10 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 {pgurl:1} returned:
		    stderr:
		    
		    stdout:
		    I200115 10:27:56.644171 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		    I200115 10:27:57.887112 14 ccl/workloadccl/fixture.go:547  loaded 106 KiB table warehouse in 1.225985842s (2000 rows, 0 index entries, 87 KiB)
		    I200115 10:29:49.523911 101 ccl/workloadccl/fixture.go:547  loaded 7.8 MiB table item in 1m52.862963832s (100000 rows, 0 index entries, 70 KiB)
		    I200115 10:32:20.104814 100 ccl/workloadccl/fixture.go:547  loaded 254 MiB table new_order in 4m23.443874062s (18000000 rows, 0 index entries, 989 KiB)
		    I200115 10:39:49.909652 15 ccl/workloadccl/fixture.go:547  loaded 2.0 MiB table district in 11m53.248450902s (20000 rows, 0 index entries, 2.8 KiB)
		    I200115 10:41:54.580601 99 ccl/workloadccl/fixture.go:547  loaded 2.7 GiB table order in 13m57.91964886s (60000000 rows, 60000000 index entries, 3.3 MiB)
		    I200115 10:43:37.261292 98 ccl/workloadccl/fixture.go:547  loaded 8.5 GiB table history in 15m40.600406417s (60000000 rows, 120000000 index entries, 9.3 MiB)
		    I200115 10:54:21.536709 16 ccl/workloadccl/fixture.go:547  loaded 34 GiB table customer in 26m24.875491817s (60000000 rows, 60000000 index entries, 22 MiB)
		    I200115 11:06:57.073101 102 ccl/workloadccl/fixture.go:547  loaded 63 GiB table stock in 39m0.411901145s (200000000 rows, 200000000 index entries, 27 MiB)
		    I200115 11:15:29.600867 103 ccl/workloadccl/fixture.go:547  loaded 46 GiB table order_line in 47m32.939856875s (600008449 rows, 600008449 index entries, 16 MiB)
		    I200115 11:15:30.002536 1 ccl/workloadccl/cliccl/fixtures.go:286  restored 155 GiB bytes in 9 tables (took 47m33.357883163s, 55.50 MiB/s)
		    Error: Could not postload: could not partition tables: Couldn't exec "\n\t\t\tALTER INDEX item@replicated_idx_2\n\t\t\tCONFIGURE ZONE USING num_replicas = COPY FROM PARENT, constraints = '{\"+rack=2\":1}', lease_preferences = '[[+rack=2]]'": pq: could not validate zone config: at least one replica is required
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-15T20:02:12Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@99a4ab5a34b310338098a6aafb6621920fd9d4cc:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200115-1690462/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:731: test timed out (10h0m0s)

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-16T22:09:03Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200116-1693614/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:2201,tpcc.go:720,tpcc.go:561,test_runner.go:716: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2192
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2200
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:720
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:561
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 2: dead

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-17T10:54:48Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.1@265263f40589020c2c6505e102a3c5d398054377:

		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1847
		    main.loadTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:628
		    main.runTPCCBench.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:718
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1695931-1579244662-25-n10cpu4:10 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 {pgurl:1} returned:
		    stderr:
		    
		    stdout:
		    I200117 10:16:06.452006 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		    I200117 10:16:08.133179 89 ccl/workloadccl/fixture.go:547  loaded 7.8 MiB table item in 1.667555099s (100000 rows, 0 index entries, 4.7 MiB)
		    I200117 10:16:09.058577 83 ccl/workloadccl/fixture.go:547  loaded 106 KiB table warehouse in 2.592776546s (2000 rows, 0 index entries, 41 KiB)
		    I200117 10:26:07.337638 84 ccl/workloadccl/fixture.go:547  loaded 2.0 MiB table district in 10m0.871592028s (20000 rows, 0 index entries, 3.4 KiB)
		    I200117 10:27:09.585225 88 ccl/workloadccl/fixture.go:547  loaded 254 MiB table new_order in 11m3.119667066s (18000000 rows, 0 index entries, 393 KiB)
		    I200117 10:28:47.976534 86 ccl/workloadccl/fixture.go:547  loaded 8.5 GiB table history in 12m41.510970838s (60000000 rows, 120000000 index entries, 12 MiB)
		    I200117 10:31:10.472515 87 ccl/workloadccl/fixture.go:547  loaded 2.7 GiB table order in 15m4.006927538s (60000000 rows, 60000000 index entries, 3.0 MiB)
		    I200117 10:42:03.232705 85 ccl/workloadccl/fixture.go:547  loaded 34 GiB table customer in 25m56.766627802s (60000000 rows, 60000000 index entries, 23 MiB)
		    I200117 10:53:37.413695 91 ccl/workloadccl/fixture.go:547  loaded 46 GiB table order_line in 37m30.947993121s (600008449 rows, 600008449 index entries, 21 MiB)
		    I200117 10:54:08.736222 90 ccl/workloadccl/fixture.go:547  loaded 63 GiB table stock in 38m2.270383462s (200000000 rows, 200000000 index entries, 28 MiB)
		    I200117 10:54:09.150032 1 ccl/workloadccl/cliccl/fixtures.go:286  restored 155 GiB bytes in 9 tables (took 38m2.697373987s, 69.38 MiB/s)
		    Error: Could not postload: could not partition tables: Couldn't exec "\n\t\t\tALTER INDEX item@replicated_idx_2\n\t\t\tCONFIGURE ZONE USING num_replicas = COPY FROM PARENT, constraints = '{\"+rack=2\":1}', lease_preferences = '[[+rack=2]]'": pq: could not validate zone config: at least one replica is required
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-17T11:20:14Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on master@96d7e50acfaa63b5b60addd7cec8e750a4a4d333:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200117-1695913/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:1296,context.go:135,cluster.go:1285,test_runner.go:753: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1695913-1579247026-25-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		8: 5676
		2: 4580
		6: 4888
		1: 5389
		7: 6472
		5: dead
		3: 5554
		9: 3876
		4: 4202
		Error:  5: dead

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-17T20:05:56Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@e00e840cad174e32112fac28e8b2264723997759:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200117-1695949/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:731: test timed out (10h0m0s)

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition

make stressrace TESTS=tpccbench/nodes=9/cpu=4/chaos/partition PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-18T10:52:32Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.1@a8444e28d3e7bf07c8ee0c6d5809e7466b57f6f9:

		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1847
		    main.loadTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:628
		    main.runTPCCBench.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:718
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1698230-1579331271-24-n10cpu4:10 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 {pgurl:1} returned:
		    stderr:
		    
		    stdout:
		    I200118 10:21:45.936561 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		    I200118 10:21:46.597729 56 ccl/workloadccl/fixture.go:547  loaded 106 KiB table warehouse in 648.47498ms (2000 rows, 0 index entries, 164 KiB)
		    I200118 10:22:57.779694 62 ccl/workloadccl/fixture.go:547  loaded 7.8 MiB table item in 1m11.830175738s (100000 rows, 0 index entries, 111 KiB)
		    I200118 10:26:54.762647 57 ccl/workloadccl/fixture.go:547  loaded 2.0 MiB table district in 5m8.813287189s (20000 rows, 0 index entries, 6.5 KiB)
		    I200118 10:29:02.337028 61 ccl/workloadccl/fixture.go:547  loaded 254 MiB table new_order in 7m16.387441193s (18000000 rows, 0 index entries, 597 KiB)
		    I200118 10:32:58.634258 59 ccl/workloadccl/fixture.go:547  loaded 8.5 GiB table history in 11m12.684860047s (60000000 rows, 120000000 index entries, 13 MiB)
		    I200118 10:34:06.812406 60 ccl/workloadccl/fixture.go:547  loaded 2.7 GiB table order in 12m20.862867461s (60000000 rows, 60000000 index entries, 3.7 MiB)
		    I200118 10:45:26.407124 64 ccl/workloadccl/fixture.go:547  loaded 46 GiB table order_line in 23m40.457796067s (600008449 rows, 600008449 index entries, 33 MiB)
		    I200118 10:51:41.669251 58 ccl/workloadccl/fixture.go:547  loaded 34 GiB table customer in 29m55.7198602s (60000000 rows, 60000000 index entries, 20 MiB)
		    I200118 10:51:57.869757 63 ccl/workloadccl/fixture.go:547  loaded 63 GiB table stock in 30m11.920257094s (200000000 rows, 200000000 index entries, 35 MiB)
		    I200118 10:51:58.133177 1 ccl/workloadccl/cliccl/fixtures.go:286  restored 155 GiB bytes in 9 tables (took 30m12.196032781s, 87.39 MiB/s)
		    Error: Could not postload: could not partition tables: Couldn't exec "\n\t\t\tALTER INDEX item@replicated_idx_2\n\t\t\tCONFIGURE ZONE USING num_replicas = COPY FROM PARENT, constraints = '{\"+rack=2\":1}', lease_preferences = '[[+rack=2]]'": pq: could not validate zone config: at least one replica is required
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
roachdash

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-19T20:13:29Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on master@3bb183403b4bc75c25fbcfc69d7d35de76d2b984:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200119-1699769/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:731: test timed out (10h0m0s)

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
roachdash

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-22T04:13:35Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on provisional_202001212141_v19.1.7@a8444e28d3e7bf07c8ee0c6d5809e7466b57f6f9:

		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1847
		    main.loadTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:628
		    main.runTPCCBench.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:718
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1702949-1579652818-24-n10cpu4:10 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 {pgurl:1} returned:
		    stderr:
		    
		    stdout:
		    I200122 03:39:17.728614 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		    I200122 03:39:19.425882 89 ccl/workloadccl/fixture.go:547  loaded 7.8 MiB table item in 1.683549417s (100000 rows, 0 index entries, 4.6 MiB)
		    I200122 03:39:36.228039 84 ccl/workloadccl/fixture.go:547  loaded 2.0 MiB table district in 18.485712517s (20000 rows, 0 index entries, 109 KiB)
		    I200122 03:41:12.655358 83 ccl/workloadccl/fixture.go:547  loaded 106 KiB table warehouse in 1m54.9130749s (2000 rows, 0 index entries, 946 B)
		    I200122 03:47:45.622658 88 ccl/workloadccl/fixture.go:547  loaded 254 MiB table new_order in 8m27.88036015s (18000000 rows, 0 index entries, 513 KiB)
		    I200122 03:52:01.494066 86 ccl/workloadccl/fixture.go:547  loaded 8.5 GiB table history in 12m43.751926696s (60000000 rows, 120000000 index entries, 12 MiB)
		    I200122 03:53:05.579096 87 ccl/workloadccl/fixture.go:547  loaded 2.7 GiB table order in 13m47.836864467s (60000000 rows, 60000000 index entries, 3.3 MiB)
		    I200122 04:04:50.629028 85 ccl/workloadccl/fixture.go:547  loaded 34 GiB table customer in 25m32.886446054s (60000000 rows, 60000000 index entries, 23 MiB)
		    I200122 04:07:52.731290 90 ccl/workloadccl/fixture.go:547  loaded 63 GiB table stock in 28m34.988751795s (200000000 rows, 200000000 index entries, 37 MiB)
		    I200122 04:12:59.395167 91 ccl/workloadccl/fixture.go:547  loaded 46 GiB table order_line in 33m41.652900523s (600008449 rows, 600008449 index entries, 23 MiB)
		    I200122 04:12:59.748199 1 ccl/workloadccl/cliccl/fixtures.go:286  restored 155 GiB bytes in 9 tables (took 33m42.019242871s, 78.32 MiB/s)
		    Error: Could not postload: could not partition tables: Couldn't exec "\n\t\t\tALTER INDEX item@replicated_idx_2\n\t\t\tCONFIGURE ZONE USING num_replicas = COPY FROM PARENT, constraints = '{\"+rack=2\":1}', lease_preferences = '[[+rack=2]]'": pq: could not validate zone config: at least one replica is required
		    Error:  exit status 1:
		  - exit status 1

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
roachdash

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-22T12:16:44Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on master@c76ad970e73e606bc55a372d93f9d2d6acb32c9c:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200122-1703406/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:1296,context.go:135,cluster.go:1285,test_runner.go:753: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1703406-1579679704-25-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: skipped
		1: dead
		9: 10787
		5: 10564
		7: 9457
		8: 8252
		6: 9338
		4: 8096
		3: 8122
		2: 7008
		Error:  1: dead

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
roachdash

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-22T20:23:09Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on release-19.2@4b808c3af1a7318f18d708c89b4d88b9e26377b9:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200122-1703424/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	test_runner.go:731: test timed out (10h0m0s)

Repro

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
roachdash

_{powered by pkg/cmd/internal/issues}

Fixes cockroachdb#44231. Fixes cockroachdb#41553. This PR fixes a bug caused by backports. In short, the backport of cockroachdb#41506 to 19.1 did not include some code in cockroachdb#40493 which allowed a subzone placeholder to inherit fields from its parents. This caused a problem the second time a index's zone was altered, because its parent (now a subzone placeholder) would no longer be able to inherit values from its parent. We couldn't backport cockroachdb#40493 because it contained features that were only released in 19.2, and since there was not a test checking this specific behavior, we didn't catch the regression on 19.1 Release note (bug fix): Fix a bug where repeated use of COPY FROM PARENT on an index or partition could cause an unexpected validation error.

cockroach-teamcity · 2020-01-23T11:21:37Z

(roachtest).tpccbench/nodes=9/cpu=4/chaos/partition failed on master@545c96573d36519c27115278187fdd441f54a64d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200123-1705428/tpccbench/nodes=9/cpu=4/chaos/partition/run_1
	cluster.go:2201,tpcc.go:720,tpcc.go:561,test_runner.go:719: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2192
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2200
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:720
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:561
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:719
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 7: dead

More

Artifacts: /tpccbench/nodes=9/cpu=4/chaos/partition
Related:

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44287 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44259 roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed C-test-failure O-roachtest O-robot branch-provisional_202001221958_v2.1.11 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

Fixes cockroachdb#44231. Fixes cockroachdb#41553. This PR fixes a bug caused by backports. In short, the backport of cockroachdb#41506 to 19.1 did not include some code in cockroachdb#40493 which allowed a subzone placeholder to inherit fields from its parents. This caused a problem the second time a index's zone was altered, because its parent (now a subzone placeholder) would no longer be able to inherit values from its parent. We couldn't backport cockroachdb#40493 because it contained features that were only released in 19.2, and since there was not a test checking this specific behavior, we didn't catch the regression on 19.1. Therefore, this PR contains a partial backport of cockroachdb#40493. Release note (bug fix): Fix a bug where repeated use of COPY FROM PARENT on an index or partition could cause an unexpected validation error.

rohany · 2020-01-23T15:54:52Z

Closing this because the #44266 should fix the zone config issue on 19.1 that keeps appearing.

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Oct 12, 2019

cockroach-teamcity added this to the 19.1 milestone Oct 12, 2019

cockroach-teamcity assigned andreimatei Oct 12, 2019

RaduBerinde mentioned this issue Nov 7, 2019

release: 19.2.0-rc.4 #42193

Closed

18 tasks

andreimatei assigned ajwerner and unassigned andreimatei Jan 2, 2020

tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020

cockroach-teamcity mentioned this issue Jan 23, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44259

Closed

rohany mentioned this issue Jan 23, 2020

zone: fix error caused subzone placeholders not inheriting from parents #44266

Merged

cockroach-teamcity mentioned this issue Jan 23, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44287

Closed

cockroach-teamcity mentioned this issue Jan 23, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #44293

Closed

rohany closed this as completed Jan 23, 2020

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #41553

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #41553

Comments

cockroach-teamcity commented Oct 12, 2019

cockroach-teamcity commented Oct 15, 2019

cockroach-teamcity commented Oct 17, 2019

cockroach-teamcity commented Oct 22, 2019

cockroach-teamcity commented Oct 23, 2019

cockroach-teamcity commented Oct 25, 2019

cockroach-teamcity commented Oct 29, 2019

cockroach-teamcity commented Oct 30, 2019

cockroach-teamcity commented Nov 3, 2019

cockroach-teamcity commented Nov 6, 2019

cockroach-teamcity commented Nov 6, 2019

cockroach-teamcity commented Nov 7, 2019

cockroach-teamcity commented Nov 7, 2019

cockroach-teamcity commented Nov 8, 2019

cockroach-teamcity commented Nov 8, 2019

cockroach-teamcity commented Nov 8, 2019

cockroach-teamcity commented Nov 12, 2019

cockroach-teamcity commented Nov 12, 2019

cockroach-teamcity commented Nov 13, 2019

cockroach-teamcity commented Nov 14, 2019

cockroach-teamcity commented Nov 20, 2019

cockroach-teamcity commented Nov 20, 2019

cockroach-teamcity commented Nov 21, 2019

cockroach-teamcity commented Nov 21, 2019

cockroach-teamcity commented Nov 21, 2019

andreimatei commented Jan 2, 2020

cockroach-teamcity commented Jan 2, 2020

cockroach-teamcity commented Jan 3, 2020

cockroach-teamcity commented Jan 7, 2020

cockroach-teamcity commented Jan 7, 2020

cockroach-teamcity commented Jan 7, 2020

cockroach-teamcity commented Jan 14, 2020

cockroach-teamcity commented Jan 15, 2020

cockroach-teamcity commented Jan 15, 2020

cockroach-teamcity commented Jan 16, 2020

cockroach-teamcity commented Jan 17, 2020

cockroach-teamcity commented Jan 17, 2020

cockroach-teamcity commented Jan 17, 2020

cockroach-teamcity commented Jan 18, 2020

cockroach-teamcity commented Jan 19, 2020

cockroach-teamcity commented Jan 22, 2020

cockroach-teamcity commented Jan 22, 2020

cockroach-teamcity commented Jan 22, 2020

cockroach-teamcity commented Jan 23, 2020

rohany commented Jan 23, 2020