Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: kv50/rangelookups/relocate/nodes=8 failed #40359

Closed
cockroach-teamcity opened this issue Aug 30, 2019 · 23 comments · Fixed by #41106
Closed

roachtest: kv50/rangelookups/relocate/nodes=8 failed #40359

cockroach-teamcity opened this issue Aug 30, 2019 · 23 comments · Fixed by #41106
Assignees
@nvanbenschoten
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
19.2

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/e8faca611a902766154ed82581d6d3a7483ad231

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1460982&tab=buildLog

The test failed on branch=provisional_201908291837_v19.2.0-beta.20190903, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190829-1460982/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1735,kv.go:587,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1567109858-63-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		.0     13.6     48.2     71.3    285.2 write
		   68.0s        0        22942.4        23035.8      1.9     18.9     32.5    104.9 read
		   68.0s        0        22750.5        23060.5     14.7     46.1     65.0    159.4 write
		   69.0s        0        23245.1        23038.7      1.7     18.9     31.5    151.0 read
		   69.0s        0        23265.1        23063.4     12.1     54.5     83.9    243.3 write
		   70.0s        0        23440.0        23044.6      1.6     17.8     31.5    260.0 read
		   70.0s        0        23775.9        23073.7     12.1     52.4     79.7    369.1 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   71.0s        0        25254.8        23075.6      1.6     16.3     28.3     58.7 read
		   71.0s        0        24865.7        23098.8     10.5     50.3     88.1    142.6 write
		   72.0s        0        25202.6        23105.0      1.5     17.8     28.3     71.3 read
		   72.0s        0        25351.7        23130.0     10.0     52.4     75.5    130.0 write
		: signal: killed
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Aug 30, 2019
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone Aug 30, 2019
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/e8faca611a902766154ed82581d6d3a7483ad231

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1462518&tab=buildLog

The test failed on branch=provisional_201908291837_v19.2.0-beta.20190903, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190830-1462518/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1735,kv.go:587,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1567188851-69-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		.5     12.6     50.3     75.5    436.2 write
		  480.0s        0        21933.7        22521.2      1.8     23.1     39.8    125.8 read
		  480.0s        0        21993.7        22527.4     13.1     54.5     79.7    142.6 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  481.0s        0        21660.0        22519.4      1.8     23.1     39.8    184.5 read
		  481.0s        0        21861.0        22526.0     12.1     58.7     83.9    251.7 write
		  482.0s        0        22068.8        22518.5      1.8     22.0     37.7    117.4 read
		  482.0s        0        21763.6        22524.4     13.1     54.5     79.7    226.5 write
		  483.0s        0        22372.5        22518.2      1.8     18.9     32.5    402.7 read
		  483.0s        0        22339.5        22524.0     13.1     54.5     88.1    369.1 write
		  484.0s        0        22875.5        22518.9      1.7     21.0     35.7     83.9 read
		  484.0s        0        23252.6        22525.5     12.6     52.4     75.5    142.6 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/4784fe3c51545db5fb5d411937ec1db2ef2b9761

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1472753&tab=buildLog

The test failed on branch=provisional_201909060000_v19.2.0-beta.20190910, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190906-1472753/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1735,kv.go:576,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1567786968-56-n9cpu8:9 -- ./workload init kv {pgurl:1-8} --splits=1000 returned:
		stderr:
		
		stdout:
		I190906 20:24:57.780237 1 workload/workloadsql/workloadsql.go:97  starting 1000 splits
		Error: ALTER TABLE kv SPLIT AT VALUES (598920262133426176): dial tcp 10.128.0.138:26257: connect: connection refused
		Error:  exit status 1
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/47bb2a58c87fc1259291ec9dde78de3e54bd8a3d

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1475396&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190909-1475396/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1735,kv.go:587,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1568010398-62-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		.1     12.1     44.0     75.5   5368.7 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  121.0s        0        22545.7        22985.0      2.0     21.0     35.7    151.0 read
		  121.0s        0        22419.6        23030.8     13.6     50.3     75.5    151.0 write
		  122.0s        0        22787.5        22983.5      1.9     19.9     35.7    318.8 read
		  122.0s        0        22796.5        23029.0     13.1     50.3     71.3    260.0 write
		  123.0s        0        23271.1        22985.8      1.8     17.8     31.5    151.0 read
		  123.0s        0        23163.1        23030.0     13.1     52.4     79.7    201.3 write
		  124.0s        0        24023.8        22994.1      1.7     19.9     32.5    100.7 read
		  124.0s        0        23923.8        23037.2     13.1     46.1     67.1    117.4 write
		  125.0s        0        24367.6        23005.4      1.7     18.9     33.6     75.5 read
		  125.0s        0        24471.5        23048.9     13.1     46.1     65.0    117.4 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/bdf41f7d03f0dafeeaf3bc6aac40f502ab069a6a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1478770&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190911-1478770/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1735,kv.go:587,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1568183721-60-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		   1.2     12.1     24.1   1879.0 read
		    2.0s        0        27346.7        27698.1      5.8     21.0     32.5   1946.2 write
		    3.0s        0        26563.8        27251.4      1.2     12.6     23.1   2550.1 read
		    3.0s        0        26478.8        27289.7      6.3     27.3     50.3   2550.1 write
		    4.0s        0        29774.5        27889.5      1.6     12.6     22.0   3087.0 read
		    4.0s        0        29267.0        27791.3     10.5     39.8     56.6   3087.0 write
		    5.0s        0        30276.6        28368.2      1.6     12.6     21.0    201.3 read
		    5.0s        0        30646.5        28363.6     11.0     32.5     46.1    176.2 write
		    6.0s        0        32025.7        28978.6      1.4     12.6     21.0     65.0 read
		    6.0s        0        32229.6        29008.8     10.0     33.6     54.5     92.3 write
		    7.0s        0        32520.9        29480.1      1.4     12.6     19.9     46.1 read
		    7.0s        0        32403.8        29489.3     10.0     33.6     46.1     75.5 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/62b1678f652461bbc1aaf6bc2c0dd03105ce0ebe

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1488785&tab=buildLog

The test failed on branch=40765, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190914-1488785/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1735,kv.go:587,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1568493418-66-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		   1.6     17.8     33.6    130.0 read
		   21.0s        0        24580.2        24276.2     11.5     50.3     92.3    184.5 write
		   22.0s        0        25529.2        24437.7      1.6     15.7     26.2    113.2 read
		   22.0s        0        25209.1        24318.2     10.0     44.0     62.9    113.2 write
		   23.0s        0        23119.9        24379.8      1.5     17.8     27.3    159.4 read
		   23.0s        0        23220.0        24269.8      9.4     44.0     75.5    192.9 write
		   24.0s        0        23399.3        24340.0      1.6     17.8     44.0   3221.2 read
		   24.0s        0        23644.0        24244.7     10.0     39.8     71.3   3355.4 write
		   25.0s        0        25016.7        24366.6      1.7     17.8     30.4    167.8 read
		   25.0s        0        25119.7        24279.4     12.1     46.1     71.3    192.9 write
		   26.0s        0        27811.3        24499.0      1.6     15.7     25.2     71.3 read
		   26.0s        0        27752.3        24412.8     12.1     39.8     52.4     88.1 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/62b1678f652461bbc1aaf6bc2c0dd03105ce0ebe

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1489712&tab=buildLog

The test failed on branch=40765, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190915-1489712/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1735,kv.go:587,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1568575032-60-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1.0s        0        15993.8        25775.2      1.6     15.7     32.5    100.7 read
		    1.0s        0        15712.1        25321.2      8.9     35.7     58.7    167.8 write
		    2.0s        0        24603.5        25173.7      1.2     10.5     18.9   1677.7 read
		    2.0s        0        24631.5        24960.7      6.0     21.0     32.5   1677.7 write
		    3.0s        0        25203.0        25188.1      1.2     12.6     24.1   2684.4 read
		    3.0s        0        25151.0        25028.7      6.6     24.1     37.7   2684.4 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c6342c90a7fa4ceb1b674faa47a95e1726d05e79

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1496387&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190919-1496387/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1735,kv.go:587,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1568869602-65-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1.0s        0        16025.1        27030.8      1.8     16.3     28.3    637.5 read
		    1.0s        0        15774.4        26607.8     12.1     35.7     54.5    637.5 write
		    2.0s        0        28506.9        27783.8      1.6     11.5     21.0     58.7 read
		    2.0s        0        28330.0        27483.8     12.1     37.7     56.6     96.5 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/073999b81ddfed3bbc8409d534912fea12b6d500

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1498740&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190920-1498740/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1735,kv.go:587,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1568956202-63-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		.4     11.5     50.3     79.7    226.5 write
		  347.0s        0        21049.5        22285.5      1.4     24.1     39.8    167.8 read
		  347.0s        0        21068.5        22273.9      8.9     52.4     88.1    117.4 write
		  348.0s        0        20619.6        22280.7      1.6     22.0     37.7     83.9 read
		  348.0s        0        20638.6        22269.2      8.9     46.1     83.9    142.6 write
		  349.0s        0        19916.1        22273.9      1.4     19.9     32.5     75.5 read
		  349.0s        0        20379.0        22263.8      7.9     39.8     62.9    192.9 write
		  350.0s        0        22029.6        22273.2      1.6     24.1     41.9   4295.0 read
		  350.0s        0        22215.5        22263.7     10.5     46.1     75.5   4160.7 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  351.0s        0        21717.5        22271.6      1.5     18.9     33.6     88.1 read
		  351.0s        0        21730.5        22262.1     10.0     44.0     65.0    134.2 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/a92c7d01d3076eabafbd536d8a344511ec9081c6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1500206&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190921-1500206/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1735,kv.go:587,cluster.go:2091,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1569042580-57-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		.7      6.6     26.2     44.0    176.2 write
		    8.0s        0        20593.1        20932.6      1.1     13.1     25.2   3758.1 read
		    8.0s        0        20448.2        20878.1      5.0     22.0     39.8   4563.4 write
		    9.0s        0        23510.8        21218.7      1.4     14.2     26.2   2952.8 read
		    9.0s        0        23652.8        21186.1      7.1     29.4     52.4   3758.1 write
		   10.0s        0        21193.9        21215.5      1.2     14.2     24.1   1946.2 read
		   10.0s        0        21391.9        21205.9      6.3     26.2     39.8   2952.8 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   11.0s        0        25162.1        21577.1      1.4     14.7     28.3   4160.7 read
		   11.0s        0        24899.4        21544.5      7.9     32.5     54.5   4295.0 write
		   12.0s        0        26493.4        21985.5      1.3     13.6     22.0     48.2 read
		   12.0s        0        26501.4        21956.3      8.1     31.5     50.3    100.7 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/9422434988be7776e733627d808192bae8800fb5

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1503079&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190924-1503079/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1736,kv.go:587,cluster.go:2092,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1569301790-62-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		.0      9.4     35.7     54.5    167.8 write
		   90.0s        0        21812.5        24105.9      1.4     18.9     35.7    159.4 read
		   90.0s        0        21734.5        24094.5      7.1     32.5     54.5   4160.7 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   91.0s        0        21955.4        24082.1      1.6     17.8     31.5    939.5 read
		   91.0s        0        21675.2        24067.8      8.1     35.7     58.7   4160.7 write
		   92.0s        0        18098.0        24017.0      1.2     21.0     39.8    113.2 read
		   92.0s        0        18192.1        24003.8      4.7     28.3     48.2    142.6 write
		   93.0s        0        19801.4        23971.9      1.3     18.9     46.1   5368.7 read
		   93.0s        0        19776.4        23958.6      6.3     31.5     62.9   5368.7 write
		   94.0s        0        22120.1        23952.3      1.2     12.6     25.2   3087.0 read
		   94.0s        0        22384.9        23942.0      6.0     26.2     48.2   4160.7 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c11a254ed74c59ac3ff083356d9a71b82db2c7d7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1505083&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190925-1505083/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1762,kv.go:587,cluster.go:2118,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1569387937-58-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		   1.7     16.8     31.5    104.9 read
		   83.0s        0        23995.7        24307.2     10.5     39.8     62.9    268.4 write
		   84.0s        0        22667.4        24309.9      1.5     16.8     31.5    142.6 read
		   84.0s        0        22984.5        24291.3      8.1     31.5     50.3    121.6 write
		   85.0s        0        23746.2        24303.3      1.6     15.7     30.4   3087.0 read
		   85.0s        0        23809.2        24285.7     10.0     39.8     65.0   3087.0 write
		   86.0s        0        23188.4        24290.4      1.6     18.9     33.6    100.7 read
		   86.0s        0        22974.4        24270.5      8.1     37.7     56.6    125.8 write
		   87.0s        0        23691.4        24283.6      1.5     15.7     30.4   4295.0 read
		   87.0s        0        24026.2        24267.8      8.9     35.7     65.0   4160.7 write
		   88.0s        0        27901.2        24325.0      1.6     15.2     28.3     67.1 read
		   88.0s        0        27651.3        24306.5      8.9     48.2     79.7    142.6 write
		: signal: killed

@nvanbenschoten nvanbenschoten self-assigned this Sep 25, 2019
@nvanbenschoten nvanbenschoten mentioned this issue Sep 26, 2019
Merged
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/77f26d185efb436aaac88243de19a27caa5da9b6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1509340&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190926-1509340/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1569512843-57-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		.7     12.1     48.2     71.3    192.9 write
		   89.0s        0        24061.9        23441.9      1.8     19.9     32.5    251.7 read
		   89.0s        0        24343.8        23457.9     12.6     46.1     65.0    167.8 write
		   90.0s        0        24375.2        23452.3      1.8     21.0     33.6    167.8 read
		   90.0s        0        24251.2        23466.7     11.5     48.2     71.3    151.0 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   91.0s        0        22968.0        23447.0      1.8     22.0     39.8    142.6 read
		   91.0s        0        22945.0        23461.0     11.5     52.4     75.5    226.5 write
		   92.0s        0        23419.7        23446.6      1.8     21.0     35.7    100.7 read
		   92.0s        0        22897.5        23454.8     12.6     52.4     88.1    142.6 write
		   93.0s        0        25147.1        23464.9      1.6     17.8     30.4     71.3 read
		   93.0s        0        25567.1        23477.5     11.0     48.2     75.5    113.2 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/fb0bd2f43fb87b26752615bbdbc759502c2c8b0b

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1509988&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190926-1509988/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1569536809-62-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		.6     11.0     50.3     75.5    167.8 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   21.0s        0        24681.2        24327.9      1.7     18.9     31.5    402.7 read
		   21.0s        0        24907.3        24345.2     12.1     46.1     65.0    251.7 write
		   22.0s        0        24761.5        24347.4      1.7     16.8     29.4    209.7 read
		   22.0s        0        24677.5        24360.1     12.1     46.1     65.0    159.4 write
		   23.0s        0        25994.7        24419.2      1.7     16.3     27.3     67.1 read
		   23.0s        0        25832.8        24424.3     11.5     41.9     65.0    100.7 write
		   24.0s        0        27076.4        24530.2      1.5     15.2     25.2     62.9 read
		   24.0s        0        27092.4        24535.7      9.4     35.7     52.4     92.3 write
		   25.0s        0        26549.2        24610.1      1.4     14.2     23.1     46.1 read
		   25.0s        0        26145.9        24599.3      8.1     32.5     48.2     79.7 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/cf5c2bd2372e633d2f63e08e5bffca7c2a7ec59f

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1510142&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190927-1510142/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1569560965-57-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		.0     13.6     48.2     71.3    159.4 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  221.0s        0        21434.6        22254.7      1.9     17.8     32.5    209.7 read
		  221.0s        0        21294.6        22231.7     12.1     48.2     75.5    159.4 write
		  222.0s        0        19485.1        22242.2      1.8     26.2     46.1   3221.2 read
		  222.0s        0        19753.9        22220.6     10.0     48.2     75.5   3355.4 write
		  223.0s        0        21705.8        22239.7      1.8     24.1     39.8     83.9 read
		  223.0s        0        21730.8        22218.4     12.6     56.6     79.7    151.0 write
		  224.0s        0        22321.6        22240.1      1.8     22.0     35.7     71.3 read
		  224.0s        0        22382.5        22219.1     12.6     54.5     83.9    142.6 write
		  225.0s        0        21890.3        22238.6      1.8     21.0     35.7    100.7 read
		  225.0s        0        22099.3        22218.6     13.6     54.5     79.7    130.0 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/250f4c36de2b88eff443cf9be9cd5d2759312c88

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1514702&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190930-1514702/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1569865186-58-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		   2.0     21.0     37.7    117.4 read
		   92.0s        0        22160.2        21708.5     13.6     50.3     71.3    142.6 write
		   93.0s        0        22181.3        21717.8      1.8     19.9     35.7    159.4 read
		   93.0s        0        22625.2        21718.5     13.6     50.3     75.5    243.3 write
		   94.0s        0        21633.2        21717.0      1.9     22.0     37.7    159.4 read
		   94.0s        0        21649.2        21717.7     14.2     52.4     75.5    218.1 write
		   95.0s        0        21734.2        21717.0      2.0     19.9     33.6    201.3 read
		   95.0s        0        21775.2        21718.3     14.7     50.3     71.3    125.8 write
		   96.0s        0        21906.8        21719.1      1.9     19.9     33.6     65.0 read
		   96.0s        0        22368.5        21725.2     13.6     52.4     75.5    134.2 write
		   97.0s        0        23669.9        21739.4      1.7     16.3     28.3     60.8 read
		   97.0s        0        23961.7        21748.4     12.1     54.5     79.7    121.6 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/09d51e9f6265ed70caf49385be905606ebf722c7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1515124&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191001-1515124/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1569906547-58-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1.0s        0        14121.2        23658.0      1.7     14.7     30.4     88.1 read
		    1.0s        0        14182.6        23761.0     10.0     37.7     56.6    121.6 write
		    2.0s        0        20884.0        22271.2      1.2     15.7     30.4   2013.3 read
		    2.0s        0        20726.0        22243.7      6.3     26.2     44.0   2013.3 write
		: signal: killed

@knz knz mentioned this issue Oct 1, 2019
Closed
18 tasks
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/4dcfd7d8899d90dd1816153bf59b5df647c9bbd3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1516560&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191002-1516560/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1569993086-60-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		   1.6     17.8     31.5    176.2 read
		   13.0s        0        24761.7        24997.3     12.1     48.2     75.5    260.0 write
		   14.0s        0        25003.8        25070.2      1.6     16.8     27.3    130.0 read
		   14.0s        0        25482.7        25032.6     12.1     46.1     62.9    121.6 write
		   15.0s        0        24378.3        25022.9      1.6     17.8     27.3     83.9 read
		   15.0s        0        24628.4        25004.5     12.6     44.0     60.8    176.2 write
		   16.0s        0        26154.1        25094.2      1.6     16.3     27.3    125.8 read
		   16.0s        0        25891.2        25060.5     12.1     41.9     60.8    184.5 write
		   17.0s        0        28797.0        25311.4      1.6     13.6     23.1     71.3 read
		   17.0s        0        28363.8        25254.2     11.5     37.7     58.7     96.5 write
		   18.0s        0        30044.9        25574.7      1.6     12.6     21.0    209.7 read
		   18.0s        0        30153.8        25526.7     10.5     37.7     52.4     88.1 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/1ed03c9811f01fef31950a5cb75a7b591af6fc26

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1518416&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191003-1518416/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570079462-57-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		   1.7     15.7     26.2    167.8 read
		    4.0s        0        26535.3        25432.2     12.1     41.9     60.8    285.2 write
		    5.0s        0        25885.1        25556.9      1.6     15.7     27.3    130.0 read
		    5.0s        0        25900.1        25525.3     11.5     44.0     62.9    130.0 write
		    6.0s        0        25310.9        25516.4      1.6     17.8     30.4    151.0 read
		    6.0s        0        25352.9        25497.1     11.0     48.2     75.5    167.8 write
		    7.0s        0        26188.6        25612.6      1.6     15.2     27.3    176.2 read
		    7.0s        0        26345.6        25618.4     11.0     48.2     71.3    251.7 write
		    8.0s        0        27995.2        25909.6      1.5     16.3     26.2     67.1 read
		    8.0s        0        28006.2        25916.1     11.0     37.7     52.4     92.3 write
		    9.0s        0        27699.9        26108.7      1.5     15.7     25.2     54.5 read
		    9.0s        0        27438.0        26085.4     11.0     41.9     56.6     92.3 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/1c99165c39c3714f1ce9986bff75ce517f977630

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1523814&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=provisional_201910051220_v19.2.0-beta.20191014, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191006-1523814/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570324325-60-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		   1.8     19.9     32.5    151.0 read
		   15.0s        0        23894.6        25166.3     12.6     46.1     67.1    159.4 write
		   16.0s        0        23568.8        25099.4      1.7     19.9     35.7    192.9 read
		   16.0s        0        23768.9        25078.5     12.6     48.2     71.3    117.4 write
		   17.0s        0        24771.1        25080.1      1.8     15.7     27.3    142.6 read
		   17.0s        0        24995.1        25073.6     13.6     39.8     56.6    176.2 write
		   18.0s        0        25761.9        25118.4      1.8     15.2     27.3    151.0 read
		   18.0s        0        25785.9        25113.6     12.6     41.9     60.8    218.1 write
		   19.0s        0        27363.7        25236.1      1.6     14.2     24.1     71.3 read
		   19.0s        0        27473.8        25237.3     11.5     39.8     58.7     96.5 write
		   20.0s        0        28316.2        25390.8      1.6     14.7     25.2     88.1 read
		   20.0s        0        27726.5        25362.4     10.5     41.9     60.8    130.0 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/8d3bdd75d0080cdb089b65d55f4e5fcf4a3ea905

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1524120&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191007-1524120/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570424814-64-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		   1.6     17.8     33.6   4831.8 read
		   14.0s        0        21938.1        23966.6      9.4     44.0     75.5   5100.3 write
		   15.0s        0        23325.4        23978.2      1.8     18.9     33.6    121.6 read
		   15.0s        0        23194.3        23914.5     13.6     48.2     75.5    159.4 write
		   16.0s        0        23755.2        23964.2      1.8     15.2     27.3    243.3 read
		   16.0s        0        23900.6        23913.7     13.6     48.2     67.1    176.2 write
		   17.0s        0        23587.9        23942.2      1.9     15.7     27.3    100.7 read
		   17.0s        0        23570.9        23893.6     14.7     44.0     60.8    167.8 write
		   18.0s        0        27796.1        24156.4      1.6     13.1     22.0     60.8 read
		   18.0s        0        27626.2        24101.0     12.1     39.8     58.7    192.9 write
		   19.0s        0        27628.5        24339.8      1.6     14.7     24.1     48.2 read
		   19.0s        0        27425.6        24276.7     12.1     39.8     54.5     92.3 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c0cab6082c2383e77f0a806c10a1506b6aa7b6bf

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1525981&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191008-1525981/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570511373-57-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		    1.0s        0        13276.2        24978.9      1.8     16.3     31.5    192.9 read
		    1.0s        0        13194.4        24824.9     11.5     39.8     65.0    234.9 write
		    2.0s        0        25524.7        25247.8      1.4     13.6     24.1     83.9 read
		    2.0s        0        25294.6        25055.8      9.4     32.5     48.2    109.1 write
		    3.0s        0        26711.0        25739.3      1.4     13.1     22.0   2147.5 read
		    3.0s        0        26883.9        25668.9      9.4     29.4     41.9   2281.7 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/27ea46b456d6b3dabac2c0f6c5c120ed316fffce

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1526784&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=provisional_201910072154_v19.2.0-beta.20191014, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191008-1526784/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570542880-59-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		   1.6     17.8     30.4    109.1 read
		   22.0s        0        24675.4        24836.2     11.0     52.4     83.9    142.6 write
		   23.0s        0        24994.1        24833.2      1.6     16.8     28.3    234.9 read
		   23.0s        0        24848.1        24836.7     10.5     54.5     96.5    226.5 write
		   24.0s        0        24505.6        24819.2      1.6     17.8     29.4    234.9 read
		   24.0s        0        24518.6        24823.2     10.5     54.5    100.7    285.2 write
		   25.0s        0        23666.3        24773.6      1.4     18.9     30.4    109.1 read
		   25.0s        0        23976.1        24789.8      9.4     60.8    109.1    192.9 write
		   26.0s        0        25499.2        24801.1      1.5     18.9     30.4     79.7 read
		   26.0s        0        25485.2        24816.1     10.5     50.3     88.1    151.0 write
		   27.0s        0        26822.6        24876.3      1.5     17.8     28.3     79.7 read
		   27.0s        0        27270.4        24907.3     10.0     44.0     75.5    134.2 write
		: signal: killed

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/06026bab5cdb3695ddeba209c94175cd9460b170

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=kv50/rangelookups/relocate/nodes=8 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1528278&tab=artifacts#/kv50/rangelookups/relocate/nodes=8

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191009-1528278/kv50/rangelookups/relocate/nodes=8/run_1
	cluster.go:1764,kv.go:587,cluster.go:2120,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570597867-61-n9cpu8:9 -- ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned:
		stderr:
		
		stdout:
		.2     12.1     58.7    104.9    167.8 write
		   88.0s        0        20010.7        21581.8      1.8     18.9     35.7    176.2 read
		   88.0s        0        20165.7        21571.0     11.5     48.2     75.5    234.9 write
		   89.0s        0        19032.6        21553.1      1.6     19.9     39.8     79.7 read
		   89.0s        0        18826.5        21540.1     10.0     39.8     67.1    142.6 write
		   90.0s        0        20640.1        21543.0      1.7     18.9     37.7   2818.6 read
		   90.0s        0        20368.3        21527.2     10.5     46.1     79.7   4563.4 write
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   91.0s        0        22794.7        21556.9      1.6     18.9     33.6   3892.3 read
		   91.0s        0        22725.8        21540.5     11.5     50.3     79.7   3892.3 write
		   92.0s        0        24390.7        21587.9      1.6     18.9     32.5     71.3 read
		   92.0s        0        24180.9        21569.3     11.5     54.5     79.7    142.6 write
		: signal: killed

bdarnell added a commit to bdarnell/cockroach that referenced this issue Oct 9, 2019
@bdarnell
roachtest: Fix error reporting in kv50 test
909b71a 
When one thread in a monitor fatals and another returns an error, the
fatal is the one that gets reported, even if it's an unhelpful
"signal: killed" from a cancelled process that was caused by a
previously returned error. Whenever any thread returns errors, they
all should, by using RunE instead of Run, etc. This is probably a much
more widespread problem in the roachtests, but for now I'm only fixing
this one which is causing opaque failures in cockroachdb#40359

Release note: None
@bdarnell bdarnell mentioned this issue Oct 9, 2019
Closed
craig bot pushed a commit that referenced this issue Oct 10, 2019
@nvanbenschoten
Merge #41106
1756e41 
41106: roachtest: include learner snapshot error in isExpectedRelocateError whitelist r=nvanbenschoten a=nvanbenschoten

Fixes #40359.

We occasionally see the error "failed to apply snapshot: raft group deleted" during a RELOCATE RANGE statement. This occurs because the newly added learner attempts to send a raft message to another node after it has been removed and then destroys itself in response to a ReplicaTooOldError. We have a similar case in TestLearnerAdminChangeReplicasRace.

The PR also makes a few changes to improve the error reporting both in this test and in general when a command fails due to a context cancellation.

Release justification: Testing only.

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
@craig craig bot closed this as completed in 14784b6 Oct 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
19.2
2 participants
@nvanbenschoten @cockroach-teamcity