Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: kv/contention/nodes=4 failed #45700

Closed
cockroach-teamcity opened this issue Mar 4, 2020 · 19 comments
Closed

roachtest: kv/contention/nodes=4 failed #45700

cockroach-teamcity opened this issue Mar 4, 2020 · 19 comments
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).kv/contention/nodes=4 failed on master@d6a52a8f6f07b3b6b32af2243075e5365fb21c45:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200304-1783198/kv/contention/nodes=4/run_1
	ts_util.go:116,kv.go:259,cluster.go:2344,errgroup.go:57: average txns per second 1.912040 was under target 100.000000

	cluster.go:2368,kv.go:262,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 4, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone Mar 4, 2020
@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@c9b189b3a2171c8da78c75fe80dd02933e18a5e5:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200305-1786843/kv/contention/nodes=4/run_1
	ts_util.go:116,kv.go:259,cluster.go:2344,errgroup.go:57: average txns per second 1.911901 was under target 100.000000

	cluster.go:2368,kv.go:262,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@954fe69d554162aec0fbc001aad1fe5103d8df13:

		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		        1.0s        0          683.9          873.7     71.3    385.9    570.4    939.5 write
		        2.0s        0          700.3          786.9     44.0    704.6   1208.0   1610.6 write
		        3.0s        0          683.9          752.6     29.4    939.5   1744.8   2550.1 write
		        4.0s        0          722.1          745.0     23.1    738.2   1946.2   3623.9 write
		        5.0s        0          751.9          746.4     25.2    771.8   1610.6   4295.0 write
		        6.0s        0          726.1          743.0     22.0    536.9   2281.7   3489.7 write
		        7.0s        0          618.0          725.1     21.0    637.5   2013.3   6174.0 write
		        8.0s        0          547.9          703.0     17.8    604.0   2952.8   5368.7 write
		        9.0s        0          440.2          673.8     13.1    906.0   3758.1   7247.8 write
		       10.0s        0          295.0          635.9     10.0   1040.2   6710.9   9663.7 write
		       11.0s        0          600.9          632.7     13.6   1040.2   4831.8   9126.8 write
		       12.0s        0          574.0          627.8     12.1    704.6   4563.4   7247.8 write
		       13.0s        0          709.0          634.1     18.9    436.2   4563.4   9126.8 write
		       14.0s        0          422.0          618.9     19.9   1208.0   5637.1   8589.9 write
		       15.0s        0          188.0          590.2      9.4   3758.1   7784.6  12348.0 write
		       16.0s        0          165.0          563.6      8.9   2818.6   8321.5   9663.7 write
		       17.0s        0           24.0          531.9      5.2   4160.7   8589.9   8589.9 write
		       18.0s        0          488.9          529.5     11.0   3221.2   4831.8  15569.3 write
		       19.0s        0          706.0          538.8     13.1   1208.0   3623.9  12884.9 write:
		  - exit status 1

	cluster.go:2368,kv.go:262,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@752dea867f3aeb142e98c22f8d320ce19041aa8d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200307-1793930/kv/contention/nodes=4/run_1
	ts_util.go:116,kv.go:259,cluster.go:2344,errgroup.go:57: average txns per second 1.998103 was under target 100.000000

	cluster.go:2368,kv.go:262,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@dfa5bd527ae7d7373dd03c62118df87a87a77130:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200308-1795062/kv/contention/nodes=4/run_1
	ts_util.go:116,kv.go:259,cluster.go:2344,errgroup.go:57: average txns per second 1.975523 was under target 100.000000

	cluster.go:2368,kv.go:262,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@c473f40078994551cebcbe00fdbf1fa388957658:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200309-1796240/kv/contention/nodes=4/run_1
	ts_util.go:116,kv.go:259,cluster.go:2344,errgroup.go:57: average txns per second 1.911947 was under target 100.000000

	cluster.go:2368,kv.go:262,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@72c4a1bd411f2f82bf9aaa22883821a946614148:

		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1799071-1583824469-50-n5cpu4:5 -- ./workload run kv --init --secondary-index --duration=1h0m0s --cycle-length=512 --concurrency=128 --batch=4 --splits=4 {pgurl:1-4} returned:
		    stderr:
		    I200310 11:26:02.780069 1 workload/cli/run.go:326  DEPRECATION: the --init flag on "workload run" will no longer be supported after 19.2
		    I200310 11:26:02.810772 1 workload/workloadsql/workloadsql.go:113  starting 4 splits
		    Error: ERROR: TransactionStatusError: already committed (REASON_TXN_COMMITTED) (SQLSTATE XXUUU)
		    Error:  exit status 1
		    
		    stdout:
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		        1.0s        0          558.6          693.8     88.1    486.5    637.5    939.5 write
		        2.0s        0          609.0          651.4     54.5    738.2   1208.0   1342.2 write
		        3.0s        0          551.9          618.3     48.2    973.1   1677.7   2415.9 write
		        4.0s        0          507.1          590.5     44.0   1342.2   2415.9   3489.7 write
		        5.0s        0          542.0          580.8     30.4   1275.1   2818.6   4563.4 write
		        6.0s        0          583.0          581.2     31.5   1208.0   3221.2   4563.4 write
		        7.0s        0          522.0          572.7     31.5   1275.1   3355.4   4831.8 write
		        8.0s        0          472.0          560.1     19.9   1140.9   3221.2   5637.1 write:
		  - exit status 1

	cluster.go:2368,kv.go:262,test_runner.go:747: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@5570c01402796edb7cd06eb8ce7f615371f22d42:

		       96.0s        0          573.9          535.5     10.0   1543.5   3087.0  34359.7 write
		       97.0s        0          710.1          537.3     12.1    906.0   1879.0   7247.8 write
		       98.0s        0          645.1          538.4     11.0   1073.7   2281.7   5905.6 write
		       99.0s        0          636.9          539.4     10.0    570.4   1677.7   5100.3 write
		      100.0s        0          278.1          536.8      9.4   1275.1   2281.7   4563.4 write
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      101.0s        0          791.5          539.3     11.0   1744.8   3355.4   9126.8 write
		      102.0s        0          945.2          543.3     13.6    771.8   2818.6  11811.2 write
		      103.0s        0          952.0          547.3     13.6    704.6   1140.9  13421.8 write
		      104.0s        0          914.0          550.8     14.2    771.8   1208.0  13421.8 write
		      105.0s        0          920.0          554.3     15.7    671.1   1409.3  11274.3 write
		      106.0s        0          637.0          555.1     11.0    486.5   1140.9   9126.8 write
		      107.0s        0           68.0          550.5      6.8   2684.4   4295.0  10737.4 write
		      108.0s        0          669.6          551.6     11.5   2080.4   4563.4   6979.3 write
		      109.0s        0          482.2          551.0     10.5    486.5   2952.8  11811.2 write
		      110.0s        0          176.0          547.6      7.6   1677.7   5637.1   7516.2 write
		      111.0s        0          607.7          548.1     10.0   1677.7   2818.6   7516.2 write
		      112.0s        0          531.2          548.0      8.9   1140.9   2818.6   8321.5 write
		      113.0s        0          719.7          549.5     12.1   1342.2   4831.8  10737.4 write
		      114.0s        0          937.1          552.9     16.3    704.6   2415.9  11811.2 write:
		  - exit status 1

	cluster.go:2368,kv.go:262,test_runner.go:747: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@793a9200c16693aff32aa6a4dd9d8bbcbddb30aa:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200312-1804460/kv/contention/nodes=4/run_1
	ts_util.go:116,kv.go:259,cluster.go:2344,errgroup.go:57: average txns per second 1.912853 was under target 100.000000

	cluster.go:2368,kv.go:262,test_runner.go:747: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@69dc87d68addedf2fabfb2b14c098cfb35b5f3d0:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200313-1806460/kv/contention/nodes=4/run_1
	ts_util.go:116,kv.go:259,cluster.go:2344,errgroup.go:57: average txns per second 2.204701 was under target 100.000000

	cluster.go:2368,kv.go:262,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Mar 14, 2020
Relates to cockroachdb#45700.

This commit fixes a bug in the lockTableWaiter where we manually
cancelled the context (to clean up resources) used by an async push
_before_ checking whether the context was cancelled. This meant that we
alway hit the `pushCtx.Err() == context.Canceled` path and always
cleared the push error. This lead to deadlocks because a transaction
would detect that it was aborted but the error would be swallowed due to
this bug so the transaction would continue to wait instead of exiting
its lock wait-queues.

Release justification: fixes a high-severity bug where user transactions
could deadlock indefinitely.
@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@f5585e933a097b53242d8d5800127b821a9a4d41:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200314-1807675/kv/contention/nodes=4/run_1
	ts_util.go:116,kv.go:259,cluster.go:2344,errgroup.go:57: average txns per second 2.130992 was under target 100.000000

	cluster.go:2368,kv.go:262,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@33d71472dc01cbc5064b3c5e1fcd666a33f606de:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200315-1808695/kv/contention/nodes=4/run_1
	ts_util.go:116,kv.go:259,cluster.go:2344,errgroup.go:57: average txns per second 7.253887 was under target 100.000000

	cluster.go:2368,kv.go:262,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@d8bdc938177ea4b5226b95ee6cc51af0951054fc:

		      393.0s        0          547.0          646.2     10.0    738.2   1677.7  10200.5 write
		      394.0s        0          362.0          645.5      6.8   1476.4   3221.2  13958.6 write
		      395.0s        0          316.0          644.6      7.6   1409.3   3355.4   5637.1 write
		      396.0s        0          792.7          645.0     11.5   1543.5   2415.9   6710.9 write
		      397.0s        0          847.1          645.5     12.6    738.2   3221.2  16106.1 write
		      398.0s        0          374.0          644.8     10.5    503.3   1140.9   1543.5 write
		      399.0s        0          362.0          644.1      6.6   1879.0   3087.0  14495.5 write
		      400.0s        0          788.9          644.5     10.0    906.0   2415.9   6442.5 write
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      401.0s        0          527.0          644.2     16.8   1342.2   5100.3  12884.9 write
		      402.0s        0          928.8          644.9     16.8    436.2   1946.2   7784.6 write
		      403.0s        0          603.2          644.8     11.0    805.3   1208.0  13958.6 write
		      404.0s        0          473.9          644.4      6.6    973.1   1543.5   2281.7 write
		      405.0s        0          565.1          644.2      9.4   1476.4   4295.0  49392.1 write
		      406.0s        0          903.9          644.8     14.7    838.9   2013.3  13421.8 write
		      407.0s        0          586.1          644.7      7.9    805.3   1811.9  12348.0 write
		      408.0s        0          537.0          644.4      7.6   1275.1   1811.9   7516.2 write
		      409.0s        0          637.0          644.4     12.6   1208.0   2550.1  14495.5 write
		      410.0s        0          643.9          644.4     12.1   1140.9   1879.0   9126.8 write
		      411.0s        0          654.0          644.4     12.1   1006.6   2281.7   9126.8 write:
		  - exit status 1

	cluster.go:2368,kv.go:262,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

craig bot pushed a commit that referenced this issue Mar 16, 2020
46015: sql: add ability to alter primary key of a new table r=jordanlewis a=rohany

Fixes #46008.
Fixes #46007.

Release justification: low risk change that adds a useful feature

Release note (sql change): This PR makes it possible to create
a table and then add/alter primary key within the same transaction.

46058: sql: add telemetry for ORDER BY and INTERLEAVE r=yuzefovich a=yuzefovich

Release justification: reporting changes only.

This commit adds the reporting of telemetry on the usage of INTERLEAVE
(in `createTableNode.startExec`) and of ORDER BY (by marking
`opt.SortOp` with "Telemetry" flag in the optbuilder and reusing the
same "infrastructure" as relational operators do, right before
constructing the sort). Also it adds a logic test for the telemetry
collection on the window functions' usage.

Addresses: #45210.

Release note: None

46059: sql: add the 'every' aggregate function r=rohany a=rohany

Fixes #45842.

Release justification: low risk update to functionality
Release note (sql change): This PR adds the `every` aggregate function.

46099: pgwire: add telemetry for portal related requests r=rohany a=otan

Resolves #43763 by covering the happy case.

This PR adds telemetry everytime a "limitedCommandResult" is made, which
signifies jdbc streaming.

Release justification: simply telemetry.

Release note: None

46101: kvserver/concurrency: check context cancellation before cancelling r=nvanbenschoten a=nvanbenschoten

Relates to #45700.

This commit fixes a bug in the lockTableWaiter where we manually
cancelled the context (to clean up resources) used by an async push
_before_ checking whether the context was cancelled. This meant that we
alway hit the `pushCtx.Err() == context.Canceled` path and always
cleared the push error. This lead to deadlocks because a transaction
would detect that it was aborted but the error would be swallowed due to
this bug so the transaction would continue to wait instead of exiting
its lock wait-queues.

Release justification: fixes a high-severity bug where user transactions
could deadlock indefinitely.

46105: roachtest: adjust tpchvec r=yuzefovich a=yuzefovich

Release justification: non-production code changes.

This commit adjusts tpchvec test in the following ways:
- it disables the verbose logging of `bytes_usage` (this was enabled to
get something to dig into in case a fatal OOM error that occurred once
would occur again, but it didn't)
- it introduces two "run configurations" into the test:
1. keeps the previous behavior with a minor tweak (disables the
randomization of `workmem` setting). This configuration is meant to be
used to check the correctness of the vectorized engine and compare the
queries' runtimes against row-by-row engine.
2. adds a new configuration in which only the vectorized engine runs all
queries once, but the workmem is randomized in [100KiB, 1000KiB) range.
Also, we enable logging on `vectorized_flow` file (to see that the
spilling does occur). This configuration is meant to stress disk spilling
of the vectorized engine. There is no comparison of the runtimes (since
the row-by-row engine is not used in this config).

Fixes: #46131.

Release note: None

46124: sql,cli: improve the handling of notices r=otan a=knz

Fixes #45833.

tldr:
- notices printed at end in SQL shell (requested by @RaduBerinde),
- new built-in function `crdb_internal.notice()`,
- notices can now be logged with vmodule `notices=2`,
- code simplification.

Details:

The main purpose of this patch was to delay the printing
of notices in the crdb SQL shell to the end, after the results.

The motivation for that is that if there are many (e.g. thousands) of
result rows, a notice printed at the beginning or "in the middle"
would get drowned and missed by the human observer.

This patch achieves that with a straightforward, local change to the
CLI SQL code.

To unit test the change, a way to introduce notices at an arbitrary
point of query execution was also needed. To achieve this, the patch
introduces a new SQL built-in function `crdb_internal.notice()` with
the corresponding plumbing throughout the execution layers. This makes
it possible to send an out-of-band notice in the middle of SQL
execution. The unit test for the first changes already uses this new
built-in; I also foresee that we will discover that our users will
start using that in fancy ways that we are not envisioning yet.

To troubleshoot the implementation of the above, a way to inspect
the notices inside the SQL execution layer was needed; so this
patch also ensures notices are logged when the vmodule config
includes `notices=2`.

Finally, the various changes discussed here revealed that
the notice code in the SQL execution package could be simplified, so
this patch does this too.

Release justification: Category 4: Low risk, high reward changes to existing functionality

Release note (cli change): The `sql` and `demo` client commands now
display out-of-band server notices at the end of execution.

46144: sql: clean up the preliminary notices r=otan a=knz

First commit from #46124.

In the lead up to 20.1, we want the first impression of client notices
by users to be clean and positive. Random notice messages with
inconsistent casing, non-standard terminology and unscoped identifiers
would likely not delight developers.

To achieve this, this commit streamlines the messages for DROP INDEX
and PK changes.


Release justification: Category 4: Low risk, high reward changes to existing functionality

Release note: None

Co-authored-by: Rohan Yadav <rohany@alumni.cmu.edu>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Oliver Tan <otan@cockroachlabs.com>
Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net>
@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@5a3d0c9539a671f0e55b680d3021b18dde9d190d:

		     1184.0s        0          659.0          482.4     15.7    570.4   5905.6  18253.6 write
		     1185.0s        0          791.0          482.7     22.0    637.5   1744.8  12348.0 write
		     1186.0s        0          718.9          482.9     23.1    604.0   1610.6  11811.2 write
		     1187.0s        0          291.1          482.7     15.7    973.1   4563.4  19327.4 write
		     1188.0s        0          380.0          482.7     11.0   1208.0   1879.0  11811.2 write
		     1189.0s        0          152.0          482.4     11.0   2415.9   6442.5   7784.6 write
		     1190.0s        0           12.0          482.0      6.8   8589.9  81604.4  81604.4 write
		     1191.0s        0          184.0          481.7      9.4   5100.3   6710.9   9663.7 write
		     1192.0s        0          556.7          481.8     19.9   3892.3   6174.0  11811.2 write
		     1193.0s        0          765.2          482.0     26.2    973.1   4563.4  13958.6 write
		     1194.0s        0          706.0          482.2     25.2    838.9   1879.0   9663.7 write
		     1195.0s        0          558.0          482.3     14.7   1040.2   1476.4   7784.6 write
		     1196.0s        0          426.0          482.2     15.2    838.9   2080.4  12348.0 write
		     1197.0s        0          420.1          482.2     12.1   1476.4   2550.1  10737.4 write
		     1198.0s        0          273.0          482.0     12.6   2147.5   4295.0  11811.2 write
		     1199.0s        0          149.0          481.7     10.5   1946.2  12348.0  19327.4 write
		     1200.0s        0          556.7          481.8     14.2   2281.7   3355.4   8589.9 write
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     1201.0s        0          450.0          481.8     23.1   1275.1   7247.8  20401.1 write
		     1202.0s        0          405.1          481.7     25.2   1543.5   4831.8  10200.5 write:
		  - exit status 1

	cluster.go:2368,kv.go:262,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@nvanbenschoten nvanbenschoten self-assigned this Mar 17, 2020
@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@b5f030223fbcf22e806c48a3c46e74a73a54f50f:

		     1493.0s        0          207.1          482.5     12.1   3355.4   7516.2  12348.0 write
		     1494.0s        0          275.0          482.4     11.5   2415.9  10200.5  11811.2 write
		     1495.0s        0          441.9          482.3     12.6   1275.1   3489.7  13421.8 write
		     1496.0s        0          152.1          482.1     13.6   2550.1  11811.2  15032.4 write
		     1497.0s        0          427.8          482.1     18.9   2281.7  13958.6  15569.3 write
		     1498.0s        0          624.2          482.2     16.3    872.4   2952.8  15569.3 write
		     1499.0s        0          110.0          481.9      8.9   1342.2  12348.0  14495.5 write
		     1500.0s        0          173.0          481.7      9.4   3758.1  13958.6  16643.0 write
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     1501.0s        0          447.9          481.7     15.2   2550.1  13958.6  18253.6 write
		     1502.0s        0          490.0          481.7     27.3   1342.2   4563.4  19327.4 write
		     1503.0s        0          801.8          481.9     25.2    838.9   2415.9  18253.6 write
		     1504.0s        0          658.1          482.0     19.9    637.5   1275.1  13958.6 write
		     1505.0s        0          464.0          482.0     12.6   1140.9   1744.8  21474.8 write
		     1506.0s        0          442.0          482.0     11.5    906.0   3623.9  24696.1 write
		     1507.0s        0          575.9          482.1     19.9   1610.6   3087.0  22548.6 write
		     1508.0s        0          666.0          482.2     25.2   1073.7   2818.6  10737.4 write
		     1509.0s        0          675.0          482.3     27.3   1040.2   1677.7  14495.5 write
		     1510.0s        0          723.0          482.5     27.3    939.5   1677.7  28991.0 write
		     1511.0s        0          716.9          482.6     21.0    503.3   1811.9   5905.6 write:
		  - exit status 1

	cluster.go:2368,kv.go:262,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@d571624ae4f833c49c717728a74cc7be78a791f0:

		      839.0s        0          591.1          580.4     14.7    906.0   1811.9  11274.3 write
		      840.0s        0          383.1          580.1     11.5   1040.2   1811.9  19327.4 write
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      841.0s        0           91.0          579.6      9.4   2684.4   7784.6  15032.4 write
		      842.0s        0          528.8          579.5     13.1   2147.5   5637.1  12348.0 write
		      843.0s        0          592.9          579.5     15.7   1006.6   3892.3  10200.5 write
		      844.0s        0          702.1          579.7     21.0    838.9   5100.3  17179.9 write
		      845.0s        0          702.1          579.8     17.8    637.5   2147.5  13958.6 write
		      846.0s        0          464.0          579.7     13.1    872.4   1275.1   9126.8 write
		      847.0s        0          329.0          579.4     10.5    771.8   1610.6   7247.8 write
		      848.0s        0          135.0          578.8     11.0   2550.1   8589.9  10737.4 write
		      849.0s        0          267.0          578.5     12.6   2415.9   6442.5  23622.3 write
		      850.0s        0          520.9          578.4     14.2   1208.0   5905.6   8321.5 write
		      851.0s        0          314.0          578.1     11.0   1140.9   6174.0  14495.5 write
		      852.0s        0          119.0          577.6     10.5   5905.6   9663.7  13958.6 write
		      853.0s        0          552.9          577.5     17.8   1879.0   9126.8  23622.3 write
		      854.0s        0          655.0          577.6     17.8    704.6   2147.5  11811.2 write
		      855.0s        0          621.8          577.7     24.1    738.2   4160.7  20401.1 write
		      856.0s        0          446.1          577.5     18.9    469.8    838.9   9663.7 write
		      857.0s        0          162.0          577.0     13.6    771.8   1879.0   3221.2 write:
		  - exit status 1

	cluster.go:2368,kv.go:262,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@andreimatei
Copy link
Contributor

The TransactionStatusError: already committed (REASON_TXN_COMMITTED) failure from 9 days ago is likely #46341.

@cockroach-teamcity
Copy link
Member Author

(roachtest).kv/contention/nodes=4 failed on master@92495ffb2182671b3338a8c6cd2734cb6de5d392:

		     2888.0s        0          690.9          592.0      9.4   1073.7   1946.2  19327.4 write
		     2889.0s        0          446.1          592.0      6.8   1140.9   3892.3  19327.4 write
		     2890.0s        0          859.9          592.1     11.0   1275.1   2684.4  18253.6 write
		     2891.0s        0         1034.0          592.2     16.8    604.0   1543.5  13421.8 write
		     2892.0s        0         1010.8          592.4     16.8    637.5   1811.9  14495.5 write
		     2893.0s        0          898.2          592.5     12.6    486.5    939.5  10200.5 write
		     2894.0s        0          287.1          592.4      7.1   1140.9   1610.6  13421.8 write
		     2895.0s        0          344.0          592.3     10.0   2147.5   3087.0  31138.5 write
		     2896.0s        0          451.9          592.2      7.9   1946.2   3087.0   9126.8 write
		     2897.0s        0          464.1          592.2      8.4   1409.3   2550.1   9126.8 write
		     2898.0s        0          231.0          592.1     10.5   1543.5   5368.7   8053.1 write
		     2899.0s        0          797.7          592.1     15.2   1543.5   2952.8   8589.9 write
		     2900.0s        0          802.2          592.2     13.6    838.9   3355.4  13421.8 write
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     2901.0s        0          813.0          592.3     13.1    671.1   1879.0   9663.7 write
		     2902.0s        0          795.0          592.4     12.6    704.6   1073.7   9663.7 write
		     2903.0s        0          313.1          592.3      6.3    520.1   1140.9   3758.1 write
		     2904.0s        0          247.0          592.1      8.4   2281.7   5100.3  11274.3 write
		     2905.0s        0          777.9          592.2     13.6   1476.4   4563.4   8321.5 write
		     2906.0s        0          446.0          592.2     11.0   1140.9   5368.7  12884.9 write:
		  - exit status 30

	cluster.go:2368,kv.go:262,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.registerKVContention.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:262
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

More

Artifacts: /kv/contention/nodes=4
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@nvanbenschoten
Copy link
Member

^ was ERROR: TransactionStatusError: already committed (REASON_TXN_COMMITTED) (SQLSTATE XXUUU) as well, which should have been fixed by #46596.

@andreimatei can we close this issue as well as #45698 and #46461?

@nvanbenschoten
Copy link
Member

I talked to @andreimatei about this. He saw 40 straight successful runs, so he's ok with closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Projects
None yet
Development

No branches or pull requests

3 participants