roachtest: tpccbench/nodes=3/cpu=16 failed #55802

cockroach-teamcity · 2020-10-21T07:26:19Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@a05641037584c6f9cbb20b77ac8aaabd60875439:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2654,tpcc.go:741,tpcc.go:576,test_runner.go:755: monitor failure: unexpected node event: 2: dead
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2642
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2650
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1657,context.go:140,cluster.go:1646,test_runner.go:836: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2381218-1603260590-25-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: dead
		1: 8582
		3: 8375
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2020-10-27T07:17:40Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@ce32adca8d32ea91b3162c79386143d64a37943a:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2648,tpcc.go:741,tpcc.go:576,test_runner.go:755: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1651,context.go:140,cluster.go:1640,test_runner.go:836: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2397319-1603779051-26-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		2: 8107
		1: 8220
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-10-29T07:45:21Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@7df78765c4e9bbaf6ed61c9c796d3d398f079325:

		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2284
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:739
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2626
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/pkg/mod/golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) output in run_071333.600_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2403616-1603951888-26-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false returned
		  | stderr:
		  | I201029 07:13:34.349021 1 ccl/workloadccl/fixture.go:316  starting import of 9 tables
		  | I201029 07:13:34.438236 67 ccl/workloadccl/fixture.go:446  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 83.884386ms, 1.55 MiB/s)
		  | I201029 07:13:38.955282 73 ccl/workloadccl/fixture.go:446  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 4.600977103s, 1.69 MiB/s)
		  | I201029 07:13:42.097545 68 ccl/workloadccl/fixture.go:446  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 7.743151853s, 0.32 MiB/s)
		  | I201029 07:14:00.510036 72 ccl/workloadccl/fixture.go:446  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 26.155662482s, 12.19 MiB/s)
		  | Error: importing fixture: importing table stock: pq: job 602555070402166785: node liveness error: restarting in background
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *main.withCommandDetails (8) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-10-31T07:17:12Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@4ea9b3bb356663cef8c945607640e9f070a8b002:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2648,tpcc.go:741,tpcc.go:576,test_runner.go:755: monitor failure: unexpected node event: 2: dead
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1651,context.go:140,cluster.go:1640,test_runner.go:836: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2408341-1604124586-18-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: dead
		3: 10240
		1: 11232
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-01T09:41:48Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@d905ffedd2fcbf6555a73b9edf479fdbce06c62c:

		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2672
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2276
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:739
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_082300.022_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2409682-1604214682-24-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false returned
		  | stderr:
		  | I201101 08:23:00.771057 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I201101 08:23:02.113312 29 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 1.337177976s, 1.84 MiB/s)
		  | I201101 08:23:03.265268 146 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 2.489179608s, 3.12 MiB/s)
		  | I201101 08:23:04.525707 28 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 3.749728928s, 0.03 MiB/s)
		  | I201101 08:23:24.057415 33 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 23.281294825s, 13.69 MiB/s)
		  | I201101 08:26:48.157886 32 ccl/workloadccl/fixture.go:472  imported 3.9 GiB in order table (75000000 rows, 75000000 index entries, took 3m47.380011814s, 17.68 MiB/s)
		  | Error: importing fixture: importing table history: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-03T08:32:11Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@2467fe11b6028992fac8f4627385cda68166b497:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2628,tpcc.go:741,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1643,context.go:135,cluster.go:1632,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2412972-1604387299-25-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		1: 11018
		2: 9990
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-06T08:30:57Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@6b6f4dff5e7155e104d7b8d952ea37a3b715de69:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2628,tpcc.go:741,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 2: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1643,context.go:135,cluster.go:1632,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2421895-1604646837-24-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: 10126
		2: dead
		1: 10934
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-12T08:37:33Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@3352a056e5af4a28f6b0bb8fbd200d73ae8a3252:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2628,tpcc.go:741,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 2: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1643,context.go:135,cluster.go:1632,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2436489-1605165581-24-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: dead
		1: 10196
		3: 9728
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-14T08:29:28Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@411bb9c5504a16997ca41c6eea970242de81795c:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2628,tpcc.go:741,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1643,context.go:135,cluster.go:1632,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2442860-1605338169-24-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		1: 10384
		2: 9645
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-18T09:28:48Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@ee306d2b2089a65191c03fe47a63e78c90766a22:

		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_081701.698_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2453732-1605683155-24-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false returned
		  | stderr:
		  | I201118 08:17:02.430964 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I201118 08:17:06.533302 86 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 4.097062517s, 0.60 MiB/s)
		  | I201118 08:17:07.197480 85 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 4.76128274s, 0.03 MiB/s)
		  | I201118 08:17:10.968868 91 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 8.532538775s, 0.91 MiB/s)
		  | I201118 08:17:31.205826 90 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 28.769485092s, 11.08 MiB/s)
		  | Error: importing fixture: importing table order: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1643,context.go:135,cluster.go:1632,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2453732-1605683155-24-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: dead
		3: 9846
		1: 10417
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-19T08:19:04Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@fb3e12fa3fca5060da5e176186df465046d85ac4:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2628,tpcc.go:741,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1643,context.go:135,cluster.go:1632,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2457342-1605769307-25-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		1: 8713
		2: 8053
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-11-21T09:45:11Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@e384e65cbd5f53a6aaf169444ea40f8a39de16e7:

		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_082541.123_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2463343-1605942030-28-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false returned
		  | stderr:
		  | I201121 08:25:41.856551 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I201121 08:25:42.759474 73 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 897.530902ms, 2.74 MiB/s)
		  | I201121 08:25:45.917099 78 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 4.055087341s, 1.92 MiB/s)
		  | I201121 08:25:46.217388 72 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 4.355444669s, 0.03 MiB/s)
		  | I201121 08:26:02.290957 77 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 20.42890962s, 15.60 MiB/s)
		  | Error: importing fixture: importing table order: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1643,context.go:135,cluster.go:1632,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2463343-1605942030-28-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: dead
		3: 7791
		1: 8252
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-12-02T08:32:48Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@999a5d5fe22fa87f18fd3212faf9d64f0b6a57d9:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2628,tpcc.go:741,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 2: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1643,context.go:135,cluster.go:1632,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2488213-1606892618-28-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: 8503
		2: dead
		3: 8847
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-12-03T08:17:06Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@2469ed807571b6c0811e07c69de5434c17e3ad71:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2628,tpcc.go:741,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 2: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1643,context.go:135,cluster.go:1632,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2491323-1606978855-26-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: dead
		3: 8250
		1: 8408
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-12-04T08:16:16Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@4fd42817e815b54ae4ff5012421ae5b23a732061:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2628,tpcc.go:741,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1643,context.go:135,cluster.go:1632,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2495116-1607065272-26-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		1: 8359
		2: 7823
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-12-05T09:36:52Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@e333a6a494036b21d7f56d7c89acd65e589faa9b:

		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2672
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2276
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:739
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_081857.602_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2498165-1607151990-26-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false returned
		  | stderr:
		  | I201205 08:18:58.337659 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I201205 08:18:58.619459 101 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 275.031345ms, 8.95 MiB/s)
		  | I201205 08:18:58.800608 100 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 456.250064ms, 0.28 MiB/s)
		  | I201205 08:19:06.880224 106 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 8.535457545s, 0.91 MiB/s)
		  | I201205 08:19:23.800283 105 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 25.455814555s, 12.52 MiB/s)
		  | Error: importing fixture: importing table order: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-12-15T09:41:57Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@93e6cb11e7479182fe2f1dff766748e1e860625c:

		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:741
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2672
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2276
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:739
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_082539.494_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2519794-1608016372-24-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false returned
		  | stderr:
		  | I201215 08:25:40.222110 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I201215 08:25:42.866535 12 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 2.638862221s, 0.05 MiB/s)
		  | I201215 08:25:43.830288 83 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 3.60253429s, 2.16 MiB/s)
		  | I201215 08:25:46.861657 13 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 6.633964517s, 0.37 MiB/s)
		  | I201215 08:26:08.419473 82 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 28.191733244s, 11.31 MiB/s)
		  | Error: importing fixture: importing table order: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --scatter --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-03-20T07:42:56Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@225c193861d96f9194e93a7743c4899ffb89d3a0:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2629,tpcc.go:735,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 2: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2795663-1616221639-26-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: dead
		3: 8014
		1: 8304
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62244 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure O-roachtest O-robot branch-release-21.1 release-blocker
roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62145 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-03-23T07:46:42Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@7c72158607012aeaf9823da66d51b259d6d8e04a:

		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_074310.262_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2802958-1616480739-18-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210323 07:43:10.968417 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210323 07:43:12.388675 118 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 1.413909333s, 1.74 MiB/s)
		  | I210323 07:43:18.483289 123 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 7.508478326s, 1.04 MiB/s)
		  | I210323 07:43:20.843798 117 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 9.869121309s, 0.01 MiB/s)
		  | I210323 07:43:39.912916 122 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 28.938159248s, 11.01 MiB/s)
		  | W210323 07:46:20.945741 1 ccl/workloadccl/fixture.go:523  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table history: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2802958-1616480739-18-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: 11354
		1: dead
		3: 12409
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62244 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure GA-blocker O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62145 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-03-24T07:41:09Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@7cddba3cf2372b2f49090e8c92e4f0edae119e8f:

		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_073717.501_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2808020-1616566766-26-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210324 07:37:18.223142 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210324 07:37:20.139090 16 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 1.911249915s, 1.29 MiB/s)
		  | I210324 07:37:21.048737 86 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 2.820873233s, 2.76 MiB/s)
		  | I210324 07:37:21.198732 15 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 2.97098189s, 0.04 MiB/s)
		  | I210324 07:37:44.948505 85 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 26.720637427s, 11.93 MiB/s)
		  | W210324 07:40:40.034131 1 ccl/workloadccl/fixture.go:523  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table customer: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2808020-1616566766-26-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		2: 7830
		3: 8366
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62244 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure GA-blocker O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62145 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-03-25T08:56:49Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@b70484005143e493b737c24121222b81e0202d1f:

		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2673
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2277
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:733
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2607
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_073114.582_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2812279-1616653162-26-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210325 07:31:15.310614 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210325 07:31:16.172186 83 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 856.508104ms, 0.15 MiB/s)
		  | I210325 07:31:18.004652 84 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 2.688997654s, 0.92 MiB/s)
		  | I210325 07:31:18.781023 89 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 3.465213854s, 2.24 MiB/s)
		  | I210325 07:31:38.480887 88 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 23.165131643s, 13.76 MiB/s)
		  | Error: importing fixture: importing table order: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62244 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure GA-blocker O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62145 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure GA-blocker O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-07T08:16:41Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@fc34bfc7449aa977172697eac6d5b7aea40e3c89:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2629,tpcc.go:735,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2854971-1617777959-27-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		2: 8478
		1: 8978
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62244 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62145 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-08T20:48:43Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@52eb35de93ce6fc002ddff776465f25efc2a4aaf:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2629,tpcc.go:735,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2862776-1617909334-28-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		1: 8934
		2: 8235
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62244 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62145 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-09T20:40:31Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@5143ce7f50e73c3261cd2426b751840d7fd8a0a9:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2629,tpcc.go:735,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2867154-1617996516-24-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		1: 10611
		2: 9805
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62244 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62145 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-12T20:36:55Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@315c250367d557ff8708ab9c10c9931c76241ca8:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2629,tpcc.go:735,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 2: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2873169-1618255147-18-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: dead
		1: 12900
		3: 11903
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62244 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62145 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-13T21:53:13Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@79dfc5bb84491145a49d17c17a34d1459694d33e:

		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2673
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2277
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:733
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2607
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_203156.424_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2878605-1618341553-25-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210413 20:31:57.203307 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210413 20:31:59.088993 48 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 1.874655439s, 4.15 MiB/s)
		  | I210413 20:31:59.846889 43 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 2.632675529s, 0.94 MiB/s)
		  | I210413 20:32:01.434018 42 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 4.219721125s, 0.03 MiB/s)
		  | I210413 20:32:24.785488 47 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 27.571209933s, 11.56 MiB/s)
		  | Error: importing fixture: importing table order: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62244 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62145 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-19T20:56:23Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@ca18ad3c762751c4f8a8f596a802b9e64da32fd8:

		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2673
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2277
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:733
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2607
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_203628.347_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2898519-1618859812-18-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210419 20:36:29.072208 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210419 20:36:29.230057 55 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 151.604915ms, 0.86 MiB/s)
		  | I210419 20:36:29.906308 56 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 828.207647ms, 2.97 MiB/s)
		  | I210419 20:36:30.703454 114 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 1.624730455s, 4.79 MiB/s)
		  | I210419 20:36:57.604374 60 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 28.526135473s, 11.17 MiB/s)
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62244 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=3/cpu=16 failed [overload] #62145 roachtest: tpccbench/nodes=3/cpu=16 failed [overload] C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-22T20:33:06Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@5e4b6339baedc91157fbf233bf01840b96ab93e6:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2629,tpcc.go:727,tpcc.go:568,test_runner.go:749: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:727
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:568
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2913147-1619118914-26-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		2: 8223
		1: 8386
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-30T21:00:50Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@abbee628ae7c92d0e3b05dd71628cde6822c6f04:

		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_205700.107_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2941653-1619811408-24-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210430 20:57:00.833139 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210430 20:57:02.783433 41 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 1.944834558s, 1.27 MiB/s)
		  | I210430 20:57:02.816660 40 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 1.978106452s, 0.07 MiB/s)
		  | I210430 20:57:04.154181 46 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 3.315535787s, 2.35 MiB/s)
		  | I210430 20:57:29.729891 45 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 28.891236447s, 11.03 MiB/s)
		  | W210430 21:00:22.549043 1 ccl/workloadccl/fixture.go:523  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table order_line: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2941653-1619811408-24-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		3: 9802
		2: 9819
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-05-01T20:26:59Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@fe64cf1e195f1927f1d8ba2dc289735a37dc61fb:

		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_202301.083_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2944572-1619896608-23-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210501 20:23:01.802523 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210501 20:23:03.751913 21 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 1.943982523s, 0.07 MiB/s)
		  | I210501 20:23:07.165269 22 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 5.357344892s, 0.46 MiB/s)
		  | I210501 20:23:09.087200 27 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 7.279106744s, 1.07 MiB/s)
		  | I210501 20:23:28.366230 26 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 26.558234873s, 12.00 MiB/s)
		  | W210501 20:26:34.406375 1 ccl/workloadccl/fixture.go:523  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table order: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2944572-1619896608-23-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		2: 9942
		3: 9472
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-05-04T20:45:29Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@3ae6b59f0ea417486c3c41f85ea5baff3b5a00df:

		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_204140.181_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2949856-1620155910-25-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210504 20:41:40.908415 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210504 20:41:42.113775 25 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 1.199450068s, 0.11 MiB/s)
		  | I210504 20:41:43.008344 31 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 2.094241296s, 3.71 MiB/s)
		  | I210504 20:41:44.969489 26 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 4.055550526s, 0.61 MiB/s)
		  | I210504 20:42:01.544746 30 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 20.630714514s, 15.45 MiB/s)
		  | W210504 20:45:06.855728 1 ccl/workloadccl/fixture.go:523  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table order_line: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2949856-1620155910-25-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		3: 9893
		2: 9907
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed #64633 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure O-roachtest O-robot branch-release-20.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-05-11T20:24:57Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@c4aec65cb998ba1fe19b02a29e9accba1baa4362:

		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_202118.447_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2969652-1620760430-24-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210511 20:21:19.163678 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210511 20:21:20.780989 58 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 1.611612084s, 1.53 MiB/s)
		  | I210511 20:21:20.984774 57 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 1.815315658s, 0.07 MiB/s)
		  | I210511 20:21:26.421952 63 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 7.252477928s, 1.07 MiB/s)
		  | I210511 20:21:46.646354 62 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 27.476916015s, 11.60 MiB/s)
		  | W210511 20:24:28.784221 1 ccl/workloadccl/fixture.go:523  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table order: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2969652-1620760430-24-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		3: 10402
		2: 10739
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed #64967 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure O-roachtest O-robot branch-release-21.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-05-12T20:39:21Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@9a63e2984592098cf2b1c12e4e55bffcc499d3ec:

		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_203511.635_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2973771-1620846775-27-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210512 20:35:12.328608 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210512 20:35:13.506013 114 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 1.172343593s, 0.11 MiB/s)
		  | I210512 20:35:14.202260 120 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 1.868679946s, 4.16 MiB/s)
		  | I210512 20:35:22.571709 115 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 10.238018612s, 0.24 MiB/s)
		  | I210512 20:35:45.070287 119 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 32.736773532s, 9.74 MiB/s)
		  | W210512 20:38:42.359391 1 ccl/workloadccl/fixture.go:523  error enabling automatic stats: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: importing fixture: importing table stock: dial tcp 127.0.0.1:26257: connect: connection refused
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2973771-1620846775-27-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: 8409
		1: dead
		3: 8562
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed #64967 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure O-roachtest O-robot branch-release-21.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-05-13T20:40:35Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@90ea8c252b1efd812336317b8c4b528cc1fe79e7:

		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:729
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:564
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2673
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2277
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:631
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:727
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2607
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_202437.890_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2978405-1620933470-24-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210513 20:24:38.666856 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210513 20:24:40.779139 68 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 2.099800333s, 0.06 MiB/s)
		  | I210513 20:24:42.486923 74 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 3.807441243s, 2.04 MiB/s)
		  | I210513 20:24:44.366989 69 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 5.687672907s, 0.43 MiB/s)
		  | I210513 20:25:00.917858 73 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 22.238414319s, 14.33 MiB/s)
		  | I210513 20:27:34.449460 72 ccl/workloadccl/fixture.go:472  imported 3.9 GiB in order table (75000000 rows, 75000000 index entries, took 2m55.766196705s, 22.87 MiB/s)
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed #64967 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure O-roachtest O-robot branch-release-21.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-05-19T20:49:19Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@dbb89ba3fb8c9d642f7a10b169cd6e453c9a19e8:

		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:729
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:564
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2673
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2277
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:631
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:727
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2607
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_202846.197_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2998309-1621452355-25-n4cpu16:1 -- ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false returned
		  | stderr:
		  | I210519 20:28:46.955342 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210519 20:28:47.473956 47 ccl/workloadccl/fixture.go:472  imported 133 KiB in warehouse table (2500 rows, 0 index entries, took 513.216321ms, 0.25 MiB/s)
		  | I210519 20:28:49.651757 133 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 2.690932356s, 2.89 MiB/s)
		  | I210519 20:28:50.951842 48 ccl/workloadccl/fixture.go:472  imported 2.5 MiB in district table (25000 rows, 0 index entries, took 3.991176411s, 0.62 MiB/s)
		  | I210519 20:29:09.854955 132 ccl/workloadccl/fixture.go:472  imported 319 MiB in new_order table (22500000 rows, 0 index entries, took 22.894097736s, 13.92 MiB/s)
		  | I210519 20:31:39.198825 131 ccl/workloadccl/fixture.go:472  imported 3.9 GiB in order table (75000000 rows, 75000000 index entries, took 2m52.23622112s, 23.34 MiB/s)
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=2500 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed #64967 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure O-roachtest O-robot branch-release-21.1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-05-19T23:19:48Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@dbb89ba3fb8c9d642f7a10b169cd6e453c9a19e8:

		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |      0          450.1          252.4     58.7    159.4    184.5    285.2 payment
		  |   257.0s        0           50.0           24.8     28.3     75.5     88.1     88.1 stockLevel
		  |   258.0s        0           45.0           25.4    100.7    159.4    176.2    176.2 delivery
		  |   258.0s        0          422.7          232.0     75.5    121.6    151.0    285.2 newOrder
		  |   258.0s        0           56.0           25.6      8.1     24.1     29.4     32.5 orderStatus
		  |   258.0s        0          465.7          253.2     52.4    121.6    159.4    192.9 payment
		  |   258.0s        0           50.0           24.9     31.5     56.6     58.7     58.7 stockLevel
		  |   259.0s        0           38.0           25.4    113.2    268.4    352.3    352.3 delivery
		  |   259.0s        0          499.2          233.0     96.5    201.3    234.9    260.0 newOrder
		  |   259.0s        0           55.0           25.8     10.0     35.7     41.9     41.9 orderStatus
		  |   259.0s        0          455.1          254.0     62.9    151.0    184.5    251.7 payment
		  |   259.0s        0           38.0           25.0     37.7     79.7    184.5    184.5 stockLevel
		  |   260.0s        0           57.0           25.6     96.5    268.4    285.2    285.2 delivery
		  |   260.0s        0          446.0          233.8    117.4    285.2    335.5    385.9 newOrder
		  |   260.0s        0           54.0           25.9      8.1     24.1     29.4     29.4 orderStatus
		  |   260.0s        0          477.0          254.9     83.9    184.5    218.1    419.4 payment
		  |   260.0s        0           35.0           25.0     29.4     71.3    142.6    142.6 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |   261.0s        0           34.0           25.6    142.6    251.7    352.3    352.3 delivery
		  |   261.0s        0          473.8          234.7    134.2    302.0    335.5    453.0 newOrder
		  |   261.0s        0           50.0           26.0      7.1     39.8     60.8     60.8 orderStatus
		  |   261.0s        0          508.8          255.8    104.9    209.7    234.9    285.2 payment
		  |   261.0s        0           57.0           25.1     30.4     79.7     88.1     96.5 stockLevel
		  |   262.0s        0           52.0           25.7    159.4    352.3    369.1    385.9 delivery
		  |   262.0s        0          473.9          235.6    134.2    285.2    318.8    520.1 newOrder
		  |   262.0s        0           50.0           26.0     10.5     31.5     39.8     39.8 orderStatus
		  |   262.0s        0          560.9          257.0    104.9    285.2    352.3    520.1 payment
		  |   262.0s        0           45.0           25.2     32.5     92.3    134.2    134.2 stockLevel
		  |   263.0s        0           51.0           25.8    100.7    251.7    268.4    268.4 delivery
		  |   263.0s        0          491.3          236.6    121.6    251.7    302.0    671.1 newOrder
		  |   263.0s        0           54.0           26.2      6.3     27.3     29.4     41.9 orderStatus
		  |   263.0s        0          468.3          257.8     92.3    234.9    268.4    637.5 payment
		  |   263.0s        0           59.0           25.3     27.3     60.8    113.2    335.5 stockLevel
		  |   264.0s        0           50.0           25.9    100.7    234.9    302.0    302.0 delivery
		  |   264.0s        0          470.8          237.5    109.1    218.1    251.7    302.0 newOrder
		  |   264.0s        0           45.0           26.2      5.8     17.8     19.9     19.9 orderStatus
		  |   264.0s        0          496.8          258.7     83.9    176.2    209.7    243.3 payment
		  |   264.0s        0           50.0           25.4     24.1     62.9     71.3     71.3 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |   265.0s        0           55.0           26.0    109.1    302.0    318.8    385.9 delivery
		  |   265.0s        0          517.0          238.6    121.6    251.7    302.0    402.7 newOrder
		  |   265.0s        0           50.0           26.3     10.0     29.4     39.8     39.8 orderStatus
		  |   265.0s        0          525.0          259.7     92.3    260.0    302.0    335.5 payment
		  |   265.0s        0           45.0           25.5     28.3     54.5     71.3     71.3 stockLevel
		Wraps: (12) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *withstack.withStack (9) *safedetails.withSafeDetails (10) *errutil.withMessage (11) *main.withCommandDetails (12) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed #64967 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure O-roachtest O-robot branch-release-21.1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-06-09T07:42:44Z

(roachtest).tpccbench/nodes=3/cpu=16 failed on release-20.1@350d70a77ce4fbce086a96992ceb2bd48871e974:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
	cluster.go:2629,tpcc.go:729,tpcc.go:564,test_runner.go:749: monitor failure: unexpected node event: 2: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:729
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:564
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3063295-1623219214-26-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: dead
		3: 12626
		1: 13323
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=3/cpu=16
Related:

roachtest: tpccbench/nodes=3/cpu=16 failed #64967 roachtest: tpccbench/nodes=3/cpu=16 failed C-test-failure O-roachtest O-robot branch-release-21.1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

tbg · 2021-11-11T11:28:04Z

Closing, inactivity

cockroach-teamcity added branch-release-20.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Oct 21, 2020

cockroach-teamcity added this to the 20.2 milestone Oct 21, 2020

nvanbenschoten mentioned this issue Oct 27, 2020

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #55599

Closed

mwang1026 assigned dt Oct 27, 2020

pbardea removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Dec 3, 2020

kenliu added the T-disaster-recovery label Dec 5, 2020

cockroach-teamcity mentioned this issue Apr 22, 2021

roachtest: tpccbench/nodes=3/cpu=16 failed [possible release-21.1 regression?] #64110

Closed

cockroach-teamcity mentioned this issue May 4, 2021

roachtest: tpccbench/nodes=3/cpu=16 failed #64633

Closed

cockroach-teamcity mentioned this issue May 10, 2021

roachtest: tpccbench/nodes=3/cpu=16 failed #64967

Closed

cockroach-teamcity mentioned this issue May 19, 2021

roachtest: tpccbench/nodes=3/cpu=16 failed #65489

Closed

cockroach-teamcity mentioned this issue Jun 9, 2021

roachtest: tpccbench/nodes=3/cpu=16 failed [full disk] #66241

Closed

This was referenced Nov 10, 2021

roachtest: tpccbench/nodes=3/cpu=16 failed [perm denied #72635] #72598

Closed

roachtest: tpccbench/nodes=3/cpu=16/no-admission failed [perm denied #72635] #72599

Closed

tbg closed this as completed Nov 11, 2021

roachtest: tpccbench/nodes=3/cpu=16 failed #55802

roachtest: tpccbench/nodes=3/cpu=16 failed #55802

Comments

cockroach-teamcity commented Oct 21, 2020 • edited by exalate-issue-sync bot Loading

cockroach-teamcity commented Oct 27, 2020

cockroach-teamcity commented Oct 29, 2020

cockroach-teamcity commented Oct 31, 2020

cockroach-teamcity commented Nov 1, 2020

cockroach-teamcity commented Nov 3, 2020

cockroach-teamcity commented Nov 6, 2020

cockroach-teamcity commented Nov 12, 2020

cockroach-teamcity commented Nov 14, 2020

cockroach-teamcity commented Nov 18, 2020

cockroach-teamcity commented Nov 19, 2020

cockroach-teamcity commented Nov 21, 2020

cockroach-teamcity commented Dec 2, 2020

cockroach-teamcity commented Dec 3, 2020

cockroach-teamcity commented Dec 4, 2020

cockroach-teamcity commented Dec 5, 2020

cockroach-teamcity commented Dec 15, 2020

cockroach-teamcity commented Mar 20, 2021

cockroach-teamcity commented Mar 23, 2021

cockroach-teamcity commented Mar 24, 2021

cockroach-teamcity commented Mar 25, 2021

cockroach-teamcity commented Apr 7, 2021

cockroach-teamcity commented Apr 8, 2021

cockroach-teamcity commented Apr 9, 2021

cockroach-teamcity commented Apr 12, 2021

cockroach-teamcity commented Apr 13, 2021

cockroach-teamcity commented Apr 19, 2021

cockroach-teamcity commented Apr 22, 2021

cockroach-teamcity commented Apr 30, 2021

cockroach-teamcity commented May 1, 2021

cockroach-teamcity commented May 4, 2021

cockroach-teamcity commented May 11, 2021

cockroach-teamcity commented May 12, 2021

cockroach-teamcity commented May 13, 2021

cockroach-teamcity commented May 19, 2021

cockroach-teamcity commented May 19, 2021

cockroach-teamcity commented Jun 9, 2021

tbg commented Nov 11, 2021

cockroach-teamcity commented Oct 21, 2020 •

edited by exalate-issue-sync bot

Loading