roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #59044

cockroach-teamcity · 2021-01-15T12:16:49Z

(roachtest).tpccbench/nodes=6/cpu=16/multi-az failed on release-20.1@df6eeda947153b1e5d5b87f3fb1c18fe3ff58901:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1
	cluster.go:2628,tpcc.go:735,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 2: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

More

Artifacts: /tpccbench/nodes=6/cpu=16/multi-az
Related:

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #58641 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-release-20.2 release-blocker
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #58298 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #55544 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-release-19.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

nvanbenschoten · 2021-01-20T20:36:13Z

One of the nodes OOMed during an IMPORT. @dt is this expected on release-20.1? Could someone on your team determine whether we should consider this a release blocker?

cockroach-teamcity · 2021-02-10T12:18:35Z

(roachtest).tpccbench/nodes=6/cpu=16/multi-az failed on release-20.1@90f78268f3b5b08ba838ac3ad164821d2f5a5362:

		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2661563-1612941348-136-n7cpu16-geo:1 -- ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false returned
		  | stderr:
		  | I210210 10:15:43.292686 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210210 10:15:45.327293 120 ccl/workloadccl/fixture.go:472  imported 266 KiB in warehouse table (5000 rows, 0 index entries, took 2.028043669s, 0.13 MiB/s)
		  | I210210 10:15:47.726522 126 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 4.427166586s, 1.76 MiB/s)
		  | I210210 10:15:49.852446 121 ccl/workloadccl/fixture.go:472  imported 4.9 MiB in district table (50000 rows, 0 index entries, took 6.553278908s, 0.75 MiB/s)
		  | I210210 10:16:36.492792 125 ccl/workloadccl/fixture.go:472  imported 641 MiB in new_order table (45000000 rows, 0 index entries, took 53.193506868s, 12.04 MiB/s)
		  | Error: importing fixture: importing table order: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

	cluster.go:1643,context.go:135,cluster.go:1632,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2661563-1612941348-136-n7cpu16-geo --oneshot --ignore-empty-nodes: exit status 1 7: skipped
		6: dead
		2: 4465
		3: 5172
		4: 4445
		1: 5378
		5: 4467
		Error: UNCLASSIFIED_PROBLEM: 6: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 6: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=6/cpu=16/multi-az
Related:

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #60301 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-60149 release-blocker
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #58641 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-release-20.2
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #58298 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-master
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #55544 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-release-19.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-18T12:15:41Z

(roachtest).tpccbench/nodes=6/cpu=16/multi-az failed on release-20.1@8f2e62a151f638bf0ee0caccc765f44685684d45:

		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2616
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2624
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2672
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2276
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:733
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_115610.994_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2688429-1613632541-63-n7cpu16-geo:1 -- ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false returned
		  | stderr:
		  | I210218 11:56:11.781187 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210218 11:56:12.032108 15 ccl/workloadccl/fixture.go:472  imported 266 KiB in warehouse table (5000 rows, 0 index entries, took 237.428876ms, 1.09 MiB/s)
		  | I210218 11:56:12.081266 16 ccl/workloadccl/fixture.go:472  imported 4.9 MiB in district table (50000 rows, 0 index entries, took 286.580391ms, 17.19 MiB/s)
		  | I210218 11:56:17.294174 70 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 5.499319361s, 1.41 MiB/s)
		  | I210218 11:56:39.835462 69 ccl/workloadccl/fixture.go:472  imported 641 MiB in new_order table (45000000 rows, 0 index entries, took 28.040661676s, 22.84 MiB/s)
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=6/cpu=16/multi-az
Related:

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #58641 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-release-20.2
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #58298 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-master
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #55544 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-release-19.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-27T12:07:32Z

(roachtest).tpccbench/nodes=6/cpu=16/multi-az failed on release-20.1@c7f84c91e1ca37c457e0c929d8a301bc88510973:

The test failed on branch=release-20.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1
	cluster.go:2629,tpcc.go:735,tpcc.go:576,test_runner.go:749: monitor failure: unexpected node event: 5: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 5: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1644,context.go:135,cluster.go:1633,test_runner.go:830: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2725126-1614410267-62-n7cpu16-geo --oneshot --ignore-empty-nodes: exit status 1 7: skipped
		5: dead
		1: 6328
		4: 6277
		2: 6263
		6: 6200
		3: 6248
		Error: UNCLASSIFIED_PROBLEM: 5: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:204
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 5: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /tpccbench/nodes=6/cpu=16/multi-az
Related:

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #61189 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-master release-blocker
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #58641 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-release-20.2
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #55544 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-release-19.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-03-16T11:19:54Z

(roachtest).tpccbench/nodes=6/cpu=16/multi-az failed on release-20.1@fc7cc3f669e5bd0a28666bb48b2193756b31a00d:

		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2673
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2277
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:733
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2607
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_104937.541_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2780717-1615875369-63-n7cpu16-geo:1 -- ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false returned
		  | stderr:
		  | I210316 10:49:38.258837 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210316 10:49:39.238173 26 ccl/workloadccl/fixture.go:472  imported 266 KiB in warehouse table (5000 rows, 0 index entries, took 972.696952ms, 0.27 MiB/s)
		  | I210316 10:49:42.011917 27 ccl/workloadccl/fixture.go:472  imported 4.9 MiB in district table (50000 rows, 0 index entries, took 3.746461662s, 1.31 MiB/s)
		  | I210316 10:49:42.499451 32 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 4.233839455s, 1.84 MiB/s)
		  | I210316 10:50:03.262234 31 ccl/workloadccl/fixture.go:472  imported 641 MiB in new_order table (45000000 rows, 0 index entries, took 24.996639806s, 25.63 MiB/s)
		  | I210316 10:57:53.223509 30 ccl/workloadccl/fixture.go:472  imported 7.9 GiB in order table (150000000 rows, 150000000 index entries, took 8m14.956078957s, 16.28 MiB/s)
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=6/cpu=16/multi-az
Related:

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #61974 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-03-24T12:49:31Z

(roachtest).tpccbench/nodes=6/cpu=16/multi-az failed on release-20.1@7cddba3cf2372b2f49090e8c92e4f0edae119e8f:

		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2673
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2277
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:733
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2607
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_104119.981_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2808020-1616566766-64-n7cpu16-geo:1 -- ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false returned
		  | stderr:
		  | I210324 10:41:20.722173 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210324 10:41:23.972127 101 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 3.243181373s, 2.40 MiB/s)
		  | I210324 10:41:24.101798 14 ccl/workloadccl/fixture.go:472  imported 266 KiB in warehouse table (5000 rows, 0 index entries, took 3.373046862s, 0.08 MiB/s)
		  | I210324 10:41:24.102145 15 ccl/workloadccl/fixture.go:472  imported 4.9 MiB in district table (50000 rows, 0 index entries, took 3.373389931s, 1.46 MiB/s)
		  | I210324 10:43:59.286653 100 ccl/workloadccl/fixture.go:472  imported 641 MiB in new_order table (45000000 rows, 0 index entries, took 2m38.557742678s, 4.04 MiB/s)
		  | Error: importing fixture: importing table order: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=6/cpu=16/multi-az
Related:

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #62339 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure GA-blocker O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #61974 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-03-25T13:51:24Z

(roachtest).tpccbench/nodes=6/cpu=16/multi-az failed on release-20.1@b70484005143e493b737c24121222b81e0202d1f:

		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2673
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2277
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:733
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2607
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_105049.912_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2812279-1616653162-63-n7cpu16-geo:1 -- ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false returned
		  | stderr:
		  | I210325 10:50:50.666022 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210325 10:50:52.382857 59 ccl/workloadccl/fixture.go:472  imported 266 KiB in warehouse table (5000 rows, 0 index entries, took 1.704247853s, 0.15 MiB/s)
		  | I210325 10:50:52.490916 65 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 1.812145617s, 4.29 MiB/s)
		  | I210325 10:50:57.844768 60 ccl/workloadccl/fixture.go:472  imported 4.9 MiB in district table (50000 rows, 0 index entries, took 7.166179373s, 0.69 MiB/s)
		  | I210325 10:51:51.208575 64 ccl/workloadccl/fixture.go:472  imported 641 MiB in new_order table (45000000 rows, 0 index entries, took 1m0.529820803s, 10.58 MiB/s)
		  | Error: importing fixture: importing table order: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=6/cpu=16/multi-az
Related:

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #62339 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure GA-blocker O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #61974 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed C-test-failure GA-blocker O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-07T13:24:34Z

(roachtest).tpccbench/nodes=6/cpu=16/multi-az failed on release-20.1@fc34bfc7449aa977172697eac6d5b7aea40e3c89:

		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2673
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2277
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:733
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2607
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_111714.093_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2854971-1617777959-58-n7cpu16-geo:1 -- ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false returned
		  | stderr:
		  | I210407 11:17:14.833504 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210407 11:17:17.148023 87 ccl/workloadccl/fixture.go:472  imported 4.9 MiB in district table (50000 rows, 0 index entries, took 2.301715944s, 2.14 MiB/s)
		  | I210407 11:17:23.374358 86 ccl/workloadccl/fixture.go:472  imported 266 KiB in warehouse table (5000 rows, 0 index entries, took 8.528104397s, 0.03 MiB/s)
		  | I210407 11:17:24.292216 92 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 9.445580511s, 0.82 MiB/s)
		  | I210407 11:18:47.955850 91 ccl/workloadccl/fixture.go:472  imported 641 MiB in new_order table (45000000 rows, 0 index entries, took 1m33.109415185s, 6.88 MiB/s)
		  | Error: importing fixture: importing table order: pq: communication error: rpc error: code = Canceled desc = context canceled
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=6/cpu=16/multi-az
Related:

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #62339 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] C-test-failure O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #61974 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-13T00:21:59Z

(roachtest).tpccbench/nodes=6/cpu=16/multi-az failed on release-20.1@315c250367d557ff8708ab9c10c9931c76241ca8:

		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2673
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2277
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:733
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2607
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_000433.719_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2873169-1618255147-65-n7cpu16-geo:1 -- ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false returned
		  | stderr:
		  | I210413 00:04:34.445536 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210413 00:04:36.199328 82 ccl/workloadccl/fixture.go:472  imported 266 KiB in warehouse table (5000 rows, 0 index entries, took 1.742179418s, 0.15 MiB/s)
		  | I210413 00:04:36.310041 83 ccl/workloadccl/fixture.go:472  imported 4.9 MiB in district table (50000 rows, 0 index entries, took 1.852918823s, 2.66 MiB/s)
		  | I210413 00:04:36.491549 88 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 2.033767861s, 3.82 MiB/s)
		  | I210413 00:04:58.933345 87 ccl/workloadccl/fixture.go:472  imported 641 MiB in new_order table (45000000 rows, 0 index entries, took 24.476074293s, 26.17 MiB/s)
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=6/cpu=16/multi-az
Related:

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #62339 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] C-test-failure O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #61974 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-04-14T00:13:40Z

(roachtest).tpccbench/nodes=6/cpu=16/multi-az failed on release-20.1@79dfc5bb84491145a49d17c17a34d1459694d33e:

		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2617
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2625
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:735
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:576
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:749
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2673
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2277
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:643
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:733
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2607
		  | github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) 2 safe details enclosed
		Wraps: (7) output in run_234355.811_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (8) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2878605-1618341553-67-n7cpu16-geo:1 -- ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false returned
		  | stderr:
		  | I210413 23:43:56.552717 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210413 23:44:00.237721 102 ccl/workloadccl/fixture.go:472  imported 4.9 MiB in district table (50000 rows, 0 index entries, took 3.678391635s, 1.34 MiB/s)
		  | I210413 23:44:00.797336 107 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 4.237834161s, 1.83 MiB/s)
		  | I210413 23:44:03.759847 101 ccl/workloadccl/fixture.go:472  imported 266 KiB in warehouse table (5000 rows, 0 index entries, took 7.200533736s, 0.04 MiB/s)
		  | I210413 23:44:25.685314 106 ccl/workloadccl/fixture.go:472  imported 641 MiB in new_order table (45000000 rows, 0 index entries, took 29.125874488s, 21.99 MiB/s)
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=5000 --checks=false
		  |   | ```
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (9) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *safedetails.withSafeDetails (7) *errutil.withMessage (8) *main.withCommandDetails (9) *exec.ExitError

More

Artifacts: /tpccbench/nodes=6/cpu=16/multi-az
Related:

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #62339 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] C-test-failure O-roachtest O-robot branch-release-21.1
roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #61974 roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

tbg · 2021-04-14T09:47:53Z

This seems to be failing like clockwork. I looked at the most recent failure and the cluster is just sitting there, while the client ssh connection drops with status 255 (ssh problem).

dt · 2021-04-14T17:06:19Z

I looked briefly at a couple of these last night. Some looked like the usual LSM backpressure, e.g.

[n1,s1] SST ingestion was delayed by 5.21129418s (5.002775196s for storage engine back-pressure)

I'm sorta tempted to just min-version this test to 20.2 and pebble since honestly I don't think anybody is going to go back and try to work on this in 20.1 at this point.

tbg · 2021-04-14T19:48:10Z

I mean, the failure looks like an infra failure to me? The conn drop seems to happen ~minutes in (or did I miss something)? It's not like this was sitting around forever and eventually flaked out. I'm fine skipping this test here but it fails like that on all the branches, or so I think, have to look again tomorrow.

dt · 2021-04-14T20:06:30Z

no, not missing anything, I was looking at an earlier failure when I looked at it yesterday

adityamaru · 2021-04-19T15:23:55Z

@tbg we discussed this in our team meeting and are leaning towards min versioning this test to 20.2 (with pebble). Given that we're kinda stretched at the moment we don't think we'll get to invest significant time into debugging 20.1 import failures. I can put out a PR for the same if you're okay with this?

tbg · 2021-04-19T15:38:25Z

I'm fine what that.

tbg · 2021-04-19T15:39:24Z

(but roachtest is taken from the branch, right? So you're really saying you're going to skipping this test on the 20.2 branch).

cockroach-teamcity added branch-release-20.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jan 15, 2021

cockroach-teamcity added this to the 20.1 milestone Jan 15, 2021

cockroach-teamcity mentioned this issue Jan 15, 2021

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #58298

Closed

JuanLeon1 assigned nvanbenschoten Jan 19, 2021

nvanbenschoten assigned dt and unassigned nvanbenschoten Jan 20, 2021

dt removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jan 20, 2021

This was referenced Feb 9, 2021

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #58641

Closed

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #60301

Closed

This was referenced Feb 24, 2021

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #61071

Closed

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #61189

Closed

This was referenced Mar 9, 2021

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #61718

Closed

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #61974

Closed

cockroach-teamcity mentioned this issue Mar 22, 2021

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed [overload] #62339

Closed

cockroach-teamcity mentioned this issue Apr 14, 2021

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #63615

Closed

dt removed their assignment Apr 14, 2021

mwang1026 assigned adityamaru Apr 19, 2021

adityamaru mentioned this issue Apr 20, 2021

roachtest: delete tpccbench/nodes=6/cpu=16/multi-az roachtest #63913

Merged

tbg closed this as completed Apr 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #59044

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #59044

cockroach-teamcity commented Jan 15, 2021

nvanbenschoten commented Jan 20, 2021

cockroach-teamcity commented Feb 10, 2021

cockroach-teamcity commented Feb 18, 2021

cockroach-teamcity commented Feb 27, 2021

cockroach-teamcity commented Mar 16, 2021

cockroach-teamcity commented Mar 24, 2021

cockroach-teamcity commented Mar 25, 2021

cockroach-teamcity commented Apr 7, 2021

cockroach-teamcity commented Apr 13, 2021

cockroach-teamcity commented Apr 14, 2021

tbg commented Apr 14, 2021

dt commented Apr 14, 2021

tbg commented Apr 14, 2021

dt commented Apr 14, 2021

adityamaru commented Apr 19, 2021

tbg commented Apr 19, 2021

tbg commented Apr 19, 2021 •

edited

Loading

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #59044

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #59044

Comments

cockroach-teamcity commented Jan 15, 2021

nvanbenschoten commented Jan 20, 2021

cockroach-teamcity commented Feb 10, 2021

cockroach-teamcity commented Feb 18, 2021

cockroach-teamcity commented Feb 27, 2021

cockroach-teamcity commented Mar 16, 2021

cockroach-teamcity commented Mar 24, 2021

cockroach-teamcity commented Mar 25, 2021

cockroach-teamcity commented Apr 7, 2021

cockroach-teamcity commented Apr 13, 2021

cockroach-teamcity commented Apr 14, 2021

tbg commented Apr 14, 2021

dt commented Apr 14, 2021

tbg commented Apr 14, 2021

dt commented Apr 14, 2021

adityamaru commented Apr 19, 2021

tbg commented Apr 19, 2021

tbg commented Apr 19, 2021 • edited Loading

tbg commented Apr 19, 2021 •

edited

Loading