roachtest: tpccbench/nodes=3/cpu=4 failed #54822

cockroach-teamcity · 2020-09-26T06:11:24Z

(roachtest).tpccbench/nodes=3/cpu=4 failed on release-20.1@a84063ed9e7e3c43c710534629399bb957f17cca:

	cluster.go:2651,tpcc.go:787,tpcc.go:622,test_runner.go:755: monitor failure: monitor task failed: output in run_061104.872_n4_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2324598-1601100222-14-n4cpu4:4 -- ./workload fixtures load tpcc --warehouses=1000 --scatter --checks=false {pgurl:1} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2639
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2647
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:787
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:622
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2695
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2287
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:689
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:785
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2629
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) output in run_061104.872_n4_workload_fixtures_load_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2324598-1601100222-14-n4cpu4:4 -- ./workload fixtures load tpcc --warehouses=1000 --scatter --checks=false {pgurl:1} returned
		  | stderr:
		  | I200926 06:11:06.783882 1 ccl/workloadccl/cliccl/fixtures.go:284  starting restore of 9 tables
		  | Error: restoring fixture: pq: storage: object doesn't exist
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ```
		  |   | ./workload fixtures load tpcc --warehouses=1000 --scatter --checks=false {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *main.withCommandDetails (8) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=4
Related:

roachtest: tpccbench/nodes=3/cpu=4 failed #54802 roachtest: tpccbench/nodes=3/cpu=4 failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

irfansharif · 2020-09-28T14:31:19Z

restoring fixture: pq: storage: object doesn't exist

nvanbenschoten · 2020-09-28T14:34:29Z

Fallout from #54772?

@RaduBerinde I stop using fixtures entirely for tpccbench in #53983. It seems like that would save a lot of work here. Should we just push that forward?

RaduBerinde · 2020-09-28T14:46:43Z

That change wasn't on any of the release branches yet.. Do we run workload from master against older versions?

dt · 2020-09-28T15:15:36Z

@RaduBerinde yeah, I think master's roachtest/workload are what are used on all the release branches (there's a lot of slack history back-and-forth on this I think)

irfansharif · 2020-09-28T16:02:45Z

Do we run workload from master against older versions?

Yes, sigh. #51897 pls

RaduBerinde · 2020-09-28T16:30:54Z

I think I see what's going on. tpccbench passes the --deprecated-fk-indexes flag for older versions; I did not regenerate the fixtures with that flag. I can start doing that today, but I agree with @nvanbenschoten we should just switch to always using import.

RaduBerinde · 2020-09-28T16:37:38Z

Hm, though I don't know why that flag wouldn't show up in this log:

		  | Wraps: (2) Node 4. Command with error:
		  |   | ```
		  |   | ./workload fixtures load tpcc --warehouses=1000 --scatter --checks=false {pgurl:1}
		  |   | ```

RaduBerinde · 2020-09-28T17:09:09Z

I think I figured it out. The fixture exists (or we would be getting a "cannot find fixture" error). The error comes from loading a backup which doesn't have the FK indexes, required by 20.1. I will try to figure out where the error comes from and if we can make it less opaque.

The problem is that we are not passing the --deprecated-fk-indexes when loading (we only pass it when importing). I will fix that and generate fixtures with that flag.

RaduBerinde · 2020-09-28T17:20:13Z

I wasn't quite right. The error is indeed coming from google cloud storage. The old version is looking for a BACKUP file inside the backup folder, and that doesn't exist (the backup contains files like BACKUP-MANIFEST, BACKUP-STATISTICS etc).

RaduBerinde · 2020-09-28T17:31:34Z

This means that I can't generate new backups that will be restorable from 20.1 or earlier :/

I think we should move forward with #51897 and/or #53983 to fix this.

cockroach-teamcity · 2020-09-29T06:13:17Z

(roachtest).tpccbench/nodes=3/cpu=4 failed on release-20.1@5a4e41010f8faa095c88e2e5b142e0b6880c9f44:

	cluster.go:2651,tpcc.go:764,tpcc.go:599,test_runner.go:755: monitor failure: monitor task failed: output in run_061258.161_n4_workload_fixtures_import_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2328549-1601359472-12-n4cpu4:4 -- ./workload fixtures import tpcc --warehouses=1000 --scatter --checks=false {pgurl:1} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2639
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2647
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:764
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:599
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2695
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2287
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:666
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:762
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2629
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) output in run_061258.161_n4_workload_fixtures_import_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2328549-1601359472-12-n4cpu4:4 -- ./workload fixtures import tpcc --warehouses=1000 --scatter --checks=false {pgurl:1} returned
		  | stderr:
		  | I200929 06:12:59.652037 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | Error: importing fixture: importing table order: pq: expected tpcc version "2.2.0" but got "2.1.0"
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ```
		  |   | ./workload fixtures import tpcc --warehouses=1000 --scatter --checks=false {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *main.withCommandDetails (8) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=4
Related:

roachtest: tpccbench/nodes=3/cpu=4 failed #54802 roachtest: tpccbench/nodes=3/cpu=4 failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

RaduBerinde · 2020-09-29T14:21:01Z

I believe #54880 plus #54884 should have fixed this. I will keep the issue open until I can confirm.

cockroach-teamcity · 2020-09-30T06:15:27Z

(roachtest).tpccbench/nodes=3/cpu=4 failed on release-20.1@d89343cdb6dcd86584584b098889795620dfd534:

	cluster.go:2651,tpcc.go:764,tpcc.go:599,test_runner.go:755: monitor failure: monitor task failed: output in run_061501.991_n4_workload_fixtures_import_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2332347-1601446053-15-n4cpu4:4 -- ./workload fixtures import tpcc --warehouses=1000 --scatter --checks=false {pgurl:1} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2639
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2647
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:764
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:599
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2695
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2287
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:666
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:762
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2629
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) output in run_061501.991_n4_workload_fixtures_import_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2332347-1601446053-15-n4cpu4:4 -- ./workload fixtures import tpcc --warehouses=1000 --scatter --checks=false {pgurl:1} returned
		  | stderr:
		  | I200930 06:15:03.439049 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | Error: importing fixture: importing table warehouse: pq: expected tpcc version "2.2.0" but got "2.1.0"
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ```
		  |   | ./workload fixtures import tpcc --warehouses=1000 --scatter --checks=false {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *main.withCommandDetails (8) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=4
Related:

roachtest: tpccbench/nodes=3/cpu=4 failed #54802 roachtest: tpccbench/nodes=3/cpu=4 failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

54743: kvcoord: properly redact 'have been waiting' message in DistSender r=knz a=tbg Release note: None 55050: roachtest: use crdb's workload for fixtures import tpcc r=nvanbenschoten a=nvanbenschoten Fixes #55042. Fixes #55041. Fixes #55039. Fixes #55038. Fixes #55037. Fixes #55036. Fixes #55035. Fixes #55033. Fixes #55029. Fixes #55024. Fixes #55022. Fixes #55020. Fixes #55019. Fixes #55018. Fixes #55017. Fixes #55016. Fixes #55013. Fixes #55010. Fixes #55009. Fixes #55008. Fixes #55003. Fixes #55002. Fixes #54998. Fixes #54995. Fixes #54822. Fixes #52693. Fixes #54802. We were already doing this in some places, but needed it in others. Co-authored-by: Tobias Grieger <tobias.b.grieger@gmail.com> Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>

Fixes #55042. Fixes #55041. Fixes #55039. Fixes #55038. Fixes #55037. Fixes #55036. Fixes #55035. Fixes #55033. Fixes #55029. Fixes #55024. Fixes #55022. Fixes #55020. Fixes #55019. Fixes #55018. Fixes #55017. Fixes #55016. Fixes #55013. Fixes #55010. Fixes #55009. Fixes #55008. Fixes #55003. Fixes #55002. Fixes #54998. Fixes #54995. Fixes #54822. Fixes #52693. Fixes #54802. We were already doing this in some places, but needed it in others.

cockroach-teamcity added branch-release-20.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 26, 2020

cockroach-teamcity added this to the 20.2 milestone Sep 26, 2020

irfansharif assigned dt Sep 28, 2020

RaduBerinde mentioned this issue Sep 28, 2020

roachtest: drop/tpcc/w=100,nodes=9 failed #54830

Closed

dt mentioned this issue Sep 28, 2020

roachtest: tpccbench/nodes=3/cpu=16 failed #54833

Closed

RaduBerinde removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Sep 28, 2020

RaduBerinde assigned RaduBerinde and unassigned dt Sep 29, 2020

This was referenced Sep 29, 2020

roachtest: drop/tpcc/w=100,nodes=9 failed #54910

Closed

roachtest: cdc/tpcc-1000 failed #54909

Closed

roachtest: cdc/sink-chaos failed #54827

Closed

This was referenced Sep 29, 2020

roachtest: cdc/poller/rangefeed=false failed #54826

Closed

roachtest: cdc/initial-scan failed #54825

Closed

cockroach-teamcity mentioned this issue Sep 30, 2020

roachtest: tpccbench/nodes=3/cpu=4 failed #54802

Closed

nvanbenschoten mentioned this issue Sep 30, 2020

roachtest: use crdb's workload for fixtures import tpcc #55050

Merged

craig bot closed this as completed in #55050 Sep 30, 2020

craig bot closed this as completed in 20fc710 Sep 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: tpccbench/nodes=3/cpu=4 failed #54822

roachtest: tpccbench/nodes=3/cpu=4 failed #54822

cockroach-teamcity commented Sep 26, 2020

irfansharif commented Sep 28, 2020

nvanbenschoten commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

dt commented Sep 28, 2020

irfansharif commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

cockroach-teamcity commented Sep 29, 2020

RaduBerinde commented Sep 29, 2020

cockroach-teamcity commented Sep 30, 2020

roachtest: tpccbench/nodes=3/cpu=4 failed #54822

roachtest: tpccbench/nodes=3/cpu=4 failed #54822

Comments

cockroach-teamcity commented Sep 26, 2020

irfansharif commented Sep 28, 2020

nvanbenschoten commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

dt commented Sep 28, 2020

irfansharif commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

RaduBerinde commented Sep 28, 2020

cockroach-teamcity commented Sep 29, 2020

RaduBerinde commented Sep 29, 2020

cockroach-teamcity commented Sep 30, 2020