Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: drop/tpcc/w=100,nodes=9 failed #55541

Closed
cockroach-teamcity opened this issue Oct 14, 2020 · 2 comments · Fixed by #55982
Closed

roachtest: drop/tpcc/w=100,nodes=9 failed #55541

cockroach-teamcity opened this issue Oct 14, 2020 · 2 comments · Fixed by #55982
Assignees
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).drop/tpcc/w=100,nodes=9 failed on release-19.2@3e9adba8b62663b9e3521faad03a67f91fb3fc7a:

		  | I201014 06:45:40.424368 76 ccl/workloadccl/fixture.go:583  Restoring from gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/stock
		  | I201014 06:45:40.424273 73 ccl/workloadccl/fixture.go:583  Restoring from gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/order
		  | I201014 06:45:40.424522 70 ccl/workloadccl/fixture.go:583  Restoring from gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/district
		  | I201014 06:45:40.424286 72 ccl/workloadccl/fixture.go:583  Restoring from gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/history
		  | I201014 06:45:40.424692 75 ccl/workloadccl/fixture.go:583  Restoring from gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/item
		  | Error: restoring fixture: backup: gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/district: pq: failed to read backup descriptor: storage: object doesn't exist
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload fixtures load tpcc --warehouses=100 --db tpcc {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2654,drop.go:158,drop.go:179,test_runner.go:755: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2642
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2650
		  | main.registerDrop.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:158
		  | main.registerDrop.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:179
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2698
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2612
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /drop/tpcc/w=100,nodes=9

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity added branch-release-19.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Oct 14, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.2 milestone Oct 14, 2020
@cockroach-teamcity
Copy link
Member Author

(roachtest).drop/tpcc/w=100,nodes=9 failed on release-19.2@2ecba8f9c68942319b8e6504bcafeece553b90b4:

		  | I201015 06:42:28.126184 82 ccl/workloadccl/fixture.go:583  Restoring from gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/order
		  | I201015 06:42:28.126211 47 ccl/workloadccl/fixture.go:583  Restoring from gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/district
		  | I201015 06:42:28.126178 85 ccl/workloadccl/fixture.go:583  Restoring from gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/stock
		  | I201015 06:42:28.126202 48 ccl/workloadccl/fixture.go:583  Restoring from gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/customer
		  | I201015 06:42:28.126240 46 ccl/workloadccl/fixture.go:583  Restoring from gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/warehouse
		  | Error: restoring fixture: backup: gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/item: pq: failed to read backup descriptor: storage: object doesn't exist
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload fixtures load tpcc --warehouses=100 --db tpcc {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2654,drop.go:158,drop.go:179,test_runner.go:755: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2642
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2650
		  | main.registerDrop.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:158
		  | main.registerDrop.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/drop.go:179
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2698
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2612
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /drop/tpcc/w=100,nodes=9

See this test on roachdash
powered by pkg/cmd/internal/issues

@tbg
Copy link
Member

tbg commented Oct 20, 2020

	  | Error: restoring fixture: backup: gs://cockroach-fixtures/workload/tpcc/version=2.2.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100/item: pq: failed to read backup descriptor: storage: object doesn't exist

cc @nvanbenschoten

craig bot pushed a commit that referenced this issue Oct 27, 2020
55877: rowflow: account for more memory usage of the row buffer r=yuzefovich a=yuzefovich

Previously, we accounted for the memory usage of the row buffer from
which the rows are pushed from in the routers only when copying the rows
from the row container, but we also have another in-memory row buffer
that we can move the rows from which was missing the memory accounting.
This is now fixed which also deflakes the router disk spill test.

Fixes: #55848.

Release note: None

55907: sql: remove vectorize=201auto option r=yuzefovich a=yuzefovich

`201auto` option of `vectorize` setting has been removed since it no
longer makes sense to keep (it was introduced as an escape hatch for
20.2 release). Note that we don't need to bump the distsql version
because of this change since it is backwards compatible - if the gateway
is running the old version that has `vectorize=201auto` set, then we
will check whether flows for all nodes don't have non-streaming
operators and possibly use `off` option on the flow setup request, then
the newer version remote node will check the vectorize setting on the
request whether it is not `off` and setup the vectorized flow if it is
not.

Release note (sql change): `201auto` value for `vectorize` session
variable and the corresponding cluster setting has been removed.

55982: roachtest: use IMPORT for all TPC-C tests r=nvanbenschoten a=nvanbenschoten

Fixes #55541.
Fixes #55580.

In #54880 and #55050, we switched to using IMPORT for most TPC-C tests,
in favor of BACKUP, which requires fixtures to be regenerated whenever
we changed the workload. In this PR, we switch over the remaining uses
of `fixtures load tpcc` to `fixtures import tpcc` to avoid compatibility
issues on older release branches.

55994: settings: delete StateMachineSetting, introduce VersionSetting r=irfansharif a=irfansharif

We introduced the custom StateMachineSetting type in #17216 to power the
underlying machinery for version upgrades:

  SET CLUSTER SETTING version = <major>-<minor>;

At the time we left it generalizable enough to make room for future
settings with arbitrary internal transitions. For cluster versions this
meant only allowing transitions from one major version to the immediate
next, all the while making sure that each binary in the cluster was able
to activate the targeted version (the proposed version fell within the
allowable range ["binary min supported version", "binary version"]).

In the three years since we haven't added any state-machine validated
settings that fit the bill, and the layers of abstractions to get to
validated cluster version updates are one too many for (my) comfort.
This PR introduces a new VersionSetting type that is more tightly
coupled with the ClusterVersion type itself.

VersionSetting uses language that only appropriate for cluster versions,
hopefully clarifying things as we go. We'll depend on this clarification
in future PRs when we remove the use of gossip in disseminating cluster
version bumps.

Release note (sql/cli change): The underlying type for the version
cluster setting has been changed. Previously it was of an internal type
representing "state machine", but now it's simply "version". This has no
operational implications, but it does reflect differently in a few
spots:
  - The `Type` column in `cockroach gen settings-list` will now show
    "version" instead of "custom validation"
  - The `setting_type` column for `version` in `SHOW CLUSTER SETTINGS`
    will now show a "v" instead of an "m"
  - The `valueType` column for `version` in `system.settings` will now
    show a "v" instead of an "m"


56006: logictest: deflake a test r=yuzefovich a=yuzefovich

We've recently merged a test that in very rare circumstances could
produce a float result that differs from the expected one by 1 in the
15th significant digit (after rounding). I believe that could occur,
e.g. when the 15th and 16th significant digits were `35`, and we matched
the spec of supporting 15 significant digits for floats, yet the
rounding makes us return an unexpected result. This commit rounds to the
precision of 1 digit less which should make the test non-flaky.

Release note: None

Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Co-authored-by: irfan sharif <irfanmahmoudsharif@gmail.com>
@craig craig bot closed this as completed in 71ea818 Oct 27, 2020
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Nov 9, 2020
Fixes cockroachdb#55541.
Fixes cockroachdb#55580.

In cockroachdb#54880 and cockroachdb#55050, we switched to using IMPORT for most TPC-C tests,
in favor of BACKUP, which requires fixtures to be regenerated whenever
we changed the workload. In this PR, we switch over the remaining uses
of `fixtures load tpcc` to `fixtures import tpcc` to avoid compatibility
issues on older release branches.
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 1, 2020
Fixes cockroachdb#55541.
Fixes cockroachdb#55580.

In cockroachdb#54880 and cockroachdb#55050, we switched to using IMPORT for most TPC-C tests,
in favor of BACKUP, which requires fixtures to be regenerated whenever
we changed the workload. In this PR, we switch over the remaining uses
of `fixtures load tpcc` to `fixtures import tpcc` to avoid compatibility
issues on older release branches.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants