Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: engine/switch/nodes=3 failed #55361

Closed
cockroach-teamcity opened this issue Oct 9, 2020 · 2 comments · Fixed by #55388
Closed

roachtest: engine/switch/nodes=3 failed #55361

cockroach-teamcity opened this issue Oct 9, 2020 · 2 comments · Fixed by #55388
Assignees
@jbowens
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Milestone
20.2

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).engine/switch/nodes=3 failed on master@7f02a7ac0949f226d617955337965f681706bace:

		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/engine_switch.go:132
		  | main.registerEngineSwitch.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/engine_switch.go:146
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2698
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2612
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	cluster.go:1657,context.go:135,cluster.go:1646,test_runner.go:836: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2350967-1602224087-03-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		3: 5252
		2: 5236
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
More

Artifacts: /engine/switch/nodes=3

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Oct 9, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.2 milestone Oct 9, 2020
@petermattis
Copy link
Collaborator 

   | ERROR: failed to start server: failed to create engines: pebble: error when replaying WAL: pebble/record: invalid chunk

@jbowens Seems possibly related to your recent changes in this area. Did this run have all of your fixes?

@petermattis petermattis mentioned this issue Oct 9, 2020
Closed
@jbowens jbowens mentioned this issue Oct 9, 2020
Merged
@jbowens
Copy link
Collaborator

jbowens commented Oct 9, 2020

@jbowens Seems possibly related to your recent changes in this area. Did this run have all of your fixes?

The latest fix isn't in the master branch (just the 20.2 release branch) yet. I created #55388 to pull it in.

craig bot pushed a commit that referenced this issue Oct 9, 2020
@knz @jbowens @mgartner
Merge #55371 #55388 #55391
f5bd55d 
55371: server,ui: Fix authentication bugs and inconsistencies r=dhartunian,bdarnell a=knz

Fixes #51781
Fixes #55370

This fixes two bugs. The one of the two that is a sec vuln only exists in 20.2.


55388: vendor: bump Pebble to b79d619f4761 r=jbowens a=jbowens

```
b79d619f sstable: skip readahead ramp-up for compactions
ab86f22f db: fix L0->LBase input expansion
879f3bfe db: changes to make iterators respect the bounds specified by the caller
91e8175b db: fix L0->LBase compaction picking bug with shared largest user keys
4e0eb739 db: fix WAL checksum error handling from previous versions
a69a94e1 db: optimize levelIter for repeated seeks with narrow and monotonic bounds
d670ff50 metamorphic: enhance iterSetBounds and position iter when setting bounds
b5c5df03 *: Add a simple test for forced compactions
```

Fix #55361.
Fix #54729.

Release note: None

55391: opt: do not mutate GenerateConstrainedScans input filters r=RaduBerinde a=mgartner

This commit fixes a bug in GenerateConstrainedScans which can cause
incorrect query results in tables with a partial index. Filters reduced
while proving implication of a partial index would mutate the input
filters. These filters would remain mutated for future iterations over
other indexes, causing constrained scans to be created without these
necessary filters.

Fixes #55387

Release justification: This fixes a critical correctness bug for a new
feature, partial indexes.

Release note (bug fix): A bug was fixed that caused incorrect query
results on tables with partial indexes. This bug did not affect any
queries involving tables without partial indexes.

Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net>
Co-authored-by: Jackson Owens <jackson@cockroachlabs.com>
Co-authored-by: Marcus Gartner <marcus@cockroachlabs.com>
@craig craig bot closed this as completed in f5571a4 Oct 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jbowens jbowens

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Projects
None yet
Milestone
20.2
Development

Successfully merging a pull request may close this issue.

vendor: bump Pebble to b79d619f4761 jbowens/cockroach
3 participants
@jbowens @petermattis @cockroach-teamcity