Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release-20.1: kvserver: sync cluster version setting to store #55503

Closed
wants to merge 1 commit into from

Conversation

tbg
Copy link
Member

@tbg tbg commented Oct 13, 2020

Backport 1/1 commits from #55240.

/cc @cockroachdb/release


Writes to a storage.Engine are not sync'ed by default, meaning that
they can get lost due to an ill-timed crash.

Fixes #54906.

(The backport will take care of #54908).

Release note (bug fix): a rare scenario in which a node would refuse
to start after updating the binary was fixed. The log message would
indicate: "store [...], last used with cockroach version [...], is too
old for running version [...] (which requires data from [...] or
later)".

Writes to a RocksDB `storage.Engine` were not sync'ed by default,
meaning that they can get lost due to an ill-timed crash. They are now,
matching pebble's behavior. This affects only WriteClusterVersion,
updateBootstrapInfoLocked, WriteLastUpTimestamp, and Compactor.Suggest,
nonw of which are performance sensitive.

Fixes cockroachdb#54906.

(The backport will take care of cockroachdb#54908).

Release note (bug fix): a rare scenario in which a node would refuse
to start after updating the binary was fixed. The log message would
indicate: "store [...], last used with cockroach version [...], is too
old for running version [...] (which requires data from [...] or
later)".
@tbg tbg requested a review from petermattis October 13, 2020 14:07
@cockroach-teamcity
Copy link
Member

This change is Reviewable

Copy link
Collaborator

@petermattis petermattis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @petermattis)

@tbg
Copy link
Member Author

tbg commented Oct 14, 2020

@petermattis

 --- FAIL: TestRemoveDeadReplicas/2/4/r=4 (10.44s)
        debug_test.go:292: could not open rocksdb instance: Corruption: Sequence numbers in WAL not in increasing order: 8759 < 8767

@jordanlewis

------- Stdout: -------
=== RUN   TestLogic/local/prepare/24578
rewrote:
CREATE TABLE xy (x INT8 PRIMARY KEY, y INT8, FAMILY (y), FAMILY (x));
CREATE TABLE ab (a INT8 PRIMARY KEY, b INT8, FAMILY (a, b));
INSERT INTO ab VALUES (1, 1), (2, 2);


--- total progress: 7508 statements/queries
            --- FAIL: TestLogic/local/prepare/24578 (12.74s)
                logic.go:1854: 
                     
                    testdata/logic_test/prepare:913: EXECUTE pg_catalog_query
                    expected:
                        seqrelid  seqtypid  seqstart  seqincrement  seqmax               seqmin  seqcache  seqcycle
                        67        20        1         1             9223372036854775807  1       1         false
                        
                    but found (query options: "colnames") :
                        seqrelid  seqtypid  seqstart  seqincrement  seqmax               seqmin  seqcache  seqcycle
                        68        20        1         1             9223372036854775807  1       1         false
                        
                    
                logic.go:1412: 
                     testdata/logic_test/prepare:919: too many errors encountered, skipping the rest of the input

@tbg
Copy link
Member Author

tbg commented Oct 14, 2020

@petermattis all of these

image

are also:

E201013 14:27:53.425255 1 cli/error.go:373 ERROR: cockroach server exited with error: failed to create engines: could not open rocksdb instance: Corruption: Sequence numbers in WAL not in increasing order: 1813 < 1818

@tbg
Copy link
Member Author

tbg commented Nov 3, 2020

#55746

@tbg tbg closed this Nov 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants