Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix dead loop with kSkipAnyCorruptedRecords mode selected in some cases (#11955) #11979

Closed
wants to merge 1 commit into from
Closed

Fix dead loop with kSkipAnyCorruptedRecords mode selected in some cases (#11955) #11979

wants to merge 1 commit into from

Conversation

qiuchengxuan
Copy link
Contributor

Summary:

With fragmented record span across multiple blocks, if any following blocks corrupted with arbitary data, and intepreted log number less than the current log number, program will fall into infinite loop due to
not skipping buffer leading bytes

Test Plan: existing unit tests

@facebook-github-bot
Copy link
Contributor

Hi @qiuchengxuan!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

@qiuchengxuan qiuchengxuan changed the title Fix potential dead loop with kSkipAnyCorruptedRecords mode selected Fix potential dead loop with kSkipAnyCorruptedRecords mode selected (#11955) Oct 18, 2023
@facebook-github-bot
Copy link
Contributor

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

@qiuchengxuan qiuchengxuan changed the title Fix potential dead loop with kSkipAnyCorruptedRecords mode selected (#11955) Fix dead loop with kSkipAnyCorruptedRecords mode selected in some cases (#11955) Oct 19, 2023
jowlyzhang
jowlyzhang reviewed Oct 23, 2023
View reviewed changes
Copy link
Contributor

@jowlyzhang jowlyzhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for finding this issue and the fix. Just curious, how did you notice this issue in the first place?

@jowlyzhang
Copy link
Contributor

Would you mind run make format and rebase, thanks!

@qiuchengxuan
Copy link
Contributor Author

qiuchengxuan commented Oct 24, 2023 •
edited
Loading

Thank you for finding this issue and the fix. Just curious, how did you notice this issue in the first place?

Partially memtioned in issue

We have a ceph cluster and several days ago encountered a power cut, and a lot OSDs refuses to start.
After investigation we figure out it's due to data corruption, which probably caused by ceph bluefs, so submitted an issue to ceph. The only thing we can do is to salvage as much data as we could so we use this mode to start ceph OSD, and encountered this issue.

@qiuchengxuan
Fix potential dead loop with kSkipAnyCorruptedRecords mode selected (#…
2866e87 
…11968)

Summary:

With fragmented record span across multiple blocks,
if any following blocks corrupted with arbitary data,
and intepreted type is recyclable type and log number
less than the current log number, program will fall into
infinite loop due to not skipping any buffer bytes

Test Plan: existing unit tests
@qiuchengxuan
Copy link
Contributor Author

Would you mind run make format and rebase, thanks!

Done

@facebook-github-bot
Copy link
Contributor

@jowlyzhang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@jowlyzhang merged this pull request in f2c9075.

@qiuchengxuan qiuchengxuan mentioned this pull request Oct 30, 2023
Closed
rockeet pushed a commit to topling/toplingdb that referenced this pull request Dec 23, 2023
@qiuchengxuan @rockeet
Fix dead loop with kSkipAnyCorruptedRecords mode selected in some cas…
f197b80 
…es (#11955) (#11979)

Summary:
With fragmented record span across multiple blocks, if any following blocks corrupted with arbitary data, and intepreted log number less than the current log number, program will fall into infinite loop due to
not skipping buffer leading bytes

Pull Request resolved: facebook/rocksdb#11979

Test Plan: existing unit tests

Reviewed By: ajkr

Differential Revision: D50604408

Pulled By: jowlyzhang

fbshipit-source-id: e50a0c7e7c3d293fb9d5afec0a3eb4a1835b7a3b
@igorcanadi igorcanadi mentioned this pull request Jan 17, 2024
Closed
rockeet pushed a commit to topling/toplingdb that referenced this pull request Sep 1, 2024
@qiuchengxuan @rockeet
Fix dead loop with kSkipAnyCorruptedRecords mode selected in some cas…
2b1af48 
…es (#11955) (#11979)

Summary:
With fragmented record span across multiple blocks, if any following blocks corrupted with arbitary data, and intepreted log number less than the current log number, program will fall into infinite loop due to
not skipping buffer leading bytes

Pull Request resolved: facebook/rocksdb#11979

Test Plan: existing unit tests

Reviewed By: ajkr

Differential Revision: D50604408

Pulled By: jowlyzhang

fbshipit-source-id: e50a0c7e7c3d293fb9d5afec0a3eb4a1835b7a3b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jowlyzhang jowlyzhang jowlyzhang approved these changes

Assignees
No one assigned
Labels
CLA Signed Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@qiuchengxuan @facebook-github-bot @jowlyzhang