Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use OutputValidator in SstFileWriter #7599

Open
ajkr opened this issue Oct 26, 2020 · 7 comments
Open

use OutputValidator in SstFileWriter #7599

ajkr opened this issue Oct 26, 2020 · 7 comments
Assignees

Comments

@ajkr
Copy link
Contributor

ajkr commented Oct 26, 2020

#7134 and #7467 add kv checksum and key order verification to files as they're created via OutputValidator. The verifications currently apply to files generated by flush and compaction. It'd be good to introduce these checks for SstFileWriter-generated files as well.

@riversand963
Copy link
Contributor

We should bump this issue up so that it's not buried in the bottom :)

@briantkim93
Copy link

Guys, I'm taking this up as my ramp up task into the project :)

@ajkr ajkr removed the up-for-grabs Up for grabs label Jan 13, 2022
@briantkim93
Copy link

briantkim93 commented May 7, 2022

Hey @ajkr I've been blocked on this for a while. I posted a question on the rocksdb google group but there's been nothing so far. Or ideally what's the right forum for me if I'm blocked?

@ajkr
Copy link
Contributor Author

ajkr commented May 8, 2022

Sorry I have not checked the google group. Let me try to catch up on the current PR and see if I can help sometime this coming week. This issue or the PR are probably the best forums although the unblocking time can still be slow as you found.. :/

@briantkim93
Copy link

@ajkr The code is here - main...briantkim93:add_outputvalidator_to_sstfilewrites sst_file_writer.cc lines 375-401.
Also should we consider opening a slack channel where folks like can get unblocked faster?

@ajkr
Copy link
Contributor Author

ajkr commented May 17, 2022

Very sorry for the delay. It takes a while for anything to reach the front of my queue.

SstFileReader is one way to read back data written by an SstFileWriter. But it is a user API and therefore returns user keys, which exclude internal metadata (seqno+value type). Meanwhile, kv_validator is constructed from internal keys, which include internal metadata as the name implies. Can you try using a table reader to read back the internal keys? It can be constructed similarly to how we construct a table builder (search for r->ioptions.table_factory->NewTableBuilder in sst_file_writer.cc).

Also should we consider opening a slack channel where folks like can get unblocked faster?

We had a slack channel but - at least for me - the difficulty was more about context switching than communication medium. That channel is gone now and I slightly prefer keeping code discussion centralized on GitHub.

@briantkim93
Copy link

Hey man, thanks a lot for your help. I raised the PR here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants