Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Export Import sst files #3822

Closed
wants to merge 3 commits into from
Closed

Export Import sst files #3822

wants to merge 3 commits into from

Conversation

vpallipadi
Copy link

@vpallipadi vpallipadi commented May 7, 2018 •
edited
Loading

This is a review request for code change needed for - #3469
"Add support for taking snapshot of a column family and creating column family from a given CF snapshot"

We have an implementation for this that we have been testing internally. We have two new APIs that together provide this functionality.

(1) ExportColumnFamilyFiles() - This API is modelled after CreateCheckpoint() as below.
// Gets the live set of SST files for a particular Column Family in
// export_tables_dir and metadata about the files in metadata_vec.
// The SST files will be created through hard links when the directory is in
// the same partition as the db, copied otherwise.
// The directory should not already exist and will be created by this API.
// The directory will be an absolute path. Triggers a flush.
virtual Status ExportColumnFamilyFiles(
ColumnFamilyHandle* column_family,
std::vector* metadata_vec,
const std::string& export_files_dir);

Internally, the API will DisableFileDeletions(), GetColumnFamilyMetaData(), Parse through
metadata, creating links/copies of all the sst files, EnableFileDeletions() and complete the call by
returning the list of file metadata.

(2) ImportExternalFile() - This API is modelled after IngestExternalFile() as below.
// ImportExternalFile() will import external SST files into the specified
// column family.
//
// (1) External SST files can be created using SstFileWriter.
// (2) External SST files can be exported from a particular column family in
// an existing DB.
virtual Status ImportExternalFile(
ColumnFamilyHandle* column_family,
const std::vector& external_file_metadata,
const ImportExternalFileOptions& options);

Internally, this API parses all the sst files and adds it to the specified column family, at the same
level and with same sequence number as in the metadata. Also performs safety checks with respect to
overlaps between the sst files being imported or between existing data and new files being imported.

If incoming sequence number is higher than current local sequence number, local sequence
number is updated to reflect this.

Note, as the sst files is are being moved across Column Families, Column Family name in sst file
will no longer match the actual column family on destination DB. The API does not modify Column
Family name or id in the sst files being imported.

Venki Pallipadi added 3 commits May 7, 2018 15:58
ExportColumnFamilyTables API
daedc82 
Introduce a new API ExportColumnFamilyTables to export all the live sst
files of a particular column family and metadata information to be used
to import these sst files.

This is part 1 of the change for
#3469
" Add support for taking snapshot of a CF and creating a CF from a given
  CF snapshot"

Partially Solves #3469
ImportExternalFile API
1bf6848 
This change adds support to import external sst files from one column
family, as is, onto another column family. The files are imported at the
same level and with same sequence number as at the source.

This is part 2 of the change for
#3469
" Add support for taking snapshot of a CF and creating a CF from a given
  CF snapshot"

Partially Solves #3469
Tests for ExportColumnFamilyTables and ImportExternalFile APIs
5009fe0 
Add new unit tests for ExportColumnFamilyTables() and
ImportExternalFile() APIs.

This is part 3 of the change for
#3469
" Add support for taking snapshot of a CF and creating a CF from a given
  CF snapshot"

Partially Solves #3469
@vpallipadi vpallipadi mentioned this pull request May 8, 2018
Closed
@siying
Copy link
Contributor

siying commented May 8, 2018

Thank you for your contribution!
Can you explain why do we need a new Import() API? Can user create a column family and use the existing bulk loading API to load the file into it?

@vpallipadi
Copy link
Author

vpallipadi commented May 8, 2018 via email

@vpallipadi
Copy link
Author

vpallipadi commented May 8, 2018 via email

@vpallipadi
Copy link
Author

@siying

Thanks for reviewing this. Do you have any comments?
If there is a clean way of implementing this over existing Ingest API, will be happy to go that way. Infact the version that we started with was using the same API. But, there were too many if/else in the ingest path that made the code cryptic and we ended up with this alternate API approach.

Thanks.

@siying
Copy link
Contributor

siying commented May 25, 2018

Sorry for the delay. It makes sense. We'll take another look.

@vpallipadi
Copy link
Author

vpallipadi commented Jun 15, 2018 via email

@vpallipadi
Copy link
Author

@siying - Any update on this. We are thinking of having this feature in one of our upcoming releases and want to get blessing on the general idea. This has been working ok in our tests for a while.
Let us know anyway we can help. If you want to get on a phone call some time to discuss this, we would be happy to.
Copying @sachja.

@vpallipadi
Copy link
Author

Any feedback on this @siying ?

@sachja
Copy link

sachja commented Aug 10, 2018

@siying Would appreciate if you or someone in rocksdb team can look at this PR. We want to use this to speed up the time for reshuffling data when nodes are added/removed in our distributed key/value store implementation.
We have been testing this internally and not seen any issues but want to be sure this will be compatible with future rocksdb changes.

Specifically pls note our proposal above for updating the sequence number and to ingest sstables of a different column family without updating the column family ID.

@ajkr
Copy link
Contributor

ajkr commented Aug 14, 2018

Do you mind sharing more details about your use case? In particular, are you planning to export all files from a column family and import all of those files to a column family in a different DB? Also, will the target column family be empty before the import happens, or it'll have existing data?

@sachja
Copy link

sachja commented Aug 15, 2018 via email

@ajkr
Copy link
Contributor

ajkr commented Aug 15, 2018

Thanks @sachja. Have you considered using multiple DB instances rather than multiple column families? Or are there limitations preventing it (e.g., there's no way to guarantee consistency across DB instances)?

I will bring this up in our team meeting next week. It seems a reasonable feature to me. One thing I'm thinking about is whether we should have a more specific API, like ExportColumnFamily and ImportColumnFamily, that only allows full CFs to be exported/imported. That's because the API ImportExternalFile is a bit confusing with IngestExternalFile, and also it sounds like we don't need the API to be so general.

@sachja
Copy link

sachja commented Aug 15, 2018 via email

@vpallipadi
Copy link
Author

vpallipadi commented Sep 22, 2018 via email

@vpallipadi
Copy link
Author

@siying and @ajkr - If you guys are going to be there at rocksdb meetup tomorrow, can we talk about this change for few minutes. Me and @sachja will be there.
Thanks.

@vpallipadi vpallipadi mentioned this pull request Apr 2, 2019
Closed
@siying
Copy link
Contributor

siying commented Sep 9, 2019

I assume #5495 is merged, so we can close this one. Let me know if I misread them.

@siying siying closed this Sep 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@vpallipadi @siying @sachja @ajkr @facebook-github-bot