Support switching on / off UDT together with in-Memtable-only feature #11623
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jowlyzhang
force-pushed
the
toggle_udt
branch
2 times, most recently
from
f9b0ed8
to
3908dc5
Compare
|
@jowlyzhang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
ltamasi
approved these changes
Jul 25, 2023
|
@jowlyzhang has updated the pull request. You must reimport the pull request before landing. |
|
@jowlyzhang has updated the pull request. You must reimport the pull request before landing. |
|
@jowlyzhang has updated the pull request. You must reimport the pull request before landing. |
|
@jowlyzhang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@jowlyzhang merged this pull request in c24ef26. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add support to allow enabling / disabling user-defined timestamps feature for an existing column family in combination with the in-Memtable only feature.
To do this, this PR includes:
Log the
persist_user_defined_timestampsoption per column family in Manifest to facilitate detecting an attempt to enable / disable UDT. This entry is enforced to be logged in the same VersionEdit as the user comparator name entry.User-defined timestamps related options are validated when re-opening a column family, including user comparator name and the
persist_user_defined_timestampsflag. These type of settings and settings change are considered valid:a) no user comparator change and no effective
persist_user_defined_timestampflag change.b) switch user comparator to enable UDT provided the immediately effective
persist_user_defined_timestampsflagis false.
c) switch user comparator to disable UDT provided that the before-change
persist_user_defined_timestampsisalready false.
when an attempt to enable UDT is detected, we mark all its existing SST files as "having no UDT" by marking its
FileMetaData.user_defined_timestamps_persistedflag to false and handle their file boundariesFileMetaData.smallest,FileMetaData.largestby padding a min timestamp.while enabling / disabling UDT feature, timestamp size inconsistency in existing WAL logs are handled to make it compatible with the running user comparator.
Test plan: