[MINOR] Fix logical timestamp issue#14315
Draft
linliu-code wants to merge 11 commits intoapache:branch-0.xfrom
Draft
[MINOR] Fix logical timestamp issue#14315linliu-code wants to merge 11 commits intoapache:branch-0.xfrom
linliu-code wants to merge 11 commits intoapache:branch-0.xfrom
Conversation
linliu-code
force-pushed
the
branch-0.x
branch
2 times, most recently
from
1905a37 to
3efea4c
Compare
linliu-code
force-pushed
the
branch-0.x
branch
from
3efea4c to
c817b81
Compare
linliu-code
force-pushed
the
branch-0.x
branch
from
f94ef98 to
9ab9a1e
Compare
linliu-code
force-pushed
the
branch-0.x
branch
from
1c14290 to
b9c9333
Compare
linliu-code
force-pushed
the
branch-0.x
branch
from
0818bd3 to
4850db0
Compare
linliu-code
changed the title
linliu-code
commented
Nov 24, 2025
| public abstract MessageType convert(Schema schema); | ||
|
|
||
| public abstract Schema convert(MessageType schema); | ||
| } No newline at end of file |
Collaborator
Author
There was a problem hiding this comment.
Need to add the extra empty line.
linliu-code
commented
Nov 24, 2025
| } | ||
|
|
||
| private void logicalAssertions(Schema tableSchema, String tableBasePath, Map<String, String> hudiOpts, int tableVersion) { | ||
| if (tableVersion > 8) { |
Collaborator
Author
There was a problem hiding this comment.
We could remove these >8 statements since they will not be executed at all.
linliu-code
force-pushed
the
branch-0.x
branch
from
4850db0 to
5d8d587
Compare
linliu-code
changed the title
linliu-code
changed the title
linliu-code
changed the title
linliu-code
changed the title
linliu-code
force-pushed
the
branch-0.x
branch
from
579964e to
c620f31
Compare
linliu-code
force-pushed
the
branch-0.x
branch
from
8293394 to
9e5dab1
Compare
linliu-code
force-pushed
the
branch-0.x
branch
from
066895f to
5ad7baf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Logs
In PR #9743, the InternalSchema has only one
timestamplogical type that representstimestamp-micros, such that the logical type system was unable to represent columns that should be of logical typetimestamp-millis. That means, when a column is oftimestamp-millistype, and its value was treated as microseconds, which caused data corruptions.PR #13711 introduced a more complete logical type system to handle schema evolution for column stats, which fixes the logical timestamp issue by introducing multiple logical timestamp types, like
timestamp-millis,timestamp-micros,local-timestamp-millisandlocal-timestamp-micros.For branch-0.x, we aim to fix the logical timestamp issue instead of the entire logical systems introduced in #13711.
In this PR, we fix the logical timestamp related issue by picking the relevant changes from #13711, and excluding irrelevant changes from this PR.
Impact
Medium.
Risk level (write none, low medium or high below)
Medium.
Documentation Update
Contributor's checklist