HIVE-29328: Orc acid footer metadata should be case insensitive #6203
+53
−10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
InvisibleProgrammer
force-pushed
the
orc_footer_fix
branch
from
1e79fc9 to
f445614
Compare
asf-ci-hive
added
tests pending
tests passed
and removed
tests passed
tests pending
labels
kasakrisz
reviewed
Nov 19, 2025
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestFixAcidKeyIndex.java
Outdated
Show resolved
Hide resolved
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestFixAcidKeyIndex.java
Outdated
Show resolved
Hide resolved
InvisibleProgrammer
force-pushed
the
orc_footer_fix
branch
from
f445614 to
e83ce6e
Compare
InvisibleProgrammer
force-pushed
the
orc_footer_fix
branch
from
e83ce6e to
82d1fa1
Compare
asf-ci-hive
added
tests failed
tests pending
and removed
tests pending
tests failed
labels
|
kasakrisz
approved these changes
Nov 24, 2025
Contributor
Author
|
Please wait an additional day merging in - our downstream test had a result that I want to double-check: There is a chance that collections4 only comes as a transitive dependency, and our downstream cluster didn't find the collections4 jar file. I want to check if I need to add this as an explicit dependency to hive-exec. |
Contributor
Author
|
Double tested. We don't need to add extra dependencies. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Orc file format allows metadata field names with ignoring casing. For example, we have manual tests when query based compaction creates Orc files with lowercase fields (the root cause for this is under unvestigation).
As OrcInputFormat.isOriginal currently checks the field names with strict casing, FixAcidKeyIndex can fail if the Orc file footer contains its metadata fields (like currentTransaction) lowercasing.
What changes were proposed in this pull request?
OrcInputFormat.isOriginal now does its checking with ignoring casing.
Why are the changes needed?
To be able to fix wrong Orc ACID files even with lower cased metadata.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added new test case:
TestFixAcidKeyIndex#testValidKeyIndex_withAcidMetadataLowerCase