-
Notifications
You must be signed in to change notification settings - Fork 3.6k
[fix](hive) Fix partition path scheme mismatch when inserting into Hive partitioned tables on object storage #57973
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ve partitioned tables on object storage
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
TPC-H: Total hot run time: 33962 ms |
TPC-DS: Total hot run time: 187549 ms |
ClickBench: Total hot run time: 27.05 s |
FE Regression Coverage ReportIncrement line coverage |
|
run buildall |
TPC-H: Total hot run time: 35043 ms |
TPC-DS: Total hot run time: 188173 ms |
ClickBench: Total hot run time: 27.53 s |
|
PR approved by at least one committer and no changes requested. |
FE Regression Coverage ReportIncrement line coverage |
…ve partitioned tables on object storage (#57973) ## Problem When inserting data into Hive partitioned tables stored on S3-compatible object storage (OSS/COS/OBS), the operation fails with authentication error because BE unifies all object storage under "s3://" scheme, but HMS expects the original scheme (e.g., "oss://"). The mismatch causes s3a FileSystem to access OSS endpoints with wrong credentials. ## Solution Changed `HMSTransaction.finishInsertTable()` line 277 to use `writePath` instead of `getTargetPath()`. The `writePath` variable already contains the correct original scheme from HMS, avoiding the scheme conversion issue. ## Test Added partition table insert tests for OSS/COS/OBS in `hive_on_hms_and_dlf.groovy`.
…ve partitioned tables on object storage (#57973) ## Problem When inserting data into Hive partitioned tables stored on S3-compatible object storage (OSS/COS/OBS), the operation fails with authentication error because BE unifies all object storage under "s3://" scheme, but HMS expects the original scheme (e.g., "oss://"). The mismatch causes s3a FileSystem to access OSS endpoints with wrong credentials. ## Solution Changed `HMSTransaction.finishInsertTable()` line 277 to use `writePath` instead of `getTargetPath()`. The `writePath` variable already contains the correct original scheme from HMS, avoiding the scheme conversion issue. ## Test Added partition table insert tests for OSS/COS/OBS in `hive_on_hms_and_dlf.groovy`.
Problem
When inserting data into Hive partitioned tables stored on S3-compatible object storage (OSS/COS/OBS), the operation fails with authentication error because
BE unifies all object storage under "s3://" scheme, but HMS expects the original scheme (e.g., "oss://"). The mismatch causes s3a FileSystem to access OSS
endpoints with wrong credentials.
Solution
Changed
HMSTransaction.finishInsertTable()line 277 to usewritePathinstead ofgetTargetPath(). ThewritePathvariable already contains the correctoriginal scheme from HMS, avoiding the scheme conversion issue.
Test
Added partition table insert tests for OSS/COS/OBS in
hive_on_hms_and_dlf.groovy.