Skip to content

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Nov 13, 2025

Problem

When inserting data into Hive partitioned tables stored on S3-compatible object storage (OSS/COS/OBS), the operation fails with authentication error because
BE unifies all object storage under "s3://" scheme, but HMS expects the original scheme (e.g., "oss://"). The mismatch causes s3a FileSystem to access OSS
endpoints with wrong credentials.

Solution

Changed HMSTransaction.finishInsertTable() line 277 to use writePath instead of getTargetPath(). The writePath variable already contains the correct
original scheme from HMS, avoiding the scheme conversion issue.

Test

Added partition table insert tests for OSS/COS/OBS in hive_on_hms_and_dlf.groovy.

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zy-kkk
Copy link
Member Author

zy-kkk commented Nov 13, 2025

run buildall

CalvinKirs
CalvinKirs previously approved these changes Nov 13, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 13, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 33962 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0c1bf8b283bb609fb639cde4b08e17369754a37b, data reload: false

------ Round 1 ----------------------------------
q1	17616	5164	5010	5010
q2	1995	312	204	204
q3	10299	1258	729	729
q4	10239	960	368	368
q5	7505	2387	2254	2254
q6	180	172	137	137
q7	919	757	637	637
q8	9350	1329	1086	1086
q9	6837	5012	5106	5012
q10	6916	2224	1808	1808
q11	500	286	276	276
q12	362	352	220	220
q13	17783	3698	3041	3041
q14	219	232	205	205
q15	587	497	524	497
q16	999	1003	932	932
q17	578	825	410	410
q18	7543	7174	7152	7152
q19	1325	939	549	549
q20	347	331	221	221
q21	3607	3155	2231	2231
q22	1053	1039	983	983
Total cold run time: 106759 ms
Total hot run time: 33962 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5131	5141	5081	5081
q2	251	320	228	228
q3	2199	2638	2290	2290
q4	1316	1728	1373	1373
q5	4144	4339	4365	4339
q6	220	173	131	131
q7	2097	1993	1798	1798
q8	2622	2583	2528	2528
q9	7231	7291	7242	7242
q10	3093	3262	2815	2815
q11	581	512	497	497
q12	668	779	843	779
q13	3411	3871	3235	3235
q14	275	300	294	294
q15	538	514	504	504
q16	1053	1122	1063	1063
q17	1166	1609	1505	1505
q18	7892	7670	7527	7527
q19	791	800	881	800
q20	2103	2099	1857	1857
q21	4693	4383	4297	4297
q22	1071	1042	1033	1033
Total cold run time: 52546 ms
Total hot run time: 51216 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187549 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0c1bf8b283bb609fb639cde4b08e17369754a37b, data reload: false

query1	1076	401	386	386
query2	6610	1752	1730	1730
query3	6764	229	225	225
query4	26127	23654	23245	23245
query5	4451	654	490	490
query6	404	257	239	239
query7	4641	535	303	303
query8	310	262	262	262
query9	8729	2628	2594	2594
query10	507	352	294	294
query11	15508	15049	14983	14983
query12	172	119	114	114
query13	1674	579	442	442
query14	10816	9237	9158	9158
query15	195	188	179	179
query16	7558	635	515	515
query17	1214	752	617	617
query18	2006	411	310	310
query19	212	207	182	182
query20	130	123	118	118
query21	221	137	113	113
query22	4089	4044	3986	3986
query23	33870	32866	33012	32866
query24	8447	2405	2442	2405
query25	661	547	500	500
query26	1235	301	163	163
query27	2742	517	361	361
query28	4421	2221	2212	2212
query29	857	637	519	519
query30	321	232	194	194
query31	897	805	752	752
query32	82	76	71	71
query33	626	386	349	349
query34	800	865	530	530
query35	824	846	742	742
query36	947	998	918	918
query37	138	107	90	90
query38	3520	3552	3414	3414
query39	1503	1425	1423	1423
query40	229	135	123	123
query41	69	65	65	65
query42	123	113	112	112
query43	489	503	471	471
query44	1234	755	745	745
query45	190	181	177	177
query46	894	987	644	644
query47	1728	1805	1715	1715
query48	404	436	326	326
query49	780	535	432	432
query50	682	690	431	431
query51	3966	3939	3861	3861
query52	110	112	101	101
query53	254	283	197	197
query54	322	301	340	301
query55	86	86	83	83
query56	328	302	312	302
query57	1221	1175	1142	1142
query58	292	276	270	270
query59	2570	2739	2541	2541
query60	350	343	326	326
query61	167	166	164	164
query62	813	719	659	659
query63	220	190	190	190
query64	4533	1155	864	864
query65	4004	3957	3919	3919
query66	1149	434	339	339
query67	15206	15062	14930	14930
query68	8377	940	602	602
query69	499	324	292	292
query70	1370	1278	1213	1213
query71	493	340	317	317
query72	5894	4862	4895	4862
query73	686	580	365	365
query74	8794	9147	8694	8694
query75	4008	3331	2786	2786
query76	3789	1150	743	743
query77	808	410	306	306
query78	9579	9665	8883	8883
query79	2049	818	595	595
query80	659	563	498	498
query81	505	259	233	233
query82	471	154	130	130
query83	265	271	263	263
query84	259	103	95	95
query85	941	496	443	443
query86	366	313	301	301
query87	3704	3710	3579	3579
query88	3815	2273	2251	2251
query89	390	330	286	286
query90	2015	221	220	220
query91	174	166	133	133
query92	87	70	66	66
query93	1733	998	646	646
query94	713	444	313	313
query95	409	323	316	316
query96	489	576	273	273
query97	2912	2948	2911	2911
query98	235	208	207	207
query99	1676	1435	1303	1303
Total cold run time: 275882 ms
Total hot run time: 187549 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 27.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0c1bf8b283bb609fb639cde4b08e17369754a37b, data reload: false

query1	0.05	0.05	0.06
query2	0.08	0.05	0.05
query3	0.26	0.08	0.09
query4	1.61	0.12	0.11
query5	0.27	0.25	0.24
query6	1.19	0.65	0.63
query7	0.03	0.03	0.03
query8	0.06	0.04	0.05
query9	0.58	0.52	0.52
query10	0.60	0.56	0.56
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.60	0.60
query14	1.00	0.99	0.99
query15	0.85	0.82	0.83
query16	0.39	0.39	0.39
query17	1.00	1.03	0.99
query18	0.21	0.19	0.20
query19	1.88	1.75	1.78
query20	0.02	0.01	0.01
query21	15.46	0.21	0.14
query22	4.92	0.07	0.05
query23	15.65	0.26	0.10
query24	2.48	0.56	0.25
query25	0.07	0.06	0.06
query26	0.14	0.13	0.13
query27	0.08	0.06	0.05
query28	3.91	1.13	0.94
query29	12.56	3.86	3.15
query30	0.29	0.13	0.11
query31	2.80	0.58	0.39
query32	3.23	0.55	0.47
query33	3.09	3.05	3.08
query34	15.85	5.13	4.53
query35	4.57	4.52	4.54
query36	0.66	0.50	0.49
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.12
query41	0.09	0.03	0.03
query42	0.04	0.04	0.02
query43	0.04	0.03	0.04
Total cold run time: 97.3 s
Total hot run time: 27.05 s

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 14, 2025
@zy-kkk
Copy link
Member Author

zy-kkk commented Nov 14, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 35043 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 987ac7aab8968480f3a51e54163ed4d98c0268a0, data reload: false

------ Round 1 ----------------------------------
q1	17634	5472	5233	5233
q2	2049	317	211	211
q3	10271	1380	743	743
q4	10229	992	373	373
q5	7547	2622	2502	2502
q6	212	181	138	138
q7	991	796	644	644
q8	9367	1486	1244	1244
q9	6923	5391	5289	5289
q10	6930	2278	1833	1833
q11	506	316	289	289
q12	337	405	235	235
q13	17838	3807	3055	3055
q14	237	250	216	216
q15	592	499	502	499
q16	1031	1008	942	942
q17	578	913	383	383
q18	7451	7272	7132	7132
q19	1150	1091	592	592
q20	356	364	230	230
q21	3929	3212	2295	2295
q22	1064	1028	965	965
Total cold run time: 107222 ms
Total hot run time: 35043 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5467	5422	5428	5422
q2	258	347	239	239
q3	2261	2771	2283	2283
q4	1404	1799	1361	1361
q5	4623	4504	4539	4504
q6	265	187	138	138
q7	2086	1973	1814	1814
q8	2896	2708	2849	2708
q9	7342	7263	7275	7263
q10	3217	3265	2854	2854
q11	644	543	524	524
q12	715	806	627	627
q13	3613	4191	3239	3239
q14	274	293	265	265
q15	544	510	502	502
q16	1138	1099	1049	1049
q17	1259	1717	1529	1529
q18	7890	7873	7615	7615
q19	880	840	877	840
q20	2016	2088	1818	1818
q21	4952	4492	4413	4413
q22	1124	1069	1002	1002
Total cold run time: 54868 ms
Total hot run time: 52009 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188173 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 987ac7aab8968480f3a51e54163ed4d98c0268a0, data reload: false

query1	1014	403	403	403
query2	6597	1724	1722	1722
query3	6742	227	221	221
query4	26693	23583	23322	23322
query5	4376	619	474	474
query6	348	253	233	233
query7	4651	495	303	303
query8	310	280	246	246
query9	8744	2590	2584	2584
query10	478	329	279	279
query11	15254	15183	14849	14849
query12	180	143	114	114
query13	1676	552	430	430
query14	10268	9149	9068	9068
query15	196	186	177	177
query16	7300	674	499	499
query17	1230	762	606	606
query18	1994	417	325	325
query19	230	197	177	177
query20	127	124	118	118
query21	214	134	113	113
query22	3919	4019	4241	4019
query23	33686	33032	33071	33032
query24	8439	2378	2403	2378
query25	590	504	445	445
query26	1250	280	157	157
query27	2768	493	347	347
query28	4436	2190	2185	2185
query29	819	613	483	483
query30	303	224	194	194
query31	913	861	733	733
query32	84	76	72	72
query33	600	374	323	323
query34	790	844	532	532
query35	839	854	738	738
query36	951	974	872	872
query37	122	106	85	85
query38	3525	3551	3456	3456
query39	1466	1399	1430	1399
query40	220	134	117	117
query41	67	62	60	60
query42	124	116	111	111
query43	494	496	472	472
query44	1221	751	749	749
query45	181	180	175	175
query46	877	986	636	636
query47	1765	1760	1757	1757
query48	386	429	334	334
query49	799	494	393	393
query50	647	685	402	402
query51	3927	3878	3938	3878
query52	112	106	99	99
query53	248	264	190	190
query54	336	315	290	290
query55	87	82	82	82
query56	306	340	322	322
query57	1207	1202	1106	1106
query58	297	273	288	273
query59	2554	2769	2532	2532
query60	358	358	346	346
query61	191	186	191	186
query62	788	747	667	667
query63	239	196	194	194
query64	4645	1276	958	958
query65	4032	3913	3962	3913
query66	1188	460	372	372
query67	15350	15263	15218	15218
query68	6122	946	605	605
query69	508	323	292	292
query70	1375	1219	1226	1219
query71	477	338	301	301
query72	5871	4949	4965	4949
query73	633	601	363	363
query74	8890	8963	8939	8939
query75	3295	3317	2840	2840
query76	3276	1126	729	729
query77	524	393	312	312
query78	9569	9871	8884	8884
query79	1646	848	588	588
query80	696	573	503	503
query81	530	264	231	231
query82	208	160	135	135
query83	274	272	251	251
query84	263	105	97	97
query85	929	496	452	452
query86	373	298	304	298
query87	3741	3725	3669	3669
query88	2830	2265	2236	2236
query89	386	340	300	300
query90	1883	225	214	214
query91	166	167	134	134
query92	76	70	62	62
query93	1172	980	656	656
query94	643	453	359	359
query95	408	335	316	316
query96	488	579	287	287
query97	2895	2965	2874	2874
query98	226	218	210	210
query99	1288	1401	1323	1323
Total cold run time: 268429 ms
Total hot run time: 188173 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 27.53 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 987ac7aab8968480f3a51e54163ed4d98c0268a0, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.06
query3	0.25	0.08	0.08
query4	1.61	0.11	0.11
query5	0.27	0.25	0.26
query6	1.18	0.66	0.65
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.58	0.52	0.51
query10	0.57	0.61	0.58
query11	0.16	0.12	0.11
query12	0.15	0.11	0.12
query13	0.62	0.61	0.60
query14	1.00	1.01	0.99
query15	0.85	0.82	0.83
query16	0.38	0.39	0.40
query17	1.03	1.04	1.03
query18	0.21	0.19	0.19
query19	1.92	1.80	1.87
query20	0.02	0.01	0.02
query21	15.45	0.19	0.13
query22	5.11	0.06	0.04
query23	15.67	0.25	0.10
query24	2.93	0.48	0.43
query25	0.07	0.06	0.06
query26	0.15	0.14	0.14
query27	0.06	0.05	0.06
query28	4.29	1.13	0.94
query29	12.65	3.98	3.26
query30	0.29	0.14	0.12
query31	2.82	0.59	0.38
query32	3.22	0.54	0.46
query33	3.12	3.08	3.01
query34	15.83	5.17	4.58
query35	4.60	4.57	4.59
query36	0.67	0.51	0.48
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.10	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 98.52 s
Total hot run time: 27.53 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 14, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 33.33% (1/3) 🎉
Increment coverage report
Complete coverage report

@morningman morningman merged commit 08b701f into apache:master Nov 14, 2025
28 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 14, 2025
…ve partitioned tables on object storage (#57973)

## Problem
When inserting data into Hive partitioned tables stored on S3-compatible
object storage (OSS/COS/OBS), the operation fails with authentication
error because
BE unifies all object storage under "s3://" scheme, but HMS expects the
original scheme (e.g., "oss://"). The mismatch causes s3a FileSystem to
access OSS
  endpoints with wrong credentials.

## Solution
Changed `HMSTransaction.finishInsertTable()` line 277 to use `writePath`
instead of `getTargetPath()`. The `writePath` variable already contains
the correct
  original scheme from HMS, avoiding the scheme conversion issue.

## Test
Added partition table insert tests for OSS/COS/OBS in
`hive_on_hms_and_dlf.groovy`.
github-actions bot pushed a commit that referenced this pull request Nov 14, 2025
…ve partitioned tables on object storage (#57973)

## Problem
When inserting data into Hive partitioned tables stored on S3-compatible
object storage (OSS/COS/OBS), the operation fails with authentication
error because
BE unifies all object storage under "s3://" scheme, but HMS expects the
original scheme (e.g., "oss://"). The mismatch causes s3a FileSystem to
access OSS
  endpoints with wrong credentials.

## Solution
Changed `HMSTransaction.finishInsertTable()` line 277 to use `writePath`
instead of `getTargetPath()`. The `writePath` variable already contains
the correct
  original scheme from HMS, avoiding the scheme conversion issue.

## Test
Added partition table insert tests for OSS/COS/OBS in
`hive_on_hms_and_dlf.groovy`.
yiguolei pushed a commit that referenced this pull request Nov 15, 2025
…ting into Hive partitioned tables on object storage #57973 (#58028)

Cherry-picked from #57973

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
morningman pushed a commit that referenced this pull request Nov 19, 2025
…ting into Hive partitioned tables on object storage #57973 (#58027)

Cherry-picked from #57973

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
@zy-kkk zy-kkk deleted the hive_new_partition branch November 26, 2025 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.3-merged dev/4.0.2-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants