Skip to content

Conversation

@morningman
Copy link
Contributor

bp #49754

@morningman morningman requested a review from dataroaring as a code owner April 22, 2025 05:08
@Thearas
Copy link
Contributor

Thearas commented Apr 22, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Problem Summary:

When insert data to a hive table on oss-hdfs, it may fail with following
error:
```
Failed to delete directories for files: [oss://xxx]
```
This is because for oss-hdfs, we should use hadoop filesystem to do the
operation.

This PR fix it.
When calling `getFSIdentity()`, we should pass the properties so that
`LocationPath` can identify
the right fs type.
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39658 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2aa92bc5d504427b34fa33848b781960a06ec08, data reload: false

------ Round 1 ----------------------------------
q1	17605	6658	6570	6570
q2	2054	170	163	163
q3	10612	1097	1174	1097
q4	10558	798	739	739
q5	7718	2837	2756	2756
q6	213	131	133	131
q7	971	628	600	600
q8	9376	1922	2049	1922
q9	6501	6386	6370	6370
q10	7070	2250	2272	2250
q11	477	263	269	263
q12	403	211	208	208
q13	17770	2998	2970	2970
q14	242	203	207	203
q15	505	459	461	459
q16	659	589	580	580
q17	992	591	577	577
q18	7125	6642	6468	6468
q19	1390	1017	1136	1017
q20	470	201	203	201
q21	3950	3099	3344	3099
q22	1096	1023	1015	1015
Total cold run time: 107757 ms
Total hot run time: 39658 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6573	6523	6558	6523
q2	335	239	246	239
q3	2942	2857	2925	2857
q4	2059	1812	1755	1755
q5	5810	5713	5721	5713
q6	205	135	134	134
q7	2212	1804	1770	1770
q8	3327	3563	3477	3477
q9	8979	8811	8839	8811
q10	3544	3523	3501	3501
q11	607	496	507	496
q12	806	630	603	603
q13	9931	3199	3117	3117
q14	306	266	297	266
q15	503	447	457	447
q16	704	641	649	641
q17	1823	1636	1634	1634
q18	8260	7679	7749	7679
q19	1632	1584	1500	1500
q20	2049	1905	1867	1867
q21	5482	5234	5146	5146
q22	1093	1048	1022	1022
Total cold run time: 69182 ms
Total hot run time: 59198 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196003 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2aa92bc5d504427b34fa33848b781960a06ec08, data reload: false

query1	1276	911	950	911
query2	6298	2037	1976	1976
query3	10812	4207	4397	4207
query4	60539	30045	23979	23979
query5	5254	456	444	444
query6	399	171	171	171
query7	5442	303	305	303
query8	298	213	219	213
query9	8628	2605	2572	2572
query10	453	251	272	251
query11	18037	15093	15716	15093
query12	155	109	106	106
query13	1507	454	420	420
query14	10841	6826	6754	6754
query15	208	177	184	177
query16	7152	498	488	488
query17	1179	583	606	583
query18	1864	328	322	322
query19	206	164	166	164
query20	112	109	111	109
query21	210	107	102	102
query22	4531	4367	4688	4367
query23	34358	34136	33829	33829
query24	6141	2883	2931	2883
query25	553	452	421	421
query26	660	166	168	166
query27	1919	370	372	370
query28	4514	2460	2451	2451
query29	672	455	420	420
query30	242	158	161	158
query31	971	797	828	797
query32	68	53	54	53
query33	398	293	277	277
query34	916	497	504	497
query35	880	744	709	709
query36	1079	995	972	972
query37	115	67	66	66
query38	4043	3982	3940	3940
query39	1549	1468	1486	1468
query40	199	99	98	98
query41	46	47	45	45
query42	116	109	102	102
query43	532	490	495	490
query44	1182	834	828	828
query45	187	167	174	167
query46	1117	704	735	704
query47	1969	1900	1966	1900
query48	480	368	388	368
query49	723	381	396	381
query50	840	441	423	423
query51	7321	7311	7154	7154
query52	101	93	90	90
query53	256	183	188	183
query54	584	479	471	471
query55	84	78	81	78
query56	273	282	258	258
query57	1239	1169	1143	1143
query58	208	208	235	208
query59	3237	3009	2948	2948
query60	293	260	244	244
query61	161	112	108	108
query62	753	656	700	656
query63	218	187	185	185
query64	1366	670	640	640
query65	3278	3194	3216	3194
query66	718	297	304	297
query67	15740	15328	15415	15328
query68	4142	578	581	578
query69	435	264	258	258
query70	1179	1110	1104	1104
query71	349	254	251	251
query72	6359	4001	3908	3908
query73	742	344	345	344
query74	10143	9025	8952	8952
query75	3390	2607	2638	2607
query76	2019	1141	1044	1044
query77	509	269	262	262
query78	10635	9578	9518	9518
query79	2043	600	605	600
query80	1384	434	420	420
query81	522	243	238	238
query82	1251	89	85	85
query83	265	148	133	133
query84	285	79	77	77
query85	1020	288	286	286
query86	382	293	288	288
query87	4368	4238	4195	4195
query88	3860	2386	2358	2358
query89	427	287	292	287
query90	2015	185	185	185
query91	182	146	146	146
query92	57	48	48	48
query93	2813	565	560	560
query94	788	290	303	290
query95	356	261	262	261
query96	617	286	277	277
query97	3263	3102	3104	3102
query98	218	198	206	198
query99	1556	1263	1301	1263
Total cold run time: 315274 ms
Total hot run time: 196003 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2aa92bc5d504427b34fa33848b781960a06ec08, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.30	0.06	0.07
query4	2.12	0.10	0.10
query5	0.51	0.51	0.52
query6	1.15	0.74	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.50	0.50
query10	0.56	0.55	0.54
query11	0.14	0.11	0.11
query12	0.14	0.13	0.11
query13	0.60	0.60	0.59
query14	2.72	2.72	2.72
query15	0.89	0.83	0.84
query16	0.38	0.39	0.38
query17	1.04	1.02	1.05
query18	0.24	0.22	0.21
query19	2.01	1.88	1.98
query20	0.02	0.01	0.01
query21	15.35	0.60	0.58
query22	2.55	1.84	2.36
query23	16.88	0.84	0.94
query24	3.27	0.95	0.75
query25	0.21	0.10	0.23
query26	0.39	0.15	0.14
query27	0.04	0.05	0.04
query28	10.43	0.48	0.48
query29	12.61	3.18	3.24
query30	0.24	0.06	0.06
query31	2.89	0.39	0.39
query32	3.23	0.46	0.46
query33	2.93	3.04	3.07
query34	17.10	4.52	4.45
query35	4.48	4.44	4.53
query36	0.64	0.48	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.25 s
Total hot run time: 31.96 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit ebbe891 into apache:branch-3.0 Apr 23, 2025
21 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants