Skip to content

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 8, 2025

Cherry-picked from #55675

### What problem does this PR solve?

Previously, HDFS High Availability (HA) configuration checks were only
performed for the HMS catalog. This PR extends HA validation to all
components that interact with HDFS, including:

- All catalog types (HMS, Hive, Iceberg, etc.)
- Table-valued functions (TVFs) accessing HDFS resources
- Any future features that depend on HDFS HA
@github-actions github-actions bot requested a review from morrySnow as a code owner September 8, 2025 06:09
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Sep 8, 2025
@dataroaring dataroaring reopened this Sep 8, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32801 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e4c1aff70f1e1723ba4f9deb83a9a30c5f2553f, data reload: false

------ Round 1 ----------------------------------
q1	17595	5757	5405	5405
q2	2021	385	289	289
q3	11846	1234	726	726
q4	10217	871	457	457
q5	7888	2385	2115	2115
q6	176	164	135	135
q7	904	732	648	648
q8	9320	1436	1230	1230
q9	5115	4982	4918	4918
q10	6753	2274	1827	1827
q11	485	286	267	267
q12	342	357	208	208
q13	17782	3599	2993	2993
q14	223	231	204	204
q15	527	468	459	459
q16	430	435	377	377
q17	593	867	374	374
q18	6975	6441	6464	6441
q19	1212	966	553	553
q20	344	334	204	204
q21	2950	2214	1992	1992
q22	1059	979	991	979
Total cold run time: 104757 ms
Total hot run time: 32801 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5492	5492	5475	5475
q2	233	322	237	237
q3	2354	2651	2323	2323
q4	1367	1782	1395	1395
q5	4393	4898	5001	4898
q6	172	162	133	133
q7	2102	1914	1854	1854
q8	2630	2810	2676	2676
q9	7395	7208	7281	7208
q10	3062	3311	2697	2697
q11	578	507	501	501
q12	679	795	635	635
q13	3379	3737	3118	3118
q14	284	292	277	277
q15	513	489	479	479
q16	447	480	443	443
q17	1229	1711	1272	1272
q18	7709	7330	7307	7307
q19	804	1113	1132	1113
q20	1996	2041	1893	1893
q21	5393	4891	4557	4557
q22	1093	1046	999	999
Total cold run time: 53304 ms
Total hot run time: 51490 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192584 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e4c1aff70f1e1723ba4f9deb83a9a30c5f2553f, data reload: false

query1	951	391	385	385
query2	6220	1990	1895	1895
query3	8702	203	193	193
query4	33706	23820	23479	23479
query5	3684	586	446	446
query6	279	206	174	174
query7	4202	491	319	319
query8	298	244	255	244
query9	9519	2624	2631	2624
query10	460	340	265	265
query11	17968	15275	15181	15181
query12	167	111	107	107
query13	1559	543	421	421
query14	10303	7456	7264	7264
query15	241	190	176	176
query16	8126	676	492	492
query17	1550	771	596	596
query18	2165	437	323	323
query19	233	188	171	171
query20	122	122	114	114
query21	216	128	110	110
query22	4583	4542	4564	4542
query23	35316	33961	34387	33961
query24	7473	2704	2711	2704
query25	539	494	399	399
query26	1283	295	175	175
query27	2036	485	353	353
query28	5169	2252	2215	2215
query29	746	579	443	443
query30	235	186	172	172
query31	1016	931	852	852
query32	89	61	60	60
query33	496	365	316	316
query34	763	873	528	528
query35	814	818	756	756
query36	1013	1059	987	987
query37	107	94	67	67
query38	3991	4025	3943	3943
query39	1616	1484	1482	1482
query40	220	120	108	108
query41	49	47	48	47
query42	126	105	109	105
query43	515	510	500	500
query44	1356	828	832	828
query45	189	176	170	170
query46	896	1075	672	672
query47	1988	1981	1952	1952
query48	428	441	350	350
query49	771	482	407	407
query50	689	698	435	435
query51	7473	7274	7354	7274
query52	100	105	93	93
query53	237	259	187	187
query54	546	558	472	472
query55	92	80	85	80
query56	297	285	244	244
query57	1315	1276	1227	1227
query58	241	224	217	217
query59	3087	3111	2958	2958
query60	306	283	276	276
query61	118	110	117	110
query62	811	765	714	714
query63	249	199	189	189
query64	4483	1004	658	658
query65	3394	3311	3263	3263
query66	1021	461	323	323
query67	16887	15848	15534	15534
query68	6392	827	545	545
query69	494	300	268	268
query70	1247	1154	1110	1110
query71	387	308	267	267
query72	5177	3737	4017	3737
query73	630	750	354	354
query74	10215	9062	9067	9062
query75	3186	3145	2666	2666
query76	3071	1183	784	784
query77	500	372	284	284
query78	10204	10418	9619	9619
query79	3138	881	600	600
query80	622	524	437	437
query81	496	263	219	219
query82	561	125	91	91
query83	160	167	146	146
query84	248	97	77	77
query85	784	362	307	307
query86	363	312	302	302
query87	4282	4280	4227	4227
query88	4703	2427	2405	2405
query89	412	340	300	300
query90	1792	192	197	192
query91	140	138	109	109
query92	59	56	55	55
query93	1744	909	555	555
query94	688	401	319	319
query95	340	283	274	274
query96	492	613	297	297
query97	3219	3250	3145	3145
query98	226	214	199	199
query99	1495	1409	1305	1305
Total cold run time: 293307 ms
Total hot run time: 192584 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5e4c1aff70f1e1723ba4f9deb83a9a30c5f2553f, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.50	0.52	0.49
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.57	0.50	0.49
query10	0.58	0.55	0.55
query11	0.15	0.11	0.10
query12	0.14	0.11	0.10
query13	0.61	0.59	0.60
query14	0.77	0.81	0.80
query15	0.84	0.83	0.83
query16	0.38	0.39	0.39
query17	1.05	1.05	1.06
query18	0.24	0.23	0.23
query19	1.90	1.86	1.91
query20	0.01	0.01	0.01
query21	15.39	0.92	0.56
query22	0.74	0.66	0.61
query23	15.29	1.44	0.57
query24	3.45	0.90	1.58
query25	0.18	0.15	0.10
query26	0.45	0.14	0.13
query27	0.05	0.05	0.05
query28	13.25	0.98	0.45
query29	12.57	3.99	3.30
query30	0.25	0.09	0.07
query31	2.82	0.60	0.39
query32	3.23	0.53	0.48
query33	3.00	3.02	3.01
query34	16.68	5.20	4.56
query35	4.65	4.55	4.52
query36	0.65	0.51	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 104 s
Total hot run time: 28.96 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 68.85% (42/61) 🎉
Increment coverage report
Complete coverage report

@morrySnow
Copy link
Contributor

ut failed, should pick mannually

@morrySnow morrySnow closed this Sep 8, 2025
@CalvinKirs CalvinKirs reopened this Sep 9, 2025
@CalvinKirs
Copy link
Member

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32378 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6b556b6b6cfa0c13c9af99ec4f9ea6c8eb769ba1, data reload: false

------ Round 1 ----------------------------------
q1	17594	5464	5396	5396
q2	2012	398	275	275
q3	11987	1229	737	737
q4	10269	856	451	451
q5	8827	2325	2107	2107
q6	186	165	132	132
q7	912	737	626	626
q8	9343	1424	1089	1089
q9	5226	4952	4888	4888
q10	6778	2268	1851	1851
q11	466	282	255	255
q12	344	348	208	208
q13	17780	3584	3011	3011
q14	233	227	209	209
q15	528	467	462	462
q16	417	421	365	365
q17	596	852	369	369
q18	6872	6300	6444	6300
q19	1202	955	531	531
q20	330	332	210	210
q21	2772	2152	1940	1940
q22	1056	1012	966	966
Total cold run time: 105730 ms
Total hot run time: 32378 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5478	5479	5557	5479
q2	244	338	237	237
q3	2279	2683	2368	2368
q4	1366	1840	1371	1371
q5	4513	5031	4985	4985
q6	168	166	131	131
q7	2074	1991	1849	1849
q8	2648	2835	2652	2652
q9	7251	7142	7246	7142
q10	2998	3333	2721	2721
q11	574	512	490	490
q12	678	777	633	633
q13	3451	3782	3142	3142
q14	295	291	260	260
q15	504	463	484	463
q16	444	470	442	442
q17	1233	1708	1256	1256
q18	7639	7449	7334	7334
q19	790	952	1163	952
q20	1983	2043	1918	1918
q21	5419	4893	4595	4595
q22	1091	1042	984	984
Total cold run time: 53120 ms
Total hot run time: 51404 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192441 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6b556b6b6cfa0c13c9af99ec4f9ea6c8eb769ba1, data reload: false

query1	960	427	445	427
query2	6276	1922	1900	1900
query3	8683	199	199	199
query4	33671	23681	23916	23681
query5	3909	591	450	450
query6	285	197	187	187
query7	4205	491	321	321
query8	309	247	260	247
query9	9503	2633	2613	2613
query10	504	328	276	276
query11	17810	15500	15197	15197
query12	154	110	107	107
query13	1571	543	421	421
query14	9956	7198	6986	6986
query15	233	186	181	181
query16	8075	679	491	491
query17	1574	768	570	570
query18	2148	409	311	311
query19	217	187	164	164
query20	126	132	118	118
query21	210	129	106	106
query22	4511	4656	4403	4403
query23	35348	34370	34151	34151
query24	7886	2775	2723	2723
query25	548	511	427	427
query26	809	293	181	181
query27	2170	503	358	358
query28	5185	2220	2204	2204
query29	716	588	475	475
query30	244	193	168	168
query31	1080	927	853	853
query32	91	65	60	60
query33	504	374	333	333
query34	730	862	517	517
query35	780	817	729	729
query36	1021	1051	967	967
query37	101	95	67	67
query38	4026	4009	4004	4004
query39	1532	1504	1474	1474
query40	213	126	102	102
query41	51	49	49	49
query42	128	107	102	102
query43	527	520	485	485
query44	1355	831	815	815
query45	190	183	170	170
query46	880	1061	670	670
query47	1963	1963	1894	1894
query48	414	437	347	347
query49	749	483	399	399
query50	666	687	427	427
query51	7389	7281	7302	7281
query52	104	104	92	92
query53	229	260	191	191
query54	549	536	470	470
query55	85	80	82	80
query56	265	270	267	267
query57	1259	1297	1213	1213
query58	241	225	224	224
query59	3044	3242	3139	3139
query60	297	291	291	291
query61	119	119	120	119
query62	799	731	720	720
query63	250	198	196	196
query64	3985	972	663	663
query65	3403	3323	3274	3274
query66	770	406	301	301
query67	16111	15730	15534	15534
query68	7745	811	538	538
query69	500	302	271	271
query70	1199	1101	1092	1092
query71	397	293	261	261
query72	5731	3729	3956	3729
query73	644	740	355	355
query74	10100	9469	9021	9021
query75	3280	3154	2666	2666
query76	3175	1165	768	768
query77	717	365	274	274
query78	10275	10329	9490	9490
query79	3395	887	575	575
query80	625	548	439	439
query81	489	256	219	219
query82	576	124	90	90
query83	168	169	152	152
query84	278	99	88	88
query85	798	377	302	302
query86	342	311	290	290
query87	4397	4296	4218	4218
query88	5009	2406	2380	2380
query89	402	337	298	298
query90	1817	192	186	186
query91	140	139	121	121
query92	61	56	52	52
query93	1860	871	547	547
query94	692	419	302	302
query95	345	284	268	268
query96	494	603	280	280
query97	3159	3292	3101	3101
query98	232	213	203	203
query99	1520	1441	1303	1303
Total cold run time: 294258 ms
Total hot run time: 192441 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6b556b6b6cfa0c13c9af99ec4f9ea6c8eb769ba1, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.02
query3	0.24	0.06	0.06
query4	1.63	0.10	0.11
query5	0.49	0.51	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.60	0.52	0.51
query10	0.55	0.55	0.55
query11	0.15	0.11	0.10
query12	0.14	0.11	0.10
query13	0.62	0.59	0.59
query14	0.77	0.81	0.80
query15	0.83	0.84	0.82
query16	0.38	0.40	0.39
query17	1.10	1.01	1.09
query18	0.24	0.23	0.23
query19	1.95	1.90	1.91
query20	0.02	0.01	0.02
query21	15.36	0.93	0.57
query22	0.73	0.80	0.90
query23	14.87	1.42	0.55
query24	3.25	1.29	1.17
query25	0.17	0.18	0.12
query26	0.21	0.15	0.14
query27	0.05	0.07	0.04
query28	13.96	0.96	0.43
query29	12.61	4.01	3.28
query30	0.26	0.09	0.07
query31	2.83	0.59	0.38
query32	3.22	0.53	0.46
query33	3.02	2.98	2.99
query34	16.68	5.26	4.56
query35	4.64	4.58	4.55
query36	0.65	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.07	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.99 s
Total hot run time: 29.4 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 72.13% (44/61) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow merged commit 797f020 into branch-3.1 Sep 9, 2025
30 of 37 checks passed
@morrySnow morrySnow deleted the auto-pick-55675-branch-3.1 branch September 9, 2025 17:36
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants