Skip to content

[opt](hive) use binary search to prune hive partitions #58877

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 4 commits into from
Dec 10, 2025
Merged

[opt](hive) use binary search to prune hive partitions #58877

morningman merged 4 commits into from
Dec 10, 2025

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Dec 9, 2025 •
edited by morningman
Loading

Followup #44586
Enable binary search partition pruning optimization for Hive external tables.

This PR adds binary search partition pruning support for Hive tables by:

  • Adding getSortedPartitionRanges() method to ExternalTable base class
  • Maintaining sorted partition ranges directly in HivePartitionValues for cache lifecycle consistency
  • Overriding getSortedPartitionRanges() in HMSExternalTable to provide sorted ranges

Performance improvement (20000 partitions, 1000 queries):

  • Binary search enabled: 4.548 seconds
  • Binary search disabled: 12.849 seconds
  • ~2.8x faster

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zy-kkk
Copy link
Member Author

zy-kkk commented Dec 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 9, 2025

TPC-H: Total hot run time: 36473 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eb2c282b47308a9f2d5b9277650b93fd59d0180e, data reload: false

------ Round 1 ----------------------------------
q1	17609	5019	4922	4922
q2	2039	364	253	253
q3	10148	1335	761	761
q4	10212	842	321	321
q5	7500	2172	1994	1994
q6	193	170	134	134
q7	995	873	729	729
q8	9360	1443	1175	1175
q9	7026	5347	5426	5347
q10	6791	2389	1952	1952
q11	514	313	306	306
q12	677	732	595	595
q13	17793	3692	3049	3049
q14	303	307	274	274
q15	593	514	511	511
q16	932	930	882	882
q17	716	799	560	560
q18	7467	7082	6997	6997
q19	1225	953	612	612
q20	391	380	263	263
q21	4330	4036	3866	3866
q22	1079	983	970	970
Total cold run time: 107893 ms
Total hot run time: 36473 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4985	4903	4913	4903
q2	332	419	328	328
q3	2191	2739	2302	2302
q4	1338	1760	1346	1346
q5	4991	4672	4520	4520
q6	261	178	131	131
q7	2018	1974	1825	1825
q8	2681	2562	2712	2562
q9	7552	7560	7483	7483
q10	3099	3248	2797	2797
q11	596	535	506	506
q12	694	733	644	644
q13	3647	3902	3332	3332
q14	309	317	354	317
q15	555	530	513	513
q16	902	921	894	894
q17	1187	1488	1430	1430
q18	7851	7786	7575	7575
q19	892	842	904	842
q20	2023	2078	1968	1968
q21	4693	4433	4194	4194
q22	1050	1059	1013	1013
Total cold run time: 53847 ms
Total hot run time: 51425 ms

@doris-robot
Copy link

doris-robot commented Dec 9, 2025

TPC-DS: Total hot run time: 182268 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eb2c282b47308a9f2d5b9277650b93fd59d0180e, data reload: false

query5	5065	609	469	469
query6	328	231	221	221
query7	4214	488	281	281
query8	324	265	249	249
query9	8810	2656	2679	2656
query10	565	392	337	337
query11	15506	14854	14835	14835
query12	179	122	127	122
query13	1257	529	404	404
query14	6327	3280	3054	3054
query14_1	2945	2954	2932	2932
query15	219	196	186	186
query16	787	473	476	473
query17	1142	711	618	618
query18	2722	453	362	362
query19	247	236	212	212
query20	119	116	113	113
query21	222	142	119	119
query22	3914	3996	3762	3762
query23	16552	16208	16048	16048
query23_1	16159	16046	16046	16046
query24	7361	1674	1254	1254
query24_1	1248	1235	1246	1235
query25	576	502	458	458
query26	1249	287	167	167
query27	2736	460	310	310
query28	4431	2164	2176	2164
query29	850	580	469	469
query30	321	245	219	219
query31	827	728	646	646
query32	82	75	73	73
query33	555	364	311	311
query34	914	908	539	539
query35	793	862	737	737
query36	891	916	833	833
query37	130	91	75	75
query38	3853	3852	3801	3801
query39	759	731	711	711
query39_1	701	697	705	697
query40	221	143	123	123
query41	67	62	63	62
query42	106	107	105	105
query43	433	424	390	390
query44	1317	758	758	758
query45	192	194	185	185
query46	872	989	615	615
query47	1682	1742	1638	1638
query48	328	325	248	248
query49	649	444	353	353
query50	658	298	230	230
query51	3911	3874	3876	3874
query52	108	112	98	98
query53	329	350	297	297
query54	286	261	249	249
query55	78	78	74	74
query56	306	308	301	301
query57	1162	1156	1086	1086
query58	278	257	260	257
query59	2400	2475	2346	2346
query60	321	311	292	292
query61	162	155	156	155
query62	695	679	629	629
query63	325	298	297	297
query64	4997	1307	1017	1017
query65	4075	3977	3998	3977
query66	1385	433	321	321
query67	15293	14904	14766	14766
query68	8415	1026	728	728
query69	505	343	320	320
query70	1068	967	996	967
query71	381	331	289	289
query72	6121	5187	5275	5187
query73	701	645	311	311
query74	8924	8846	8827	8827
query75	3600	3538	3165	3165
query76	4047	1153	760	760
query77	592	409	293	293
query78	9417	9645	8845	8845
query79	1627	875	628	628
query80	723	640	566	566
query81	497	273	242	242
query82	200	131	106	106
query83	263	253	234	234
query84	256	115	96	96
query85	899	521	463	463
query86	388	305	288	288
query87	4012	4025	4002	4002
query88	3890	2282	2277	2277
query89	474	423	404	404
query90	2198	160	152	152
query91	175	167	143	143
query92	83	69	61	61
query93	1992	916	563	563
query94	472	313	292	292
query95	583	386	303	303
query96	595	467	211	211
query97	2603	2624	2566	2566
query98	221	193	193	193
query99	1316	1316	1244	1244
Total cold run time: 265575 ms
Total hot run time: 182268 ms

@doris-robot
Copy link

doris-robot commented Dec 9, 2025

ClickBench: Total hot run time: 27.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eb2c282b47308a9f2d5b9277650b93fd59d0180e, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.25	0.09	0.09
query4	1.61	0.12	0.11
query5	0.28	0.26	0.27
query6	1.15	0.65	0.62
query7	0.03	0.02	0.02
query8	0.06	0.04	0.04
query9	0.58	0.50	0.51
query10	0.56	0.56	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	0.98	0.98	0.97
query15	0.81	0.80	0.79
query16	0.38	0.39	0.40
query17	1.01	1.02	0.98
query18	0.24	0.22	0.22
query19	1.95	1.88	1.86
query20	0.02	0.01	0.01
query21	15.43	0.26	0.14
query22	4.84	0.06	0.05
query23	16.21	0.28	0.10
query24	1.72	0.25	0.33
query25	0.07	0.11	0.06
query26	0.14	0.14	0.13
query27	0.05	0.06	0.05
query28	4.63	1.23	1.01
query29	12.59	4.02	3.25
query30	0.29	0.16	0.12
query31	2.83	0.61	0.40
query32	3.24	0.55	0.46
query33	3.10	3.08	3.15
query34	16.67	5.24	4.52
query35	4.56	4.58	4.60
query36	0.67	0.50	0.49
query37	0.11	0.07	0.07
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.16	0.13
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 98.58 s
Total hot run time: 27.37 s

@zy-kkk
Copy link
Member Author

zy-kkk commented Dec 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 9, 2025

TPC-H: Total hot run time: 36315 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1a46590096eaff7a5cf99d33963a32066f635462, data reload: false

------ Round 1 ----------------------------------
q1	17616	5051	4934	4934
q2	2050	366	245	245
q3	10148	1295	739	739
q4	10239	870	333	333
q5	7537	2182	1900	1900
q6	195	171	138	138
q7	1006	873	723	723
q8	9361	1434	1142	1142
q9	6995	5312	5386	5312
q10	6787	2393	1999	1999
q11	553	316	304	304
q12	661	740	588	588
q13	17767	3704	3075	3075
q14	292	297	278	278
q15	588	515	521	515
q16	947	910	876	876
q17	695	817	533	533
q18	7922	7163	7137	7137
q19	956	964	612	612
q20	413	372	252	252
q21	4246	3918	3714	3714
q22	1058	978	966	966
Total cold run time: 108032 ms
Total hot run time: 36315 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4947	4953	4889	4889
q2	334	388	319	319
q3	2129	2707	2284	2284
q4	1308	1746	1273	1273
q5	4965	4673	4537	4537
q6	221	191	130	130
q7	2097	1962	1824	1824
q8	2690	2702	2520	2520
q9	7974	7991	7855	7855
q10	3213	3654	2940	2940
q11	658	519	517	517
q12	788	774	572	572
q13	3908	3979	3453	3453
q14	292	405	297	297
q15	586	581	520	520
q16	913	957	890	890
q17	1227	1499	1424	1424
q18	7859	7685	7843	7685
q19	919	871	866	866
q20	2018	2080	1850	1850
q21	4543	4307	4161	4161
q22	1069	1085	969	969
Total cold run time: 54658 ms
Total hot run time: 51775 ms

@doris-robot
Copy link

doris-robot commented Dec 9, 2025

TPC-DS: Total hot run time: 182033 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1a46590096eaff7a5cf99d33963a32066f635462, data reload: false

query5	5898	677	503	503
query6	436	248	230	230
query7	5254	475	292	292
query8	353	261	248	248
query9	8773	2636	2662	2636
query10	598	366	337	337
query11	15689	14804	15133	14804
query12	197	120	115	115
query13	1242	503	371	371
query14	7749	3328	3114	3114
query14_1	2936	3014	2963	2963
query15	273	202	184	184
query16	907	526	479	479
query17	1154	698	608	608
query18	2733	444	341	341
query19	314	228	210	210
query20	127	116	113	113
query21	219	146	114	114
query22	3820	3866	3943	3866
query23	16803	16088	15928	15928
query23_1	16065	16033	15930	15930
query24	6698	1673	1246	1246
query24_1	1249	1257	1271	1257
query25	552	499	418	418
query26	1224	271	170	170
query27	2702	481	312	312
query28	4516	2172	2142	2142
query29	797	564	452	452
query30	322	249	226	226
query31	814	719	643	643
query32	82	68	73	68
query33	521	334	313	313
query34	886	911	547	547
query35	781	807	742	742
query36	873	914	846	846
query37	141	105	87	87
query38	3954	3844	3771	3771
query39	753	741	734	734
query39_1	706	704	703	703
query40	227	142	128	128
query41	68	62	65	62
query42	110	105	105	105
query43	427	447	392	392
query44	1393	758	771	758
query45	194	193	186	186
query46	898	1002	636	636
query47	1634	1717	1600	1600
query48	312	350	254	254
query49	636	438	378	378
query50	685	312	231	231
query51	3852	3921	3812	3812
query52	109	113	104	104
query53	325	363	305	305
query54	307	290	277	277
query55	80	82	78	78
query56	340	345	339	339
query57	1159	1143	1087	1087
query58	277	274	261	261
query59	2380	2446	2348	2348
query60	361	335	321	321
query61	198	188	191	188
query62	696	692	636	636
query63	340	301	312	301
query64	5075	1410	1138	1138
query65	4058	3982	3978	3978
query66	1395	476	334	334
query67	15201	14838	14688	14688
query68	6470	1027	747	747
query69	504	356	330	330
query70	1061	1034	985	985
query71	410	325	300	300
query72	6096	5010	4831	4831
query73	692	617	306	306
query74	8905	8873	8661	8661
query75	3566	3545	3184	3184
query76	3791	1177	791	791
query77	513	397	304	304
query78	9496	9845	8832	8832
query79	1602	943	615	615
query80	1365	661	562	562
query81	564	269	238	238
query82	420	127	105	105
query83	269	256	243	243
query84	265	114	98	98
query85	911	509	456	456
query86	376	321	280	280
query87	4002	4097	3978	3978
query88	3218	2318	2261	2261
query89	476	434	386	386
query90	2014	159	157	157
query91	178	167	141	141
query92	73	67	62	62
query93	1279	935	576	576
query94	536	305	277	277
query95	552	326	366	326
query96	606	455	211	211
query97	2623	2632	2605	2605
query98	213	198	193	193
query99	1291	1312	1221	1221
Total cold run time: 266033 ms
Total hot run time: 182033 ms

@doris-robot
Copy link

doris-robot commented Dec 9, 2025

ClickBench: Total hot run time: 27.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1a46590096eaff7a5cf99d33963a32066f635462, data reload: false

query1	0.05	0.04	0.05
query2	0.10	0.04	0.04
query3	0.26	0.09	0.08
query4	1.60	0.12	0.11
query5	0.27	0.27	0.26
query6	1.17	0.64	0.63
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.52	0.52
query10	0.56	0.55	0.55
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.63	0.61	0.59
query14	0.99	0.98	0.99
query15	0.82	0.79	0.81
query16	0.41	0.41	0.41
query17	1.02	1.06	1.01
query18	0.22	0.21	0.21
query19	1.94	1.80	1.84
query20	0.02	0.01	0.01
query21	15.47	0.29	0.14
query22	4.88	0.05	0.05
query23	16.07	0.28	0.10
query24	1.48	1.10	0.62
query25	0.09	0.08	0.05
query26	0.15	0.14	0.14
query27	0.05	0.05	0.06
query28	5.04	1.22	1.03
query29	12.61	4.05	3.19
query30	0.29	0.14	0.13
query31	2.81	0.62	0.38
query32	3.23	0.56	0.46
query33	3.05	3.03	3.03
query34	16.84	5.22	4.55
query35	4.55	4.64	4.56
query36	0.66	0.49	0.49
query37	0.11	0.07	0.06
query38	0.07	0.05	0.04
query39	0.04	0.03	0.04
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.05	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 98.86 s
Total hot run time: 27.63 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 9, 2025

FE UT Coverage Report

Increment line coverage 10.00% (2/20) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 9, 2025

FE Regression Coverage Report

Increment line coverage 55.00% (11/20) 🎉
Increment coverage report
Complete coverage report

@zy-kkk
Copy link
Member Author

zy-kkk commented Dec 10, 2025

run buildall

924060929
924060929 reviewed Dec 10, 2025
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
});

return new SortedPartitionRanges<>(sortedRanges, defaultPartitions);
}
Copy link
Contributor

@924060929 924060929 Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should extract code from NereidsSortedPartitionsCacheManager.loadCache and reuse the same utility function

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-H: Total hot run time: 36931 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 22e3e49a305a54a8ec210d4545be2253a54128ab, data reload: false

------ Round 1 ----------------------------------
q1	17659	4257	4064	4064
q2	2035	353	236	236
q3	10155	1370	750	750
q4	10222	845	316	316
q5	7546	2175	1963	1963
q6	193	173	141	141
q7	1008	876	723	723
q8	9372	1477	1199	1199
q9	7004	5362	5374	5362
q10	6872	2396	1960	1960
q11	544	319	312	312
q12	724	737	578	578
q13	17806	3708	3095	3095
q14	287	295	273	273
q15	601	527	519	519
q16	944	912	889	889
q17	713	854	568	568
q18	8197	8077	7762	7762
q19	1245	1027	668	668
q20	430	381	266	266
q21	4653	4365	4249	4249
q22	1147	1157	1038	1038
Total cold run time: 109357 ms
Total hot run time: 36931 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4268	4252	4268	4252
q2	344	417	397	397
q3	2401	2812	2524	2524
q4	1496	2179	1437	1437
q5	4498	4595	4458	4458
q6	201	168	129	129
q7	2028	1934	1924	1924
q8	2716	2551	2504	2504
q9	7441	7474	7396	7396
q10	2926	3082	2680	2680
q11	571	491	471	471
q12	610	697	559	559
q13	3245	3668	3053	3053
q14	281	283	268	268
q15	534	494	493	493
q16	870	896	850	850
q17	1140	1377	1378	1377
q18	7351	7213	7031	7031
q19	864	793	823	793
q20	1924	1960	1816	1816
q21	4672	4328	4223	4223
q22	1110	1017	1004	1004
Total cold run time: 51491 ms
Total hot run time: 49639 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-DS: Total hot run time: 181153 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 22e3e49a305a54a8ec210d4545be2253a54128ab, data reload: false

query5	4841	602	464	464
query6	325	231	217	217
query7	4221	464	279	279
query8	298	253	234	234
query9	8767	2560	2592	2560
query10	510	381	324	324
query11	15449	14859	14916	14859
query12	168	114	114	114
query13	1258	505	398	398
query14	6030	3306	2970	2970
query14_1	2876	2850	2872	2850
query15	211	198	197	197
query16	887	495	422	422
query17	1146	674	578	578
query18	2462	439	338	338
query19	229	239	204	204
query20	122	113	110	110
query21	217	136	113	113
query22	3950	4008	3827	3827
query23	16659	16175	15901	15901
query23_1	16007	16132	16098	16098
query24	7340	1625	1242	1242
query24_1	1228	1226	1238	1226
query25	556	506	445	445
query26	1249	269	162	162
query27	2776	484	310	310
query28	4486	2169	2140	2140
query29	793	549	438	438
query30	315	247	222	222
query31	811	708	634	634
query32	78	71	65	65
query33	523	341	285	285
query34	898	889	544	544
query35	789	797	716	716
query36	872	905	833	833
query37	132	95	74	74
query38	3971	3865	3806	3806
query39	752	742	747	742
query39_1	695	686	709	686
query40	225	142	120	120
query41	66	62	62	62
query42	109	106	105	105
query43	433	436	405	405
query44	1355	760	755	755
query45	193	189	184	184
query46	871	986	614	614
query47	1652	1698	1624	1624
query48	320	331	245	245
query49	635	435	350	350
query50	655	303	226	226
query51	3975	3849	3827	3827
query52	112	107	102	102
query53	317	341	300	300
query54	289	274	261	261
query55	82	74	71	71
query56	291	294	293	293
query57	1165	1142	1068	1068
query58	268	249	261	249
query59	2311	2410	2321	2321
query60	344	324	315	315
query61	211	184	179	179
query62	726	692	646	646
query63	336	303	305	303
query64	5170	1329	1000	1000
query65	4049	3926	3993	3926
query66	1452	437	322	322
query67	15275	14983	14648	14648
query68	4656	1037	751	751
query69	494	353	315	315
query70	1068	965	981	965
query71	371	325	285	285
query72	6185	4972	5066	4972
query73	692	619	308	308
query74	8897	8779	8570	8570
query75	3577	3527	3227	3227
query76	3800	1135	753	753
query77	525	402	310	310
query78	9492	9555	8811	8811
query79	1043	887	643	643
query80	1153	691	591	591
query81	553	276	237	237
query82	411	134	108	108
query83	341	259	240	240
query84	256	130	105	105
query85	1047	563	464	464
query86	381	297	284	284
query87	4066	4178	3960	3960
query88	3195	2308	2278	2278
query89	469	423	404	404
query90	1940	166	156	156
query91	181	168	145	145
query92	73	68	66	66
query93	1022	944	577	577
query94	528	294	282	282
query95	592	340	394	340
query96	594	467	223	223
query97	2608	2634	2588	2588
query98	210	192	202	192
query99	1274	1294	1186	1186
Total cold run time: 259789 ms
Total hot run time: 181153 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

ClickBench: Total hot run time: 27.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 22e3e49a305a54a8ec210d4545be2253a54128ab, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.04	0.05
query3	0.26	0.09	0.09
query4	1.60	0.11	0.10
query5	0.30	0.29	0.26
query6	1.17	0.64	0.64
query7	0.04	0.03	0.02
query8	0.06	0.05	0.04
query9	0.57	0.50	0.51
query10	0.56	0.55	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.60	0.61
query14	0.99	0.98	0.98
query15	0.81	0.80	0.81
query16	0.39	0.40	0.39
query17	1.05	1.05	1.03
query18	0.23	0.23	0.22
query19	1.96	1.85	1.85
query20	0.02	0.02	0.01
query21	15.43	0.27	0.14
query22	5.03	0.05	0.05
query23	16.08	0.29	0.10
query24	0.95	0.42	0.18
query25	0.06	0.06	0.06
query26	0.14	0.14	0.13
query27	0.07	0.06	0.05
query28	3.61	1.22	1.03
query29	12.62	4.07	3.26
query30	0.28	0.13	0.11
query31	2.82	0.63	0.40
query32	3.22	0.54	0.45
query33	2.98	3.08	2.99
query34	16.90	5.21	4.60
query35	4.57	4.54	4.54
query36	0.66	0.49	0.51
query37	0.10	0.06	0.07
query38	0.08	0.04	0.04
query39	0.04	0.03	0.03
query40	0.16	0.14	0.14
query41	0.09	0.03	0.04
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 97.09 s
Total hot run time: 27.33 s

@github-actions
Copy link
Contributor

github-actions bot commented Dec 10, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 10, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 10, 2025

PR approved by anyone and no changes requested.

@zy-kkk
Copy link
Member Author

zy-kkk commented Dec 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-H: Total hot run time: 36566 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a5da2d7af4be796b451d720b4ec119da10d7b46, data reload: false

------ Round 1 ----------------------------------
q1	17671	4245	4088	4088
q2	2031	367	239	239
q3	10184	1309	745	745
q4	10227	869	311	311
q5	7593	2104	1924	1924
q6	184	166	136	136
q7	1029	884	717	717
q8	9353	1475	1110	1110
q9	6991	5326	5339	5326
q10	6791	2388	2010	2010
q11	525	324	296	296
q12	671	768	584	584
q13	17796	3638	3034	3034
q14	291	301	288	288
q15	576	521	515	515
q16	964	927	889	889
q17	718	807	534	534
q18	7385	7749	7823	7749
q19	1132	1016	665	665
q20	438	375	262	262
q21	4687	4276	4123	4123
q22	1097	1067	1021	1021
Total cold run time: 108334 ms
Total hot run time: 36566 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4379	4233	4298	4233
q2	351	390	340	340
q3	2377	2943	2558	2558
q4	1408	1896	1380	1380
q5	4591	4721	4428	4428
q6	205	171	128	128
q7	2021	1943	1770	1770
q8	2676	2556	2495	2495
q9	7591	7542	7486	7486
q10	3038	3243	2792	2792
q11	574	494	471	471
q12	669	685	570	570
q13	3297	3691	3041	3041
q14	265	294	266	266
q15	533	489	485	485
q16	872	870	835	835
q17	1161	1465	1338	1338
q18	7262	7084	7031	7031
q19	851	830	840	830
q20	1916	1960	1812	1812
q21	4626	4327	4250	4250
q22	1045	1039	984	984
Total cold run time: 51708 ms
Total hot run time: 49523 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-DS: Total hot run time: 180827 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a5da2d7af4be796b451d720b4ec119da10d7b46, data reload: false

query5	4841	623	467	467
query6	340	231	215	215
query7	4211	463	272	272
query8	293	253	226	226
query9	8760	2579	2557	2557
query10	534	410	347	347
query11	15507	14793	14556	14556
query12	176	113	111	111
query13	1265	502	391	391
query14	6061	3212	2943	2943
query14_1	2841	2889	2874	2874
query15	216	194	189	189
query16	904	470	450	450
query17	1085	694	591	591
query18	2478	435	331	331
query19	242	232	204	204
query20	129	111	111	111
query21	216	135	117	117
query22	4031	4127	3903	3903
query23	16662	16381	15949	15949
query23_1	16064	16103	15999	15999
query24	7322	1659	1210	1210
query24_1	1222	1234	1212	1212
query25	546	463	437	437
query26	1261	306	162	162
query27	2719	473	304	304
query28	4472	2199	2177	2177
query29	791	543	428	428
query30	312	244	210	210
query31	808	710	654	654
query32	80	69	68	68
query33	533	330	275	275
query34	899	954	544	544
query35	800	805	724	724
query36	861	894	820	820
query37	127	92	79	79
query38	3935	3803	3748	3748
query39	757	730	726	726
query39_1	704	712	695	695
query40	223	150	129	129
query41	68	64	65	64
query42	106	103	100	100
query43	415	427	401	401
query44	1323	760	757	757
query45	192	186	183	183
query46	883	976	605	605
query47	1715	1715	1638	1638
query48	335	330	258	258
query49	622	434	377	377
query50	670	286	213	213
query51	3801	3816	3823	3816
query52	109	115	112	112
query53	322	351	295	295
query54	314	271	272	271
query55	84	74	72	72
query56	306	333	292	292
query57	1139	1151	1072	1072
query58	264	252	269	252
query59	2359	2524	2356	2356
query60	326	314	291	291
query61	165	161	154	154
query62	710	669	618	618
query63	329	293	297	293
query64	5036	1314	992	992
query65	3999	3956	3960	3956
query66	1474	439	306	306
query67	15233	14801	14706	14706
query68	2789	1059	764	764
query69	458	365	333	333
query70	1026	1011	981	981
query71	336	311	286	286
query72	5868	5109	5017	5017
query73	483	549	309	309
query74	8627	8980	8695	8695
query75	3483	3554	3206	3206
query76	2903	1140	731	731
query77	367	444	292	292
query78	9507	9588	8834	8834
query79	2399	891	627	627
query80	1692	654	562	562
query81	564	264	234	234
query82	1017	131	102	102
query83	348	249	241	241
query84	253	124	102	102
query85	952	501	464	464
query86	396	321	276	276
query87	4106	4036	4000	4000
query88	3301	2302	2291	2291
query89	482	429	386	386
query90	1991	156	151	151
query91	170	164	147	147
query92	70	69	65	65
query93	1134	916	581	581
query94	530	310	263	263
query95	573	380	313	313
query96	597	471	205	205
query97	2592	2627	2579	2579
query98	201	200	190	190
query99	1262	1314	1201	1201
Total cold run time: 257668 ms
Total hot run time: 180827 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

ClickBench: Total hot run time: 27.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a5da2d7af4be796b451d720b4ec119da10d7b46, data reload: false

query1	0.05	0.04	0.05
query2	0.10	0.05	0.05
query3	0.25	0.09	0.09
query4	1.61	0.11	0.11
query5	0.28	0.26	0.26
query6	1.16	0.64	0.62
query7	0.03	0.03	0.02
query8	0.05	0.04	0.04
query9	0.56	0.52	0.51
query10	0.57	0.56	0.55
query11	0.15	0.11	0.12
query12	0.14	0.12	0.11
query13	0.61	0.60	0.59
query14	0.97	0.99	0.98
query15	0.82	0.79	0.81
query16	0.41	0.40	0.40
query17	1.03	1.02	1.03
query18	0.23	0.21	0.22
query19	1.90	1.85	1.76
query20	0.01	0.01	0.01
query21	15.44	0.29	0.14
query22	4.76	0.04	0.04
query23	15.98	0.28	0.10
query24	1.40	0.64	0.43
query25	0.08	0.06	0.07
query26	0.14	0.14	0.14
query27	0.06	0.04	0.07
query28	4.40	1.22	1.02
query29	12.57	4.05	3.20
query30	0.27	0.13	0.12
query31	2.83	0.64	0.40
query32	3.23	0.55	0.46
query33	2.93	3.08	3.03
query34	16.97	5.17	4.56
query35	4.56	4.57	4.63
query36	0.68	0.50	0.49
query37	0.10	0.06	0.07
query38	0.07	0.05	0.04
query39	0.05	0.03	0.04
query40	0.17	0.14	0.13
query41	0.08	0.03	0.04
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 97.78 s
Total hot run time: 27.41 s

@morningman
Copy link
Contributor

morningman commented Dec 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-H: Total hot run time: 35130 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a5da2d7af4be796b451d720b4ec119da10d7b46, data reload: false

------ Round 1 ----------------------------------
q1	17613	4183	4012	4012
q2	2023	363	236	236
q3	10190	1317	717	717
q4	10215	832	313	313
q5	7532	2133	1863	1863
q6	182	171	138	138
q7	984	833	710	710
q8	9345	1452	1099	1099
q9	6923	5327	5264	5264
q10	6845	2391	1969	1969
q11	547	318	289	289
q12	666	728	599	599
q13	17797	3652	2989	2989
q14	292	285	276	276
q15	582	518	517	517
q16	929	907	864	864
q17	707	793	546	546
q18	8471	7441	7212	7212
q19	1208	987	615	615
q20	391	367	260	260
q21	4344	4101	3680	3680
q22	1067	976	962	962
Total cold run time: 108853 ms
Total hot run time: 35130 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4345	4020	4003	4003
q2	337	398	326	326
q3	2146	2655	2282	2282
q4	1332	1760	1313	1313
q5	4224	4738	4550	4550
q6	244	186	144	144
q7	2088	2046	1823	1823
q8	2672	2475	2463	2463
q9	7630	7424	7572	7424
q10	3070	3297	2816	2816
q11	583	549	557	549
q12	698	817	608	608
q13	3609	3878	3418	3418
q14	299	298	290	290
q15	546	509	531	509
q16	922	909	884	884
q17	1181	1365	1433	1365
q18	8044	7419	7606	7419
q19	879	855	830	830
q20	1893	1962	1829	1829
q21	4605	4201	4130	4130
q22	1096	1028	974	974
Total cold run time: 52443 ms
Total hot run time: 49949 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

TPC-DS: Total hot run time: 181501 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a5da2d7af4be796b451d720b4ec119da10d7b46, data reload: false

query5	5082	620	467	467
query6	331	243	212	212
query7	4234	466	274	274
query8	299	246	243	243
query9	8759	2532	2577	2532
query10	547	370	338	338
query11	15139	14863	14524	14524
query12	173	120	111	111
query13	1280	496	395	395
query14	5852	3218	3007	3007
query14_1	2890	2983	2871	2871
query15	222	202	180	180
query16	885	480	454	454
query17	1126	718	614	614
query18	2545	447	359	359
query19	238	246	214	214
query20	123	120	114	114
query21	220	147	120	120
query22	3858	3913	3913	3913
query23	16436	16199	15903	15903
query23_1	16101	16161	16065	16065
query24	7399	1662	1254	1254
query24_1	1249	1204	1259	1204
query25	588	506	471	471
query26	1255	290	170	170
query27	2729	477	315	315
query28	4426	2190	2182	2182
query29	852	582	483	483
query30	326	249	214	214
query31	837	707	629	629
query32	79	74	68	68
query33	547	343	302	302
query34	884	902	545	545
query35	809	828	744	744
query36	876	894	835	835
query37	130	142	84	84
query38	3883	3870	3860	3860
query39	751	731	709	709
query39_1	705	706	701	701
query40	222	136	121	121
query41	69	62	63	62
query42	104	105	108	105
query43	431	436	408	408
query44	1320	767	750	750
query45	190	192	180	180
query46	872	982	616	616
query47	1635	1706	1629	1629
query48	315	323	247	247
query49	631	434	346	346
query50	661	290	222	222
query51	3779	3855	3783	3783
query52	105	108	100	100
query53	320	346	289	289
query54	311	258	249	249
query55	79	73	72	72
query56	298	319	304	304
query57	1138	1165	1069	1069
query58	273	253	248	248
query59	2342	2386	2449	2386
query60	311	316	288	288
query61	159	155	158	155
query62	680	656	619	619
query63	332	301	299	299
query64	5005	1350	998	998
query65	4047	3938	3981	3938
query66	1427	451	334	334
query67	15070	14985	14806	14806
query68	8104	1013	783	783
query69	497	340	309	309
query70	1056	1014	1028	1014
query71	377	305	283	283
query72	6077	4909	5021	4909
query73	660	583	309	309
query74	8763	8808	8653	8653
query75	3549	3512	3167	3167
query76	3934	1141	728	728
query77	545	396	294	294
query78	9378	9640	8997	8997
query79	1737	889	628	628
query80	736	649	565	565
query81	544	268	236	236
query82	203	133	104	104
query83	263	255	247	247
query84	267	120	101	101
query85	895	507	463	463
query86	387	309	273	273
query87	4063	4114	3951	3951
query88	3227	2327	2313	2313
query89	476	431	414	414
query90	2237	161	157	157
query91	175	168	169	168
query92	86	70	68	68
query93	2115	915	575	575
query94	490	306	275	275
query95	567	332	303	303
query96	587	484	211	211
query97	2616	2624	2568	2568
query98	217	201	190	190
query99	1258	1284	1196	1196
Total cold run time: 262615 ms
Total hot run time: 181501 ms

@doris-robot
Copy link

doris-robot commented Dec 10, 2025

ClickBench: Total hot run time: 27.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a5da2d7af4be796b451d720b4ec119da10d7b46, data reload: false

query1	0.06	0.05	0.06
query2	0.10	0.05	0.04
query3	0.25	0.08	0.09
query4	1.60	0.11	0.11
query5	0.27	0.25	0.26
query6	1.17	0.64	0.62
query7	0.03	0.03	0.02
query8	0.05	0.04	0.05
query9	0.58	0.51	0.51
query10	0.54	0.55	0.55
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	0.98	0.99	0.98
query15	0.83	0.80	0.81
query16	0.40	0.40	0.40
query17	1.07	1.04	1.05
query18	0.24	0.22	0.22
query19	1.90	1.82	1.74
query20	0.01	0.02	0.02
query21	15.47	0.26	0.14
query22	4.76	0.05	0.05
query23	15.94	0.28	0.10
query24	0.95	0.57	1.09
query25	0.11	0.06	0.07
query26	0.14	0.13	0.13
query27	0.06	0.05	0.05
query28	5.43	1.23	1.02
query29	12.58	4.15	3.31
query30	0.31	0.14	0.12
query31	2.83	0.64	0.40
query32	3.23	0.55	0.46
query33	3.05	3.05	3.08
query34	16.83	5.18	4.49
query35	4.60	4.59	4.52
query36	0.64	0.51	0.49
query37	0.11	0.07	0.07
query38	0.08	0.04	0.04
query39	0.05	0.03	0.04
query40	0.18	0.15	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 98.54 s
Total hot run time: 27.64 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 10, 2025

FE Regression Coverage Report

Increment line coverage 76.36% (42/55) 🎉
Increment coverage report
Complete coverage report

@morningman morningman merged commit 87dc631 into apache:master Dec 10, 2025
29 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 10, 2025
@zy-kkk
[opt](hive) use binary search to prune hive partitions (#58877)
5003198 
Followup #44586
Enable binary search partition pruning optimization for Hive external
tables.

This PR adds binary search partition pruning support for Hive tables by:
- Adding `getSortedPartitionRanges()` method to `ExternalTable` base
class
- Maintaining sorted partition ranges directly in `HivePartitionValues`
for cache lifecycle consistency
- Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to
provide sorted ranges

  **Performance improvement (20000 partitions, 1000 queries):**
  - Binary search enabled: **4.548 seconds**
  - Binary search disabled: **12.849 seconds**
  - **~2.8x faster**
@github-actions github-actions bot mentioned this pull request Dec 10, 2025
Merged
zy-kkk added a commit to zy-kkk/doris that referenced this pull request Dec 11, 2025
@zy-kkk
[opt](hive) use binary search to prune hive partitions (apache#58877)
d86afdb 
Followup apache#44586
Enable binary search partition pruning optimization for Hive external
tables.

This PR adds binary search partition pruning support for Hive tables by:
- Adding `getSortedPartitionRanges()` method to `ExternalTable` base
class
- Maintaining sorted partition ranges directly in `HivePartitionValues`
for cache lifecycle consistency
- Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to
provide sorted ranges

  **Performance improvement (20000 partitions, 1000 queries):**
  - Binary search enabled: **4.548 seconds**
  - Binary search disabled: **12.849 seconds**
  - **~2.8x faster**
@zy-kkk zy-kkk mentioned this pull request Dec 11, 2025
Merged
starocean999 pushed a commit that referenced this pull request Dec 11, 2025
@zy-kkk @starocean999
[opt](hive) use binary search to prune hive partitions (#58877)
000532e 
Followup #44586
Enable binary search partition pruning optimization for Hive external
tables.

This PR adds binary search partition pruning support for Hive tables by:
- Adding `getSortedPartitionRanges()` method to `ExternalTable` base
class
- Maintaining sorted partition ranges directly in `HivePartitionValues`
for cache lifecycle consistency
- Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to
provide sorted ranges

  **Performance improvement (20000 partitions, 1000 queries):**
  - Binary search enabled: **4.548 seconds**
  - Binary search disabled: **12.849 seconds**
  - **~2.8x faster**
yiguolei pushed a commit that referenced this pull request Dec 12, 2025
@github-actions @zy-kkk
branch-4.0: [opt](hive) use binary search to prune hive partitions #5…
99ea4a8 
…8877 (#58921)

Cherry-picked from #58877

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
@zy-kkk zy-kkk deleted the hive_pt branch December 12, 2025 03:04
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@zy-kkk @nagisa-kunhah
[opt](hive) use binary search to prune hive partitions (apache#58877)
3607dbd 
Followup apache#44586
Enable binary search partition pruning optimization for Hive external
tables.

This PR adds binary search partition pruning support for Hive tables by:
- Adding `getSortedPartitionRanges()` method to `ExternalTable` base
class
- Maintaining sorted partition ranges directly in `HivePartitionValues`
for cache lifecycle consistency
- Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to
provide sorted ranges

  **Performance improvement (20000 partitions, 1000 queries):**
  - Binary search enabled: **4.548 seconds**
  - Binary search disabled: **12.849 seconds**
  - **~2.8x faster**
zy-kkk added a commit to zy-kkk/doris that referenced this pull request Dec 16, 2025
@zy-kkk
branch-3.1: [opt](hive) use binary search to prune hive partitions (a…
738c09f 
…pache#58877)
morrySnow pushed a commit that referenced this pull request Dec 17, 2025
@zy-kkk
branch-3.1: [opt](hive) use binary search to prune hive partitions #5…
5c650d9 
…8877 (#58932)

pick #58877
@morningman morningman mentioned this pull request Dec 28, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@924060929 924060929 924060929 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.x dev/3.1.x-conflict dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@zy-kkk @hello-stephen @doris-robot @morningman @924060929 @yiguolei @morrySnow