Skip to content

branch-3.1: [opt](nereids) use binary search to prune partitions #44586 #51966

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jun 20, 2025
Merged

branch-3.1: [opt](nereids) use binary search to prune partitions #44586 #51966

morrySnow merged 1 commit into from
Jun 20, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jun 19, 2025

Cherry-picked from #44586

@924060929
[opt](nereids) use binary search to prune partitions (#44586)
33ddcf9 
use binary search to speed up prune partitions.

test: datetime partition column which range from 2020-07-02 to
2024-11-13, one date for one partition, query one partition for example
`dt = '2024-01-02'`, this pr can speed up from QPS 160 to QPS 610
@github-actions github-actions bot requested a review from morrySnow as a code owner June 19, 2025 12:14
@Thearas
Copy link
Contributor

Thearas commented Jun 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jun 19, 2025
@Thearas
Copy link
Contributor

Thearas commented Jun 19, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 19, 2025

TPC-H: Total hot run time: 39345 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 33ddcf95a05149b933c77b74191f3678b2cc252d, data reload: false

------ Round 1 ----------------------------------
q1	17606	7173	6551	6551
q2	2059	171	159	159
q3	10565	1063	1143	1063
q4	10591	780	734	734
q5	7729	2834	2818	2818
q6	212	134	128	128
q7	976	625	610	610
q8	9349	1951	1980	1951
q9	6531	6403	6374	6374
q10	6988	2257	2317	2257
q11	467	254	262	254
q12	409	203	208	203
q13	17797	2953	2980	2953
q14	247	200	210	200
q15	504	455	462	455
q16	489	365	362	362
q17	967	520	553	520
q18	7131	6701	6535	6535
q19	1393	1087	1006	1006
q20	467	201	199	199
q21	3857	3245	3047	3047
q22	1092	966	972	966
Total cold run time: 107426 ms
Total hot run time: 39345 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6577	6511	6503	6503
q2	327	235	226	226
q3	2835	2740	2772	2740
q4	2049	1725	1790	1725
q5	5717	5753	5720	5720
q6	209	126	128	126
q7	2204	1802	1777	1777
q8	3339	3531	3560	3531
q9	8974	8718	8852	8718
q10	3544	3480	3469	3469
q11	586	491	505	491
q12	798	617	620	617
q13	10047	3139	3188	3139
q14	302	264	269	264
q15	514	465	460	460
q16	481	427	416	416
q17	1817	1601	1620	1601
q18	8073	7806	7666	7666
q19	1693	1611	1623	1611
q20	2092	1789	1798	1789
q21	5206	5070	4920	4920
q22	1130	1046	1014	1014
Total cold run time: 68514 ms
Total hot run time: 58523 ms

@doris-robot
Copy link

doris-robot commented Jun 19, 2025

TPC-DS: Total hot run time: 195859 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 33ddcf95a05149b933c77b74191f3678b2cc252d, data reload: false

query1	1292	907	876	876
query2	6283	1893	1863	1863
query3	10854	4361	4351	4351
query4	60469	28634	23561	23561
query5	5258	449	470	449
query6	409	182	193	182
query7	5494	306	308	306
query8	304	227	225	225
query9	8674	2588	2571	2571
query10	478	296	246	246
query11	18107	15194	15453	15194
query12	157	107	105	105
query13	1415	439	429	429
query14	10395	7155	6932	6932
query15	209	171	192	171
query16	7235	476	501	476
query17	1206	597	595	595
query18	1872	313	317	313
query19	223	167	174	167
query20	112	111	112	111
query21	212	104	104	104
query22	4748	4389	4727	4389
query23	34665	34417	33904	33904
query24	6157	2881	2923	2881
query25	537	410	422	410
query26	690	171	167	167
query27	2064	359	373	359
query28	3927	2134	2111	2111
query29	715	470	463	463
query30	244	163	158	158
query31	1014	814	825	814
query32	74	55	62	55
query33	451	308	305	305
query34	941	510	502	502
query35	838	708	730	708
query36	1072	945	950	945
query37	116	66	67	66
query38	4065	4048	3940	3940
query39	1500	1447	1495	1447
query40	197	101	98	98
query41	50	45	48	45
query42	107	99	105	99
query43	534	471	488	471
query44	1168	798	804	798
query45	185	167	166	166
query46	1127	721	718	718
query47	2048	1898	1920	1898
query48	474	386	386	386
query49	725	391	374	374
query50	842	417	439	417
query51	7347	7111	7159	7111
query52	105	91	94	91
query53	258	188	198	188
query54	562	475	455	455
query55	76	79	75	75
query56	261	267	247	247
query57	1342	1216	1224	1216
query58	213	202	230	202
query59	3170	3063	2985	2985
query60	279	258	266	258
query61	110	113	133	113
query62	774	697	690	690
query63	214	189	186	186
query64	1361	655	612	612
query65	3246	3206	3205	3205
query66	706	292	308	292
query67	15950	15453	15570	15453
query68	4261	574	566	566
query69	433	271	262	262
query70	1139	1059	1107	1059
query71	346	261	252	252
query72	6391	4079	4014	4014
query73	752	345	363	345
query74	10245	9088	9202	9088
query75	3336	2617	2674	2617
query76	2071	1132	945	945
query77	535	269	259	259
query78	10568	9602	9500	9500
query79	2112	584	590	584
query80	1306	403	414	403
query81	527	215	220	215
query82	1242	88	91	88
query83	161	148	137	137
query84	284	78	77	77
query85	975	298	284	284
query86	376	261	299	261
query87	4428	4235	4211	4211
query88	3946	2406	2389	2389
query89	417	283	287	283
query90	1989	185	182	182
query91	184	148	147	147
query92	61	51	50	50
query93	2812	546	533	533
query94	776	295	281	281
query95	360	260	255	255
query96	652	281	281	281
query97	3301	3117	3144	3117
query98	209	197	199	197
query99	1599	1294	1277	1277
Total cold run time: 315701 ms
Total hot run time: 195859 ms

@doris-robot
Copy link

doris-robot commented Jun 19, 2025

ClickBench: Total hot run time: 29.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 33ddcf95a05149b933c77b74191f3678b2cc252d, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.52	0.51	0.50
query6	1.13	0.74	0.74
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.56	0.50	0.50
query10	0.56	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	0.78	0.80	0.80
query15	0.85	0.82	0.82
query16	0.36	0.38	0.41
query17	1.00	1.01	1.02
query18	0.24	0.21	0.21
query19	1.84	1.81	1.85
query20	0.01	0.01	0.02
query21	15.40	0.58	0.60
query22	2.23	1.80	1.47
query23	17.24	0.88	0.79
query24	2.93	1.06	1.13
query25	0.30	0.18	0.10
query26	0.31	0.14	0.14
query27	0.05	0.04	0.04
query28	10.61	0.49	0.50
query29	12.56	3.26	3.27
query30	0.24	0.06	0.06
query31	2.85	0.39	0.37
query32	3.27	0.46	0.45
query33	2.98	3.05	3.02
query34	17.19	4.45	4.48
query35	4.57	4.49	4.51
query36	0.64	0.48	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.58 s
Total hot run time: 29.93 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Jun 20, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor Author

github-actions bot commented Jun 20, 2025

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 3b868c9 into branch-3.1 Jun 20, 2025
23 of 26 checks passed
@morrySnow morrySnow deleted the auto-pick-44586-branch-3.1 branch June 20, 2025 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas @doris-robot @morrySnow @dataroaring @924060929