Skip to content

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jan 2, 2025

Cherry-picked from #46261

improve prune partition with lots of `in (xxx)`

this can reduce from 35s to 300ms when there have 4000 options
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Jan 2, 2025
@dataroaring dataroaring reopened this Jan 2, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40666 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eaa8e5f92478b33408b2dafe2f537b9eb1eea397, data reload: false

------ Round 1 ----------------------------------
q1	17575	7391	7218	7218
q2	2055	180	171	171
q3	10591	1063	1181	1063
q4	10635	752	797	752
q5	7854	2819	2738	2738
q6	242	148	141	141
q7	971	633	618	618
q8	9366	1959	2002	1959
q9	6503	6352	6376	6352
q10	7034	2321	2297	2297
q11	480	278	275	275
q12	405	220	215	215
q13	17776	2978	3006	2978
q14	245	209	228	209
q15	587	523	511	511
q16	678	611	597	597
q17	990	599	487	487
q18	7238	6696	6778	6696
q19	1386	1118	1003	1003
q20	453	212	200	200
q21	3947	3186	3318	3186
q22	1111	1003	1000	1000
Total cold run time: 108122 ms
Total hot run time: 40666 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7307	7214	7209	7209
q2	328	232	239	232
q3	2915	2925	2913	2913
q4	2023	1819	1824	1819
q5	5758	5688	5729	5688
q6	222	139	144	139
q7	2250	1829	1838	1829
q8	3337	3549	3451	3451
q9	8799	8921	8786	8786
q10	3591	3574	3514	3514
q11	590	498	498	498
q12	870	629	639	629
q13	9883	3172	3147	3147
q14	315	270	278	270
q15	565	538	514	514
q16	698	668	672	668
q17	1844	1626	1582	1582
q18	8222	7764	7646	7646
q19	1646	1394	1597	1394
q20	2103	1856	1882	1856
q21	5474	5481	5445	5445
q22	1126	1014	1067	1014
Total cold run time: 69866 ms
Total hot run time: 60243 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197743 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eaa8e5f92478b33408b2dafe2f537b9eb1eea397, data reload: false

query1	1272	919	912	912
query2	6231	2032	2070	2032
query3	10796	4371	4339	4339
query4	65810	28452	23511	23511
query5	5017	447	439	439
query6	397	171	172	171
query7	5616	311	319	311
query8	305	235	228	228
query9	9383	2706	2699	2699
query10	484	279	247	247
query11	17409	15229	15957	15229
query12	156	99	100	99
query13	1550	462	447	447
query14	10605	7137	6960	6960
query15	201	184	180	180
query16	6678	471	511	471
query17	1099	614	584	584
query18	1713	323	336	323
query19	207	166	166	166
query20	124	114	113	113
query21	59	48	43	43
query22	4777	4644	4538	4538
query23	34755	34191	34355	34191
query24	6085	2894	2963	2894
query25	540	433	425	425
query26	655	177	169	169
query27	1683	313	308	308
query28	4399	2471	2461	2461
query29	728	476	456	456
query30	243	176	165	165
query31	1015	871	842	842
query32	62	56	53	53
query33	392	288	283	283
query34	899	510	526	510
query35	866	747	734	734
query36	1086	975	985	975
query37	122	70	76	70
query38	4096	4085	4125	4085
query39	1496	1466	1472	1466
query40	138	83	83	83
query41	53	52	50	50
query42	117	102	95	95
query43	543	509	483	483
query44	1182	835	831	831
query45	190	179	171	171
query46	1163	736	756	736
query47	2036	1980	1930	1930
query48	481	372	386	372
query49	727	391	383	383
query50	836	420	427	420
query51	7364	7351	7296	7296
query52	102	85	86	85
query53	256	177	180	177
query54	552	443	442	442
query55	80	78	75	75
query56	262	245	246	245
query57	1212	1111	1095	1095
query58	208	199	201	199
query59	3124	2969	2908	2908
query60	278	255	253	253
query61	107	110	114	110
query62	778	662	659	659
query63	215	188	193	188
query64	1397	690	639	639
query65	3255	3157	3241	3157
query66	717	305	305	305
query67	15915	15707	15552	15552
query68	4080	597	569	569
query69	454	282	264	264
query70	1181	1113	1106	1106
query71	360	275	263	263
query72	6490	4107	3996	3996
query73	748	351	348	348
query74	10185	8944	8975	8944
query75	3339	2651	2693	2651
query76	1866	1123	1080	1080
query77	503	275	282	275
query78	10439	9606	9656	9606
query79	1667	611	601	601
query80	1102	421	417	417
query81	533	243	238	238
query82	206	116	117	116
query83	162	150	149	149
query84	279	83	81	81
query85	999	313	293	293
query86	392	302	304	302
query87	4440	4280	4314	4280
query88	3897	2383	2363	2363
query89	406	295	293	293
query90	2036	191	192	191
query91	188	147	153	147
query92	65	50	49	49
query93	2353	552	553	552
query94	878	304	304	304
query95	364	259	254	254
query96	615	278	280	278
query97	3350	3248	3170	3170
query98	211	203	196	196
query99	1610	1299	1317	1299
Total cold run time: 317697 ms
Total hot run time: 197743 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eaa8e5f92478b33408b2dafe2f537b9eb1eea397, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.07	0.07
query4	1.63	0.11	0.11
query5	0.51	0.52	0.51
query6	1.14	0.73	0.74
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.49	0.51
query10	0.56	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.58	0.58
query14	2.92	2.96	3.02
query15	0.89	0.83	0.82
query16	0.39	0.38	0.37
query17	1.00	0.99	1.06
query18	0.24	0.21	0.22
query19	1.94	1.89	2.02
query20	0.01	0.01	0.02
query21	15.36	0.61	0.60
query22	3.11	2.34	1.47
query23	16.80	0.99	0.98
query24	2.77	0.89	1.67
query25	0.16	0.11	0.16
query26	0.51	0.14	0.14
query27	0.04	0.06	0.05
query28	10.46	1.11	1.07
query29	12.55	3.24	3.24
query30	0.24	0.06	0.06
query31	2.88	0.40	0.39
query32	3.25	0.46	0.47
query33	2.98	2.98	3.07
query34	17.14	4.46	4.42
query35	4.50	4.48	4.50
query36	0.64	0.48	0.48
query37	0.10	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.98 s
Total hot run time: 32.73 s

@morrySnow morrySnow merged commit 04b18d9 into branch-3.0 Jan 3, 2025
21 checks passed
@morrySnow morrySnow deleted the auto-pick-46261-branch-3.0 branch January 3, 2025 11:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants