Skip to content

branch-3.1: [enhance](runtime filter) impl partition pruning in runtime filer (#47025) #52049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jun 20, 2025
Merged

branch-3.1: [enhance](runtime filter) impl partition pruning in runtime filer (#47025) #52049

morrySnow merged 1 commit into from
Jun 20, 2025

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Jun 20, 2025

bp #47025

@suxiaogang223
[enhance](runtime filter) impl partition pruning in runtime filer (ap…
89da227 
…ache#47025)

This PR implements partition pruning through runtime filters. When
executing a SQL query like:
```sql
SELECT count(*)
FROM int_partition_table
WHERE partition_col = (
    SELECT partition_col
    FROM int_partition_table
    GROUP BY partition_col
    HAVING count(*) > 0
    ORDER BY partition_col DESC
    LIMIT 1
)
```
During execution, the backend (BE) will receive a dynamic runtime filter
condition `partition_col = xxx`. Since partition_col is a partitioning
column, we can use its value to determine if the partition can be
pruned.

Additionally, this mechanism also supports filtering queries like:
```sql
SELECT count(*)
FROM int_partition_table
WHERE func(partition_col) = xxx
```
If func cannot be evaluated at the frontend (FE), the frontend will not
perform partition pruning. However, since the backend can compute func,
this mechanism allows us to handle pruning scenarios that are not
possible at the frontend, providing a more efficient pruning process on
the backend side.
@suxiaogang223 suxiaogang223 requested a review from morrySnow as a code owner June 20, 2025 07:21
@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jun 20, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 20, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.19% (1099/1321)
Line Coverage 66.44% (18504/27852)
Region Coverage 66.18% (9155/13833)
Branch Coverage 56.01% (4948/8834)

@doris-robot
Copy link

doris-robot commented Jun 20, 2025

TPC-H: Total hot run time: 39991 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 89da2278073a61c94a1cc0cbfc1a1f19ab11afa9, data reload: false

------ Round 1 ----------------------------------
q1	17684	7177	6605	6605
q2	2088	174	159	159
q3	10659	1125	1144	1125
q4	10563	731	731	731
q5	7774	2848	2851	2848
q6	222	140	139	139
q7	953	610	620	610
q8	9350	1936	2028	1936
q9	6620	6458	6434	6434
q10	7042	2280	2350	2280
q11	466	259	266	259
q12	397	211	206	206
q13	17777	2976	2986	2976
q14	232	213	215	213
q15	502	471	465	465
q16	471	388	372	372
q17	990	536	597	536
q18	7349	6717	6652	6652
q19	1492	1102	1092	1092
q20	482	196	202	196
q21	4036	3157	3194	3157
q22	1123	1000	1031	1000
Total cold run time: 108272 ms
Total hot run time: 39991 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6603	6588	6630	6588
q2	329	233	239	233
q3	2908	2782	2901	2782
q4	2043	1815	1773	1773
q5	5695	5758	5705	5705
q6	220	131	133	131
q7	2278	1795	1829	1795
q8	3388	3577	3516	3516
q9	8905	8730	8876	8730
q10	3566	3590	3516	3516
q11	589	499	491	491
q12	788	599	619	599
q13	8688	3140	3183	3140
q14	318	267	271	267
q15	504	462	467	462
q16	486	441	420	420
q17	1860	1624	1627	1624
q18	8364	7891	7878	7878
q19	1726	1607	1725	1607
q20	2143	1823	1819	1819
q21	5142	5098	4999	4999
q22	1141	1065	1043	1043
Total cold run time: 67684 ms
Total hot run time: 59118 ms

@doris-robot
Copy link

doris-robot commented Jun 20, 2025

TPC-DS: Total hot run time: 197733 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 89da2278073a61c94a1cc0cbfc1a1f19ab11afa9, data reload: false

query1	1304	923	916	916
query2	6326	2058	1999	1999
query3	10985	4536	4469	4469
query4	33052	23679	23613	23613
query5	3501	471	453	453
query6	297	189	172	172
query7	3991	313	316	313
query8	282	212	210	210
query9	9471	2573	2565	2565
query10	464	259	254	254
query11	17959	15202	15212	15202
query12	160	105	100	100
query13	1546	442	413	413
query14	8467	7221	7042	7042
query15	273	179	191	179
query16	8106	512	503	503
query17	1619	597	598	597
query18	2183	321	320	320
query19	262	169	166	166
query20	125	113	115	113
query21	209	112	111	111
query22	4849	4462	4404	4404
query23	34933	34709	34127	34127
query24	11242	2890	2896	2890
query25	646	449	424	424
query26	1224	176	172	172
query27	2372	354	378	354
query28	7316	2189	2198	2189
query29	886	485	463	463
query30	270	170	164	164
query31	1076	822	811	811
query32	98	58	57	57
query33	748	337	303	303
query34	913	510	529	510
query35	846	751	730	730
query36	1132	940	943	940
query37	126	61	72	61
query38	4041	3942	3956	3942
query39	1514	1540	1429	1429
query40	202	102	100	100
query41	51	49	49	49
query42	117	103	99	99
query43	547	482	504	482
query44	1301	819	826	819
query45	187	180	168	168
query46	1176	739	736	736
query47	2034	1894	1940	1894
query48	500	381	404	381
query49	980	390	393	390
query50	833	418	436	418
query51	7459	7313	7305	7305
query52	98	89	90	89
query53	268	184	184	184
query54	1192	475	475	475
query55	80	81	79	79
query56	275	272	258	258
query57	1338	1192	1183	1183
query58	228	226	223	223
query59	3273	3047	3049	3047
query60	303	257	252	252
query61	187	114	117	114
query62	842	746	680	680
query63	225	192	194	192
query64	4635	666	633	633
query65	3350	3307	3308	3307
query66	745	302	306	302
query67	15978	15619	15621	15619
query68	4641	569	561	561
query69	416	268	268	268
query70	1164	1108	1130	1108
query71	332	256	269	256
query72	6324	4052	4036	4036
query73	761	349	350	349
query74	10047	9002	8903	8903
query75	3370	2660	2636	2636
query76	2718	1014	1128	1014
query77	378	270	272	270
query78	10599	9752	9680	9680
query79	1614	599	596	596
query80	1131	436	426	426
query81	545	222	224	222
query82	913	89	89	89
query83	230	138	144	138
query84	236	84	81	81
query85	1139	315	308	308
query86	363	312	297	297
query87	4383	4281	4229	4229
query88	3587	2385	2367	2367
query89	415	294	290	290
query90	1812	185	184	184
query91	178	148	154	148
query92	57	50	50	50
query93	1750	555	553	553
query94	679	301	303	301
query95	356	261	266	261
query96	620	282	274	274
query97	3341	3175	3162	3162
query98	211	201	202	201
query99	1541	1320	1317	1317
Total cold run time: 299069 ms
Total hot run time: 197733 ms

@doris-robot
Copy link

doris-robot commented Jun 20, 2025

ClickBench: Total hot run time: 31.39 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 89da2278073a61c94a1cc0cbfc1a1f19ab11afa9, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.65	0.08	0.08
query5	0.53	0.52	0.51
query6	1.12	0.74	0.75
query7	0.05	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.55	0.54	0.54
query11	0.17	0.12	0.12
query12	0.18	0.14	0.12
query13	0.61	0.60	0.59
query14	0.79	0.81	0.82
query15	0.86	0.85	0.83
query16	0.38	0.38	0.38
query17	1.03	1.06	0.97
query18	0.19	0.19	0.20
query19	1.81	1.76	1.77
query20	0.02	0.01	0.01
query21	15.40	0.68	0.67
query22	4.06	6.24	2.34
query23	18.23	1.41	1.32
query24	2.19	0.24	0.22
query25	0.15	0.09	0.09
query26	0.28	0.17	0.17
query27	0.08	0.08	0.08
query28	13.27	0.59	0.58
query29	12.61	3.41	3.40
query30	0.25	0.06	0.05
query31	2.89	0.43	0.41
query32	3.24	0.48	0.48
query33	3.02	2.99	3.04
query34	17.10	4.54	4.56
query35	4.72	4.70	4.61
query36	0.66	0.48	0.48
query37	0.19	0.17	0.17
query38	0.16	0.15	0.16
query39	0.06	0.04	0.05
query40	0.16	0.13	0.13
query41	0.11	0.06	0.05
query42	0.06	0.06	0.06
query43	0.05	0.06	0.05
Total cold run time: 109.83 s
Total hot run time: 31.39 s

@doris-robot
Copy link

doris-robot commented Jun 20, 2025

BE UT Coverage Report

Increment line coverage 8.94% (11/123) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.14% (10900/26494)
Line Coverage 31.95% (93243/291802)
Region Coverage 31.02% (48080/155002)
Branch Coverage 27.50% (24635/89582)

@morrySnow morrySnow merged commit 842c5c0 into apache:branch-3.1 Jun 20, 2025
20 of 22 checks passed
@suxiaogang223 suxiaogang223 deleted the partition_prune_runtime_filter_31 branch June 24, 2025 09:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@suxiaogang223 @hello-stephen @doris-robot @morrySnow