Skip to content

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Nov 24, 2025

What problem does this PR solve?

PushDownTopNThroughJoin
PushDownLimitDistinctThroughJoin
PushDownTopNDistinctThroughJoin
in pr #46773, we made above rules both applicable to RBO and CBO. In order to avoid dead loop, we have checked rewritten plan. But the check condition is not suitable to CBO. This leads optimizor apply these rules until group expression count reach memo_max_group_expression_size

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@englefly
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

TPC-H: Total hot run time: 35057 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ba8b6247d54864766d87783d59f0c00456d3d27, data reload: false

------ Round 1 ----------------------------------
q1	17604	5040	4916	4916
q2	2017	308	204	204
q3	10250	1320	745	745
q4	10237	892	357	357
q5	7510	2362	2346	2346
q6	181	174	137	137
q7	917	794	624	624
q8	9352	1383	1175	1175
q9	7088	5427	5482	5427
q10	6832	2244	1803	1803
q11	506	310	283	283
q12	332	366	226	226
q13	17790	3611	3021	3021
q14	246	232	218	218
q15	584	508	502	502
q16	1032	1019	941	941
q17	581	868	367	367
q18	7523	7533	8092	7533
q19	1252	949	550	550
q20	360	361	242	242
q21	4420	3491	2423	2423
q22	1119	1063	1017	1017
Total cold run time: 107733 ms
Total hot run time: 35057 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5197	5278	5105	5105
q2	311	398	311	311
q3	2380	2967	2476	2476
q4	1478	1882	1476	1476
q5	4615	4441	4509	4441
q6	212	174	127	127
q7	2005	1965	1832	1832
q8	2579	2529	2468	2468
q9	7632	7750	7528	7528
q10	3059	3282	2773	2773
q11	572	510	491	491
q12	646	702	572	572
q13	3239	3630	3026	3026
q14	263	279	262	262
q15	522	498	474	474
q16	996	1040	977	977
q17	1069	1470	1316	1316
q18	7365	7182	7078	7078
q19	749	767	816	767
q20	1997	1952	1821	1821
q21	4666	4332	4385	4332
q22	1107	1046	1008	1008
Total cold run time: 52659 ms
Total hot run time: 50661 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187349 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ba8b6247d54864766d87783d59f0c00456d3d27, data reload: false

query1	1049	412	421	412
query2	6560	1690	1684	1684
query3	6764	220	220	220
query4	27017	22933	22626	22626
query5	4446	659	478	478
query6	324	227	221	221
query7	4641	493	305	305
query8	296	247	243	243
query9	8685	2622	2615	2615
query10	486	371	293	293
query11	16029	14947	14799	14799
query12	185	116	113	113
query13	1689	566	458	458
query14	11220	9156	9020	9020
query15	193	191	182	182
query16	7684	699	511	511
query17	1243	737	625	625
query18	2024	427	336	336
query19	259	201	176	176
query20	126	125	123	123
query21	221	135	115	115
query22	4085	4190	4101	4101
query23	34018	32958	33387	32958
query24	8516	2371	2364	2364
query25	583	524	455	455
query26	1226	283	157	157
query27	2737	497	355	355
query28	4353	2208	2196	2196
query29	784	637	483	483
query30	301	231	199	199
query31	931	815	713	713
query32	90	73	71	71
query33	588	383	318	318
query34	806	859	525	525
query35	818	829	771	771
query36	932	977	908	908
query37	117	109	87	87
query38	3511	3579	3509	3509
query39	1483	1420	1433	1420
query40	241	131	123	123
query41	67	80	62	62
query42	122	108	111	108
query43	492	511	472	472
query44	1242	772	774	772
query45	188	179	177	177
query46	886	981	645	645
query47	1761	1773	1725	1725
query48	406	414	320	320
query49	761	476	419	419
query50	648	684	396	396
query51	3921	3941	3848	3848
query52	116	110	102	102
query53	236	261	195	195
query54	320	300	279	279
query55	85	89	88	88
query56	310	329	338	329
query57	1182	1195	1118	1118
query58	305	290	287	287
query59	2519	2634	2611	2611
query60	374	360	340	340
query61	212	199	188	188
query62	800	711	685	685
query63	228	196	210	196
query64	4601	1277	978	978
query65	4034	3968	3965	3965
query66	1121	454	349	349
query67	15556	15108	14937	14937
query68	7426	947	639	639
query69	507	333	285	285
query70	1327	1298	1310	1298
query71	439	345	330	330
query72	6144	4861	4888	4861
query73	658	581	369	369
query74	9224	9039	8985	8985
query75	3325	3341	2824	2824
query76	3319	1138	715	715
query77	518	411	318	318
query78	9696	9673	8830	8830
query79	2590	840	618	618
query80	759	601	502	502
query81	528	261	235	235
query82	502	155	129	129
query83	274	266	255	255
query84	262	117	93	93
query85	888	481	456	456
query86	416	310	278	278
query87	3687	3637	3555	3555
query88	4051	2217	2253	2217
query89	380	326	296	296
query90	1971	224	223	223
query91	173	167	137	137
query92	80	68	64	64
query93	2588	1018	669	669
query94	807	443	346	346
query95	400	326	320	320
query96	480	569	280	280
query97	2882	2982	2895	2895
query98	244	218	214	214
query99	1292	1399	1254	1254
Total cold run time: 277591 ms
Total hot run time: 187349 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3ba8b6247d54864766d87783d59f0c00456d3d27, data reload: false

query1	0.05	0.05	0.04
query2	0.10	0.04	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.11
query5	0.28	0.26	0.25
query6	1.21	0.65	0.64
query7	0.03	0.02	0.03
query8	0.05	0.04	0.04
query9	0.59	0.53	0.52
query10	0.58	0.57	0.58
query11	0.16	0.11	0.12
query12	0.14	0.12	0.11
query13	0.62	0.60	0.60
query14	1.01	1.00	0.99
query15	0.85	0.83	0.84
query16	0.39	0.39	0.42
query17	0.98	1.08	1.01
query18	0.21	0.20	0.19
query19	1.96	1.79	1.78
query20	0.02	0.01	0.01
query21	15.50	0.18	0.13
query22	5.08	0.07	0.05
query23	15.69	0.26	0.10
query24	1.84	1.58	1.28
query25	0.07	0.07	0.08
query26	0.15	0.13	0.14
query27	0.06	0.06	0.06
query28	5.71	1.13	0.93
query29	12.56	3.90	3.21
query30	0.28	0.13	0.11
query31	2.81	0.58	0.39
query32	3.23	0.57	0.46
query33	3.07	3.15	3.11
query34	15.89	5.20	4.59
query35	4.54	4.56	4.55
query36	0.67	0.50	0.48
query37	0.10	0.06	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.13	0.15
query41	0.09	0.04	0.03
query42	0.04	0.04	0.03
query43	0.04	0.03	0.04
Total cold run time: 98.79 s
Total hot run time: 28.33 s

Copy link
Contributor

@morrySnow morrySnow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add ut and regression test

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 43.48% (10/23) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 78.26% (18/23) 🎉
Increment coverage report
Complete coverage report

@englefly
Copy link
Contributor Author

run p0

@englefly
Copy link
Contributor Author

run nonConcurrent

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 78.26% (18/23) 🎉
Increment coverage report
Complete coverage report

@englefly englefly force-pushed the pushTopnJoinDeadLoop branch from 3ba8b62 to cf5b09a Compare December 1, 2025 03:27
@englefly
Copy link
Contributor Author

englefly commented Dec 1, 2025

run buildall

@englefly englefly force-pushed the pushTopnJoinDeadLoop branch from 3918932 to f5179e7 Compare December 1, 2025 03:31
@englefly
Copy link
Contributor Author

englefly commented Dec 1, 2025

run buildall

Copy link
Contributor

@seawinde seawinde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 34303 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f5179e7ddb5c36793f743f09b35894df766655a8, data reload: false

------ Round 1 ----------------------------------
q1	17653	4990	4899	4899
q2	2018	357	202	202
q3	10227	1309	741	741
q4	10235	932	375	375
q5	7542	2503	2236	2236
q6	191	167	133	133
q7	978	786	643	643
q8	9352	1368	1110	1110
q9	6965	5344	5292	5292
q10	6819	2248	1807	1807
q11	504	306	281	281
q12	341	377	234	234
q13	17772	3672	3100	3100
q14	244	245	211	211
q15	581	526	515	515
q16	853	868	828	828
q17	604	773	528	528
q18	7300	6979	7110	6979
q19	896	953	579	579
q20	361	342	230	230
q21	2928	3210	2432	2432
q22	1014	1010	948	948
Total cold run time: 105378 ms
Total hot run time: 34303 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4953	4955	4930	4930
q2	340	397	321	321
q3	2138	2695	2298	2298
q4	1328	1739	1277	1277
q5	4245	4226	4666	4226
q6	233	179	132	132
q7	2083	2034	1836	1836
q8	2613	2537	2558	2537
q9	7474	7479	7715	7479
q10	3075	3293	2829	2829
q11	598	526	500	500
q12	687	766	643	643
q13	3611	3927	3388	3388
q14	291	303	281	281
q15	545	516	528	516
q16	907	1016	937	937
q17	1167	1328	1458	1328
q18	8028	7627	7549	7549
q19	816	783	815	783
q20	2073	2101	1884	1884
q21	4956	4582	4178	4178
q22	1078	1025	966	966
Total cold run time: 53239 ms
Total hot run time: 50818 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182726 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f5179e7ddb5c36793f743f09b35894df766655a8, data reload: false

query1	1043	421	399	399
query2	6630	1176	1152	1152
query3	6758	234	231	231
query4	25474	23257	23158	23158
query5	4680	645	513	513
query6	350	239	228	228
query7	4643	523	317	317
query8	312	247	262	247
query9	8737	2589	2629	2589
query10	509	374	331	331
query11	15552	15285	14951	14951
query12	186	120	120	120
query13	1707	589	456	456
query14	9523	6055	6123	6055
query15	215	206	189	189
query16	7553	699	539	539
query17	1241	777	640	640
query18	2050	446	352	352
query19	244	242	199	199
query20	141	125	127	125
query21	216	138	117	117
query22	3837	3893	3823	3823
query23	33125	32093	31955	31955
query24	8592	2448	2460	2448
query25	648	555	506	506
query26	1253	275	166	166
query27	2731	501	345	345
query28	4332	2158	2134	2134
query29	817	627	503	503
query30	306	237	204	204
query31	833	710	625	625
query32	81	78	75	75
query33	601	389	340	340
query34	839	937	555	555
query35	833	832	737	737
query36	905	935	825	825
query37	124	110	87	87
query38	3929	3902	3799	3799
query39	1464	1406	1415	1406
query40	228	136	126	126
query41	70	66	62	62
query42	129	114	113	113
query43	440	445	420	420
query44	1310	762	757	757
query45	199	186	179	179
query46	879	1000	653	653
query47	1667	1726	1687	1687
query48	403	437	337	337
query49	822	499	424	424
query50	663	694	415	415
query51	3954	3924	3885	3885
query52	116	119	109	109
query53	247	271	198	198
query54	321	319	288	288
query55	99	96	97	96
query56	339	332	334	332
query57	1142	1177	1099	1099
query58	284	282	285	282
query59	2271	2369	2281	2281
query60	377	350	339	339
query61	167	165	164	164
query62	786	706	683	683
query63	231	195	201	195
query64	4604	1221	921	921
query65	4061	3953	3969	3953
query66	1130	458	338	338
query67	15043	15164	14725	14725
query68	8297	948	629	629
query69	528	348	314	314
query70	1084	1018	1040	1018
query71	428	351	321	321
query72	5861	5025	5075	5025
query73	655	578	351	351
query74	8911	8715	8618	8618
query75	3056	3043	2534	2534
query76	3410	1164	743	743
query77	535	424	326	326
query78	9446	9817	8928	8928
query79	2500	822	580	580
query80	693	585	508	508
query81	508	276	245	245
query82	217	138	114	114
query83	268	271	255	255
query84	262	121	101	101
query85	957	508	453	453
query86	387	297	301	297
query87	4065	4097	4024	4024
query88	4309	2332	2330	2330
query89	405	336	299	299
query90	1952	233	234	233
query91	183	172	145	145
query92	81	70	70	70
query93	2182	1014	661	661
query94	733	446	349	349
query95	526	411	409	409
query96	525	553	290	290
query97	2620	2680	2572	2572
query98	260	218	209	209
query99	1338	1407	1273	1273
Total cold run time: 272178 ms
Total hot run time: 182726 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 27.38 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f5179e7ddb5c36793f743f09b35894df766655a8, data reload: false

query1	0.06	0.04	0.05
query2	0.10	0.04	0.05
query3	0.25	0.09	0.09
query4	1.65	0.11	0.10
query5	0.27	0.25	0.27
query6	1.16	0.66	0.63
query7	0.03	0.03	0.02
query8	0.04	0.04	0.05
query9	0.58	0.51	0.49
query10	0.57	0.56	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.63	0.61	0.60
query14	0.98	0.99	0.98
query15	0.82	0.80	0.81
query16	0.39	0.39	0.39
query17	1.00	0.99	1.07
query18	0.23	0.24	0.21
query19	1.99	1.77	1.86
query20	0.02	0.02	0.02
query21	15.46	0.25	0.13
query22	5.04	0.05	0.05
query23	16.17	0.26	0.11
query24	0.94	0.24	0.62
query25	0.10	0.08	0.07
query26	0.14	0.14	0.13
query27	0.06	0.05	0.06
query28	4.10	1.21	1.03
query29	12.60	3.97	3.27
query30	0.28	0.14	0.11
query31	2.82	0.62	0.42
query32	3.23	0.54	0.46
query33	3.07	3.04	3.13
query34	16.99	5.30	4.54
query35	4.56	4.73	4.67
query36	0.67	0.50	0.49
query37	0.10	0.07	0.06
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.05	0.03	0.04
Total cold run time: 97.86 s
Total hot run time: 27.38 s

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 78.26% (18/23) 🎉
Increment coverage report
Complete coverage report

@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 1, 2025
@morrySnow morrySnow merged commit 1a39f5d into apache:master Dec 1, 2025
28 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 1, 2025
- PushDownTopNThroughJoin
- PushDownLimitDistinctThroughJoin
- PushDownTopNDistinctThroughJoin
in pr#46773, we made above rules both applicable to RBO and CBO. In
order to avoid dead loop, we have checked rewritten plan. But the check
condition is not suitable to CBO. This leads optimizor apply these rules
until group expression count reach memo_max_group_expression_size
yiguolei pushed a commit that referenced this pull request Dec 2, 2025
…58305 (#58564)

Cherry-picked from #58305

Co-authored-by: minghong <zhouminghong@selectdb.com>
englefly added a commit to englefly/incubator-doris that referenced this pull request Dec 4, 2025
- PushDownTopNThroughJoin
- PushDownLimitDistinctThroughJoin
- PushDownTopNDistinctThroughJoin
in pr#46773, we made above rules both applicable to RBO and CBO. In
order to avoid dead loop, we have checked rewritten plan. But the check
condition is not suitable to CBO. This leads optimizor apply these rules
until group expression count reach memo_max_group_expression_size
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
- PushDownTopNThroughJoin
- PushDownLimitDistinctThroughJoin
- PushDownTopNDistinctThroughJoin
in pr#46773, we made above rules both applicable to RBO and CBO. In
order to avoid dead loop, we have checked rewritten plan. But the check
condition is not suitable to CBO. This leads optimizor apply these rules
until group expression count reach memo_max_group_expression_size
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.2-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants