Skip to content

branch-3.0: [fix](nereids) fix sum0 cannot pass multi distinct check #51234 #52950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 4 commits into from
Jul 9, 2025
Merged

branch-3.0: [fix](nereids) fix sum0 cannot pass multi distinct check #51234 #52950

morrySnow merged 4 commits into from
Jul 9, 2025

Conversation

@feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Jul 8, 2025

cherry-pick #51234 to branch-3.0

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Jul 8, 2025

run buildall

2 similar comments
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Jul 8, 2025

run buildall

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Jul 8, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-H: Total hot run time: 39829 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit faa41fdd60e1e760f3816b88c8e02fa8a2838f38, data reload: false

------ Round 1 ----------------------------------
q1	17571	6933	6630	6630
q2	2088	196	184	184
q3	10588	1130	1108	1108
q4	10287	774	760	760
q5	7751	2919	2843	2843
q6	209	132	129	129
q7	973	620	619	619
q8	9366	1964	1957	1957
q9	6625	6387	6436	6387
q10	7029	2252	2268	2252
q11	457	262	252	252
q12	390	210	215	210
q13	17766	3009	3004	3004
q14	248	210	201	201
q15	509	466	473	466
q16	481	386	375	375
q17	989	580	596	580
q18	7353	6621	6530	6530
q19	1454	1090	1083	1083
q20	471	197	201	197
q21	3874	3100	3070	3070
q22	1104	992	1002	992
Total cold run time: 107583 ms
Total hot run time: 39829 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6622	6567	6539	6539
q2	322	226	231	226
q3	2905	2901	2949	2901
q4	2017	1839	1840	1839
q5	5677	5708	5749	5708
q6	207	127	130	127
q7	2202	1795	1855	1795
q8	3401	3570	3501	3501
q9	8874	8916	8859	8859
q10	3554	3504	3511	3504
q11	594	501	494	494
q12	804	590	597	590
q13	7342	3137	3165	3137
q14	306	259	278	259
q15	506	465	474	465
q16	498	443	435	435
q17	1834	1618	1598	1598
q18	8255	7892	7747	7747
q19	1701	1682	1507	1507
q20	2066	1828	1799	1799
q21	5170	4964	4891	4891
q22	1102	1044	1035	1035
Total cold run time: 65959 ms
Total hot run time: 58956 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

TPC-DS: Total hot run time: 196319 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit faa41fdd60e1e760f3816b88c8e02fa8a2838f38, data reload: false

query1	1305	934	884	884
query2	6197	1886	1858	1858
query3	10857	4337	4212	4212
query4	33117	23410	23482	23410
query5	3387	464	438	438
query6	269	179	187	179
query7	3979	312	320	312
query8	276	232	234	232
query9	9656	2588	2576	2576
query10	465	263	255	255
query11	18183	15194	15198	15194
query12	164	101	100	100
query13	1568	433	430	430
query14	8735	6668	7325	6668
query15	245	184	192	184
query16	8170	519	489	489
query17	1592	624	591	591
query18	2125	324	318	318
query19	220	167	174	167
query20	130	118	112	112
query21	210	102	111	102
query22	4929	4561	4321	4321
query23	35258	34062	34450	34062
query24	11843	2930	2916	2916
query25	705	434	430	430
query26	1850	170	174	170
query27	2924	356	352	352
query28	7798	2175	2180	2175
query29	1084	471	470	470
query30	263	178	163	163
query31	1035	856	833	833
query32	95	55	57	55
query33	765	285	306	285
query34	1009	527	518	518
query35	898	721	748	721
query36	1093	955	961	955
query37	268	64	67	64
query38	4127	3912	3939	3912
query39	1518	1507	1453	1453
query40	257	103	108	103
query41	49	50	45	45
query42	119	103	112	103
query43	523	483	495	483
query44	1297	823	807	807
query45	185	169	178	169
query46	1162	722	735	722
query47	1999	1891	1940	1891
query48	495	392	390	390
query49	1148	406	420	406
query50	834	439	428	428
query51	7415	7417	7392	7392
query52	98	95	92	92
query53	256	192	189	189
query54	1485	481	468	468
query55	94	78	82	78
query56	274	246	244	244
query57	1332	1191	1206	1191
query58	218	203	203	203
query59	3215	2906	2916	2906
query60	288	261	254	254
query61	113	111	112	111
query62	882	705	693	693
query63	216	190	194	190
query64	5036	671	635	635
query65	3386	3321	3272	3272
query66	1412	291	294	291
query67	16119	15486	15644	15486
query68	5586	604	575	575
query69	422	272	251	251
query70	1202	1124	1121	1121
query71	334	276	254	254
query72	6218	4096	4009	4009
query73	739	348	361	348
query74	10552	9295	9058	9058
query75	3345	2675	2639	2639
query76	3277	1108	1111	1108
query77	378	277	273	273
query78	10499	9685	9500	9500
query79	2214	579	607	579
query80	1121	416	441	416
query81	542	214	214	214
query82	912	89	87	87
query83	220	150	150	150
query84	230	78	79	78
query85	1373	298	298	298
query86	461	296	270	270
query87	4333	4255	4335	4255
query88	3847	2404	2378	2378
query89	417	289	289	289
query90	1959	185	185	185
query91	178	148	148	148
query92	67	49	48	48
query93	2274	556	551	551
query94	868	294	294	294
query95	356	259	253	253
query96	608	288	285	285
query97	3331	3126	3142	3126
query98	216	204	194	194
query99	1484	1315	1277	1277
Total cold run time: 307782 ms
Total hot run time: 196319 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2025

ClickBench: Total hot run time: 30.73 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit faa41fdd60e1e760f3816b88c8e02fa8a2838f38, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.63	0.11	0.10
query5	0.51	0.53	0.51
query6	1.13	0.73	0.74
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.55	0.56	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.59
query14	0.80	0.80	0.77
query15	0.84	0.82	0.82
query16	0.38	0.37	0.38
query17	1.03	1.06	1.01
query18	0.24	0.22	0.21
query19	1.93	1.85	1.89
query20	0.02	0.01	0.01
query21	15.39	0.57	0.56
query22	2.26	1.84	1.55
query23	17.02	0.86	0.99
query24	3.34	2.30	1.84
query25	0.23	0.05	0.07
query26	0.83	0.13	0.14
query27	0.05	0.04	0.04
query28	8.37	0.51	0.50
query29	12.61	3.21	3.16
query30	0.24	0.06	0.06
query31	2.87	0.39	0.38
query32	3.23	0.46	0.46
query33	2.94	3.02	3.02
query34	17.16	4.44	4.51
query35	4.54	4.48	4.47
query36	0.65	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.13 s
Total hot run time: 30.73 s

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Jul 9, 2025

run external

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Jul 9, 2025

run p0

@feiniaofeiafei
[fix](nereids) fix sum0 cannot pass multi distinct check (apache#51234)
6ce0fc9 
### What problem does this PR solve?

Related PR: apache#32541

Problem Summary:
Before this pr, execute below sql will report error: sum0(DISTINCT c#2) can't support multi distinct.
This pr change the check, and the sql can be executed. sql is:
select sum0(distinct b),sum(distinct c) from test_sum0_multi_distinct_with_group_by group by a
@feiniaofeiafei
[fix](nereids) fix sum0 cannot pass multi distinct check (apache#51234)
ab6b7df
@feiniaofeiafei
fix style
dfd33fc
@feiniaofeiafei
fix style
8adc248
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Jul 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 9, 2025

TPC-H: Total hot run time: 40251 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8adc248f0760415c7c6d82a8697e64cac8eeb289, data reload: false

------ Round 1 ----------------------------------
q1	17589	7015	6680	6680
q2	2355	188	210	188
q3	11563	1172	1214	1172
q4	11259	755	713	713
q5	7908	3010	2903	2903
q6	221	140	140	140
q7	1008	654	625	625
q8	10426	2015	2015	2015
q9	8342	6486	6491	6486
q10	7007	2269	2341	2269
q11	460	266	273	266
q12	397	209	210	209
q13	17778	2968	2950	2950
q14	240	211	221	211
q15	519	471	455	455
q16	491	389	380	380
q17	1000	542	617	542
q18	7457	6668	6787	6668
q19	1399	1116	1049	1049
q20	483	194	205	194
q21	4136	3166	3166	3166
q22	1103	970	987	970
Total cold run time: 113141 ms
Total hot run time: 40251 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6636	6628	6699	6628
q2	352	244	245	244
q3	3156	2963	2925	2925
q4	2086	1825	1767	1767
q5	5658	5734	5719	5719
q6	212	129	130	129
q7	2190	1807	1783	1783
q8	3346	3534	3580	3534
q9	8846	8775	8808	8775
q10	3574	3530	3529	3529
q11	604	513	496	496
q12	800	582	620	582
q13	4708	3134	3121	3121
q14	296	271	281	271
q15	522	472	459	459
q16	488	438	453	438
q17	1846	1627	1605	1605
q18	8098	7825	7944	7825
q19	1708	1621	1593	1593
q20	2130	1859	1844	1844
q21	5117	4916	4789	4789
q22	1075	998	976	976
Total cold run time: 63448 ms
Total hot run time: 59032 ms

@doris-robot
Copy link

doris-robot commented Jul 9, 2025

TPC-DS: Total hot run time: 189007 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8adc248f0760415c7c6d82a8697e64cac8eeb289, data reload: false

query1	960	378	385	378
query2	6515	1943	1947	1943
query3	6713	222	223	222
query4	34104	23929	23351	23351
query5	4313	451	456	451
query6	287	185	190	185
query7	4627	314	317	314
query8	293	223	218	218
query9	9702	2608	2595	2595
query10	490	267	265	265
query11	18410	15254	14993	14993
query12	159	98	99	98
query13	1624	427	409	409
query14	8799	6706	7135	6706
query15	230	174	178	174
query16	8298	459	466	459
query17	1627	576	566	566
query18	2082	308	317	308
query19	250	173	162	162
query20	117	109	114	109
query21	213	105	103	103
query22	4385	4178	4226	4178
query23	34153	33823	33548	33548
query24	11789	2875	2907	2875
query25	740	420	420	420
query26	1891	175	173	173
query27	3020	353	351	351
query28	8011	2128	2127	2127
query29	1116	465	452	452
query30	329	167	166	166
query31	1020	790	801	790
query32	104	62	66	62
query33	802	311	314	311
query34	924	507	528	507
query35	957	730	703	703
query36	1065	965	945	945
query37	260	76	75	75
query38	3964	3863	3876	3863
query39	1497	1407	1476	1407
query40	300	104	103	103
query41	53	52	53	52
query42	113	106	104	104
query43	525	473	492	473
query44	1272	793	812	793
query45	188	177	175	175
query46	1157	731	744	731
query47	1941	1816	1815	1815
query48	494	378	376	376
query49	1328	413	406	406
query50	839	412	414	412
query51	7394	7229	7090	7090
query52	102	93	95	93
query53	264	185	191	185
query54	1298	482	485	482
query55	86	85	78	78
query56	277	256	258	256
query57	1304	1170	1154	1154
query58	254	215	221	215
query59	3109	3063	2972	2972
query60	303	265	276	265
query61	172	139	156	139
query62	854	640	658	640
query63	225	198	196	196
query64	5410	644	625	625
query65	3305	3201	3237	3201
query66	1455	311	306	306
query67	15985	15639	15606	15606
query68	4612	565	587	565
query69	446	290	270	270
query70	1168	1110	1032	1032
query71	400	257	260	257
query72	6720	2565	3831	2565
query73	765	346	362	346
query74	10223	9159	8987	8987
query75	3361	2671	2671	2671
query76	2941	1060	1066	1060
query77	476	316	279	279
query78	10501	9779	9641	9641
query79	1081	595	597	595
query80	756	454	432	432
query81	484	221	221	221
query82	1277	92	86	86
query83	252	150	160	150
query84	239	81	83	81
query85	1112	332	281	281
query86	317	298	299	298
query87	4378	4259	4240	4240
query88	3467	2504	2361	2361
query89	397	298	300	298
query90	2090	187	188	187
query91	176	154	174	154
query92	60	52	55	52
query93	1075	559	550	550
query94	814	302	288	288
query95	364	261	266	261
query96	615	278	283	278
query97	3303	3131	3148	3131
query98	215	200	185	185
query99	1476	1331	1300	1300
Total cold run time: 301408 ms
Total hot run time: 189007 ms

@doris-robot
Copy link

doris-robot commented Jul 9, 2025

ClickBench: Total hot run time: 29.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8adc248f0760415c7c6d82a8697e64cac8eeb289, data reload: false

query1	0.03	0.04	0.03
query2	0.08	0.03	0.03
query3	0.23	0.06	0.07
query4	1.63	0.10	0.10
query5	0.50	0.50	0.50
query6	1.12	0.72	0.74
query7	0.03	0.03	0.02
query8	0.04	0.03	0.03
query9	0.56	0.49	0.48
query10	0.54	0.55	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.60	0.60	0.60
query14	0.77	0.79	0.80
query15	0.86	0.82	0.83
query16	0.40	0.40	0.38
query17	1.08	1.08	1.02
query18	0.22	0.22	0.21
query19	1.96	1.85	1.78
query20	0.01	0.01	0.02
query21	15.40	0.58	0.59
query22	2.23	1.44	1.69
query23	16.98	0.85	0.80
query24	2.95	1.28	0.93
query25	0.22	0.22	0.08
query26	0.35	0.14	0.13
query27	0.04	0.04	0.04
query28	10.40	0.49	0.51
query29	12.60	3.18	3.16
query30	0.25	0.05	0.06
query31	2.85	0.38	0.39
query32	3.24	0.47	0.46
query33	2.98	3.00	2.99
query34	17.16	4.54	4.50
query35	4.49	4.51	4.51
query36	0.67	0.48	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.23 s
Total hot run time: 29.65 s

dataroaring
dataroaring approved these changes Jul 9, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 9, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jul 9, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 9, 2025

PR approved by anyone and no changes requested.

@morrySnow morrySnow changed the title [fix](nereids) fix sum0 cannot pass multi distinct check (#51234) branch-3.0: [fix](nereids) fix sum0 cannot pass multi distinct check #51234 Jul 9, 2025
@morrySnow morrySnow merged commit 916da87 into apache:branch-3.0 Jul 9, 2025
21 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@feiniaofeiafei @hello-stephen @doris-robot @dataroaring @morrySnow