branch-3.0: [fix](nereids) do eliminate constant group by key in normalizeagg #49589 #50215

feiniaofeiafei · 2025-04-21T04:22:10Z

Cherry-picked from #49589

…ache#49589) ### What problem does this PR solve? Related PR: apache#32878 apache#49473 Problem Summary: SELECT IF( t.`gender` IN ('女'), ( TIMESTAMPDIFF( YEAR, NOW(), NOW() ) ), 1 ) AS x0, TIMESTAMPDIFF( YEAR, NOW(), NOW() ) AS x1 FROM t1 AS t GROUP BY x0, x1; after EliminateGroupByConstant, this sql will be rewritten to SELECT IF( t.`gender` IN ('女'), 0, 1 ) AS x0, 0 AS x1 FROM t1 AS t GROUP BY IF( t.`gender` IN ('女'), ( TIMESTAMPDIFF( YEAR, NOW(), NOW() ) ), 1 ) ; The select expression and the group by expression is different, and will report error in normalizeagg. The fix in PR apache#49473 may introduce another issue. Consider the following query: SELECT func2(100) FROM t GROUP BY func1(), func2(func1()); If func1() can be constant-folded to 100, then func2(func1()) will be replaced with func2(100), allowing the query to execute successfully. However, when func1() cannot be folded to 100, the query will fail. This creates an inconsistent behavior where query execution depends on whether func1() can be constant-folded or not, which is not an ideal implementation. To address this issue, this PR modifies the normalizeAgg logic to eliminate constant group by keys. With this change, the query will consistently fail regardless of whether func1() can be folded or not, ensuring more predictable behavior.

hello-stephen · 2025-04-21T04:22:15Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

feiniaofeiafei · 2025-04-21T06:09:03Z

run buildall

feiniaofeiafei · 2025-04-21T06:10:09Z

run buildall

doris-robot · 2025-04-21T06:47:35Z

TPC-H: Total hot run time: 39566 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a96181b3988065942c0784cb9b01f2d18d7db3e3, data reload: false

------ Round 1 ----------------------------------
q1	17584	6801	6582	6582
q2	2058	162	180	162
q3	10619	1085	1168	1085
q4	10563	783	733	733
q5	7772	2826	2773	2773
q6	212	133	130	130
q7	960	611	598	598
q8	9348	1932	2152	1932
q9	6528	6321	6312	6312
q10	7037	2268	2282	2268
q11	473	264	258	258
q12	425	215	204	204
q13	17791	2964	2958	2958
q14	237	212	213	212
q15	529	451	453	451
q16	657	606	569	569
q17	966	494	609	494
q18	7236	6615	6519	6519
q19	1377	1084	1013	1013
q20	483	200	206	200
q21	4002	3143	3241	3143
q22	1115	997	970	970
Total cold run time: 107972 ms
Total hot run time: 39566 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6605	6586	6545	6545
q2	331	237	229	229
q3	2893	2731	2808	2731
q4	2053	1771	1812	1771
q5	5863	5801	5846	5801
q6	209	123	134	123
q7	2256	1846	1825	1825
q8	3383	3597	3502	3502
q9	8869	8694	8825	8694
q10	3584	3540	3493	3493
q11	599	493	497	493
q12	812	639	666	639
q13	10860	3124	3187	3124
q14	294	265	267	265
q15	507	462	480	462
q16	678	630	646	630
q17	1846	1613	1591	1591
q18	8206	7716	7681	7681
q19	1672	1664	1478	1478
q20	2055	1824	1805	1805
q21	5538	5395	5345	5345
q22	1107	1093	1004	1004
Total cold run time: 70220 ms
Total hot run time: 59231 ms

doris-robot · 2025-04-21T07:00:10Z

TPC-DS: Total hot run time: 196553 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a96181b3988065942c0784cb9b01f2d18d7db3e3, data reload: false

query1	1301	910	873	873
query2	6250	2008	1987	1987
query3	10966	4484	4460	4460
query4	60816	29663	23424	23424
query5	5146	456	457	456
query6	404	187	177	177
query7	5412	335	303	303
query8	314	221	221	221
query9	8584	2602	2589	2589
query10	472	262	259	259
query11	17723	15163	15670	15163
query12	170	109	103	103
query13	1433	460	445	445
query14	10579	7006	7088	7006
query15	199	177	174	174
query16	7175	480	469	469
query17	1210	582	592	582
query18	1894	331	328	328
query19	224	160	157	157
query20	125	110	112	110
query21	208	103	105	103
query22	4620	4502	4746	4502
query23	34551	33757	34178	33757
query24	6247	2904	2910	2904
query25	532	407	409	407
query26	654	167	170	167
query27	2323	373	369	369
query28	4377	2442	2432	2432
query29	719	455	431	431
query30	235	165	162	162
query31	989	808	788	788
query32	64	56	55	55
query33	476	277	277	277
query34	908	501	511	501
query35	857	724	715	715
query36	1059	945	942	942
query37	121	67	65	65
query38	4010	3971	3974	3971
query39	1544	1471	1475	1471
query40	195	102	99	99
query41	48	48	48	48
query42	120	116	101	101
query43	541	486	492	486
query44	1155	831	797	797
query45	184	166	166	166
query46	1126	707	710	707
query47	2036	1943	1943	1943
query48	495	374	392	374
query49	761	381	401	381
query50	854	438	427	427
query51	7346	7309	7149	7149
query52	101	91	88	88
query53	260	181	183	181
query54	583	459	483	459
query55	78	77	79	77
query56	273	246	245	245
query57	1292	1176	1150	1150
query58	209	213	215	213
query59	3086	3085	2882	2882
query60	275	245	248	245
query61	137	108	112	108
query62	779	695	701	695
query63	229	192	199	192
query64	1427	692	665	665
query65	3262	3172	3172	3172
query66	746	297	297	297
query67	15895	15595	15473	15473
query68	4226	599	572	572
query69	423	260	261	260
query70	1128	1116	1088	1088
query71	333	252	256	252
query72	6378	3998	4022	3998
query73	750	345	343	343
query74	10526	9240	9019	9019
query75	3348	2613	2656	2613
query76	2019	1079	1054	1054
query77	459	292	271	271
query78	10585	9644	9587	9587
query79	1825	599	599	599
query80	1377	422	435	422
query81	544	238	235	235
query82	1245	86	87	86
query83	270	142	157	142
query84	280	76	76	76
query85	975	299	280	280
query86	397	285	288	285
query87	4415	4241	4185	4185
query88	3800	2410	2390	2390
query89	426	289	288	288
query90	1961	179	181	179
query91	179	148	150	148
query92	62	49	49	49
query93	2160	567	558	558
query94	780	307	300	300
query95	356	254	251	251
query96	613	283	279	279
query97	3307	3131	3151	3131
query98	219	206	203	203
query99	1668	1294	1285	1285
Total cold run time: 315418 ms
Total hot run time: 196553 ms

doris-robot · 2025-04-21T07:05:38Z

ClickBench: Total hot run time: 32.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a96181b3988065942c0784cb9b01f2d18d7db3e3, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.24	0.06	0.07
query4	1.62	0.11	0.11
query5	0.52	0.53	0.51
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.06	0.03	0.03
query9	0.58	0.50	0.49
query10	0.55	0.56	0.56
query11	0.15	0.11	0.12
query12	0.15	0.11	0.11
query13	0.63	0.60	0.61
query14	2.74	2.74	2.76
query15	0.90	0.82	0.83
query16	0.38	0.39	0.37
query17	1.01	1.08	1.04
query18	0.22	0.22	0.21
query19	1.85	1.81	2.01
query20	0.01	0.01	0.02
query21	15.38	0.59	0.60
query22	2.79	1.92	1.63
query23	16.94	0.94	0.79
query24	3.22	1.29	1.28
query25	0.28	0.12	0.04
query26	0.40	0.14	0.13
query27	0.04	0.04	0.04
query28	9.84	0.54	0.50
query29	12.60	3.25	3.25
query30	0.25	0.07	0.06
query31	2.88	0.40	0.40
query32	3.22	0.45	0.47
query33	3.02	3.02	2.97
query34	17.13	4.52	4.61
query35	4.63	4.48	4.54
query36	0.69	0.53	0.51
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.63 s
Total hot run time: 32.32 s

dataroaring

LGTM

github-actions · 2025-04-22T03:36:11Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-22T03:36:14Z

PR approved by anyone and no changes requested.

feiniaofeiafei · 2025-04-22T03:55:34Z

run buildall

doris-robot · 2025-04-22T05:36:01Z

TPC-H: Total hot run time: 40430 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a865e24367b33c31f931b347de5cca3a3fb96ab9, data reload: false

------ Round 1 ----------------------------------
q1	17585	6854	6676	6676
q2	2062	166	170	166
q3	10648	1085	1178	1085
q4	10538	783	700	700
q5	7727	2901	2858	2858
q6	224	136	133	133
q7	996	630	606	606
q8	9356	1968	2057	1968
q9	6707	6439	6407	6407
q10	6970	2276	2352	2276
q11	472	271	261	261
q12	414	220	219	219
q13	17794	2994	2989	2989
q14	238	215	208	208
q15	517	476	469	469
q16	684	598	587	587
q17	998	584	574	574
q18	7365	6603	6631	6603
q19	1400	1085	1086	1085
q20	463	211	203	203
q21	4192	3351	3415	3351
q22	1098	1006	1014	1006
Total cold run time: 108448 ms
Total hot run time: 40430 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6610	6580	6574	6574
q2	329	240	232	232
q3	2947	2768	2971	2768
q4	2062	1878	1876	1876
q5	5834	5783	5759	5759
q6	208	127	133	127
q7	2268	1832	1835	1832
q8	3400	3597	3532	3532
q9	8880	8905	8993	8905
q10	3582	3501	3491	3491
q11	599	497	517	497
q12	810	598	611	598
q13	9804	3211	3140	3140
q14	317	277	289	277
q15	521	457	465	457
q16	699	665	653	653
q17	1838	1650	1646	1646
q18	8369	7770	7766	7766
q19	1700	1486	1657	1486
q20	2054	1885	1881	1881
q21	5563	5431	5374	5374
q22	1157	1043	1022	1022
Total cold run time: 69551 ms
Total hot run time: 59893 ms

doris-robot · 2025-04-22T05:48:35Z

TPC-DS: Total hot run time: 196589 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a865e24367b33c31f931b347de5cca3a3fb96ab9, data reload: false

query1	1281	918	887	887
query2	6341	2051	1991	1991
query3	10816	4262	4297	4262
query4	61568	28679	23454	23454
query5	5202	470	442	442
query6	372	167	174	167
query7	5451	309	309	309
query8	299	236	229	229
query9	8468	2642	2594	2594
query10	465	265	258	258
query11	17929	15256	15769	15256
query12	167	105	104	104
query13	1436	448	451	448
query14	10832	6753	7030	6753
query15	203	174	187	174
query16	7156	438	451	438
query17	1245	578	577	577
query18	1931	324	325	324
query19	212	165	184	165
query20	121	111	109	109
query21	203	99	105	99
query22	4643	4502	4795	4502
query23	34683	33714	33898	33714
query24	6165	2951	2929	2929
query25	537	439	432	432
query26	662	167	169	167
query27	1939	350	348	348
query28	4060	2473	2480	2473
query29	713	493	436	436
query30	255	171	167	167
query31	999	811	842	811
query32	64	54	55	54
query33	395	294	287	287
query34	917	516	522	516
query35	879	721	735	721
query36	1068	926	985	926
query37	113	70	69	69
query38	4088	3970	3939	3939
query39	1510	1452	1453	1452
query40	203	98	98	98
query41	49	47	51	47
query42	123	100	99	99
query43	543	497	496	496
query44	1236	804	847	804
query45	181	168	167	167
query46	1146	719	732	719
query47	2016	1936	1895	1895
query48	480	387	387	387
query49	720	392	403	392
query50	863	436	435	435
query51	7438	7326	7286	7286
query52	105	88	94	88
query53	263	196	189	189
query54	559	459	470	459
query55	89	77	80	77
query56	256	249	249	249
query57	1290	1197	1165	1165
query58	224	213	216	213
query59	3253	3017	3059	3017
query60	280	261	250	250
query61	113	138	107	107
query62	758	675	677	675
query63	217	191	187	187
query64	1356	683	658	658
query65	3275	3177	3191	3177
query66	696	298	294	294
query67	15813	15577	15536	15536
query68	4317	567	580	567
query69	448	274	266	266
query70	1132	1097	1107	1097
query71	359	254	253	253
query72	6357	3993	4074	3993
query73	756	351	366	351
query74	10299	9043	8993	8993
query75	3381	2645	2605	2605
query76	1930	1034	1159	1034
query77	498	282	264	264
query78	10532	9632	9560	9560
query79	2204	604	598	598
query80	1336	431	424	424
query81	522	245	241	241
query82	1251	87	87	87
query83	155	144	143	143
query84	282	84	82	82
query85	972	294	280	280
query86	379	295	298	295
query87	4415	4272	4238	4238
query88	3944	2416	2364	2364
query89	408	288	293	288
query90	2008	185	189	185
query91	197	150	174	150
query92	64	51	50	50
query93	2942	560	559	559
query94	790	292	300	292
query95	364	263	263	263
query96	637	280	291	280
query97	3327	3144	3186	3144
query98	217	201	195	195
query99	1574	1299	1326	1299
Total cold run time: 316930 ms
Total hot run time: 196589 ms

doris-robot · 2025-04-22T05:54:03Z

ClickBench: Total hot run time: 32.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a865e24367b33c31f931b347de5cca3a3fb96ab9, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.24	0.07	0.06
query4	1.63	0.11	0.11
query5	0.52	0.52	0.51
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.57	0.50	0.50
query10	0.55	0.54	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.12
query13	0.60	0.60	0.58
query14	2.71	2.76	2.76
query15	0.89	0.83	0.83
query16	0.41	0.39	0.39
query17	0.96	1.04	1.06
query18	0.22	0.22	0.22
query19	1.88	1.84	1.96
query20	0.01	0.01	0.01
query21	15.35	0.59	0.59
query22	2.54	1.66	2.11
query23	16.97	1.02	0.83
query24	2.81	1.65	1.22
query25	0.25	0.18	0.11
query26	0.41	0.13	0.13
query27	0.05	0.06	0.04
query28	9.97	0.47	0.50
query29	12.61	3.24	3.23
query30	0.25	0.06	0.06
query31	2.88	0.38	0.37
query32	3.25	0.46	0.47
query33	2.98	3.04	3.00
query34	16.77	4.50	4.46
query35	4.53	4.54	4.49
query36	0.70	0.48	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.03	0.02
query40	0.15	0.13	0.13
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.54 s
Total hot run time: 32.32 s

dataroaring

LGTM

feiniaofeiafei added 2 commits April 15, 2025 03:30

fix regression

a96181b

feiniaofeiafei requested a review from dataroaring as a code owner April 21, 2025 04:22

dataroaring previously approved these changes Apr 22, 2025

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 22, 2025

924060929 previously approved these changes Apr 22, 2025

View reviewed changes

fix

a865e24

feiniaofeiafei dismissed stale reviews from 924060929 and dataroaring via a865e24 April 22, 2025 03:55

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 22, 2025

dataroaring approved these changes May 7, 2025

View reviewed changes

dataroaring merged commit 33cbf77 into apache:branch-3.0 May 7, 2025
23 checks passed

branch-3.0: [fix](nereids) do eliminate constant group by key in normalizeagg #49589 #50215

branch-3.0: [fix](nereids) do eliminate constant group by key in normalizeagg #49589 #50215

Uh oh!

Conversation

feiniaofeiafei commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hello-stephen commented Apr 21, 2025

Uh oh!

feiniaofeiafei commented Apr 21, 2025

Uh oh!

feiniaofeiafei commented Apr 21, 2025

Uh oh!

doris-robot commented Apr 21, 2025

Uh oh!

doris-robot commented Apr 21, 2025

Uh oh!

doris-robot commented Apr 21, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 22, 2025

Uh oh!

github-actions bot commented Apr 22, 2025

Uh oh!

feiniaofeiafei commented Apr 22, 2025

Uh oh!

doris-robot commented Apr 22, 2025

Uh oh!

doris-robot commented Apr 22, 2025

Uh oh!

doris-robot commented Apr 22, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

feiniaofeiafei commented Apr 21, 2025 •

edited

Loading