Skip to content

branch-3.1: [fix](nereids) Fix the expr id are same but different expr when agg table with random distribute #52993 #53621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Jul 21, 2025
Merged

branch-3.1: [fix](nereids) Fix the expr id are same but different expr when agg table with random distribute #52993 #53621

morrySnow merged 2 commits into from
Jul 21, 2025

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Jul 21, 2025 •
edited by morrySnow
Loading

cherry picked from #52993

@seawinde
[fix](nereids) Fix the expr id are same but different expr when agg t…
4a7060e 
…able with random distribute (apache#52993)

If agg table is random hash distribute, would add aggregate node on
scan.
The aggregate function alias expr id is same to the child expr id of
alias.

such as query sql is `select * from db1.tagg`
the query plan is as following, and the `sum(b#1) AS `b`#1`, alias expr
id is same to the child expr id of alias, the id is 1
this would cause hidden problems
```sql
LogicalResultSink[30] ( outputExprs=[a#0, b#1] )
+--LogicalAggregate[29] ( groupByExpr=[a#0], outputExpr=[a#0, sum(b#1) AS `b`#1], hasRepeat=false, stats=1 )
   +--LogicalOlapScan ( qualified=internal.db1.tagg, indexName=<index_not_selected>, selectedIndexId=1752042452160, preAgg=ON, operativeCol=[a#0, b#1], stats=1 )
```

the pr fix this, and the expression  change to `sum(b#1) AS `b`apache#4`  
```sql
LogicalResultSink[30] ( outputExprs=[a#0, b#4] )
+--LogicalAggregate[29] ( groupByExpr=[a#0], outputExpr=[a#0, sum(b#1) AS `b`apache#4], hasRepeat=false, stats=1 )
   +--LogicalOlapScan ( qualified=internal.db1.tagg, indexName=<index_not_selected>, selectedIndexId=1752042065062, preAgg=ON, operativeCol=[a#0, b#1], stats=1 )
```
@seawinde seawinde requested a review from morrySnow as a code owner July 21, 2025 03:59
@seawinde
Copy link
Contributor Author

seawinde commented Jul 21, 2025

run buildall

@Thearas
Copy link
Contributor

Thearas commented Jul 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

seawinde commented Jul 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 21, 2025

TPC-H: Total hot run time: 40133 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd54e2972f73e981785444cc185002bc712279cb, data reload: false

------ Round 1 ----------------------------------
q1	17648	6887	6664	6664
q2	2094	201	185	185
q3	10481	1148	1175	1148
q4	10232	790	747	747
q5	7768	2868	2811	2811
q6	226	154	139	139
q7	1023	631	633	631
q8	9602	2016	2064	2016
q9	8233	6424	6379	6379
q10	7015	2304	2305	2304
q11	481	287	283	283
q12	423	230	226	226
q13	17783	2997	3022	2997
q14	245	218	218	218
q15	550	507	532	507
q16	488	402	396	396
q17	989	604	536	536
q18	7288	6756	6635	6635
q19	1359	1134	964	964
q20	487	221	222	221
q21	3920	3145	3108	3108
q22	1161	1041	1018	1018
Total cold run time: 109496 ms
Total hot run time: 40133 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6678	6604	6605	6604
q2	346	260	250	250
q3	3061	2916	2930	2916
q4	2072	1830	1834	1830
q5	5714	5734	5745	5734
q6	214	138	140	138
q7	2260	1816	1803	1803
q8	3440	3546	3523	3523
q9	8871	8749	8904	8749
q10	3593	3572	3544	3544
q11	612	513	507	507
q12	864	587	632	587
q13	10841	3143	3230	3143
q14	320	275	290	275
q15	557	501	510	501
q16	496	446	444	444
q17	1858	1624	1628	1624
q18	8145	7740	7842	7740
q19	1753	1711	1459	1459
q20	2115	1913	1896	1896
q21	5268	5035	4894	4894
q22	1143	1093	1016	1016
Total cold run time: 70221 ms
Total hot run time: 59177 ms

@doris-robot
Copy link

doris-robot commented Jul 21, 2025

TPC-DS: Total hot run time: 199540 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd54e2972f73e981785444cc185002bc712279cb, data reload: false

query1	1296	925	914	914
query2	6469	1961	1969	1961
query3	11008	4345	4611	4345
query4	33448	23559	23466	23466
query5	3697	546	544	544
query6	277	195	190	190
query7	4009	336	338	336
query8	405	344	348	344
query9	9482	2597	2599	2597
query10	529	309	284	284
query11	17996	15338	15052	15052
query12	165	121	125	121
query13	1591	475	478	475
query14	9089	7525	6984	6984
query15	296	215	200	200
query16	8074	465	525	465
query17	1639	642	596	596
query18	2160	327	345	327
query19	258	179	176	176
query20	133	140	127	127
query21	221	126	127	126
query22	4950	4400	4323	4323
query23	35345	34183	34297	34183
query24	11743	2940	2959	2940
query25	720	443	421	421
query26	1875	184	187	184
query27	2776	361	366	361
query28	7513	2189	2168	2168
query29	1068	452	451	451
query30	284	180	184	180
query31	1088	873	847	847
query32	104	68	68	68
query33	809	350	353	350
query34	1088	524	547	524
query35	890	773	773	773
query36	1142	998	959	959
query37	293	87	82	82
query38	4148	4020	3978	3978
query39	1552	1526	1489	1489
query40	275	126	126	126
query41	104	97	97	97
query42	127	111	113	111
query43	540	501	482	482
query44	1331	832	816	816
query45	204	191	202	191
query46	1198	773	772	772
query47	2098	1940	1995	1940
query48	456	378	373	373
query49	1130	492	518	492
query50	865	469	458	458
query51	7501	7203	7370	7203
query52	113	106	106	106
query53	287	213	211	211
query54	1347	515	515	515
query55	92	85	84	84
query56	316	301	288	288
query57	1377	1235	1260	1235
query58	290	265	259	259
query59	3185	3125	3034	3034
query60	342	300	305	300
query61	143	141	141	141
query62	893	737	719	719
query63	246	219	209	209
query64	5035	767	751	751
query65	3346	3286	3353	3286
query66	1590	487	491	487
query67	16713	16260	15752	15752
query68	4983	601	599	599
query69	476	288	286	286
query70	1254	1104	1129	1104
query71	370	276	294	276
query72	6202	4165	4077	4077
query73	770	365	371	365
query74	10760	9287	8956	8956
query75	3435	2697	2691	2691
query76	2808	1087	1145	1087
query77	447	340	338	338
query78	10543	9626	9648	9626
query79	1579	627	604	604
query80	942	490	479	479
query81	554	243	234	234
query82	876	101	100	100
query83	283	183	173	173
query84	247	86	92	86
query85	1618	346	346	346
query86	490	328	316	316
query87	4400	4206	4267	4206
query88	4108	2458	2465	2458
query89	430	305	310	305
query90	2070	204	202	202
query91	181	120	126	120
query92	79	61	65	61
query93	2443	561	562	561
query94	884	290	321	290
query95	377	277	270	270
query96	620	291	289	289
query97	3311	3148	3180	3148
query98	234	220	211	211
query99	1527	1327	1354	1327
Total cold run time: 310075 ms
Total hot run time: 199540 ms

@doris-robot
Copy link

doris-robot commented Jul 21, 2025

ClickBench: Total hot run time: 30.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd54e2972f73e981785444cc185002bc712279cb, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.52	0.52	0.51
query6	1.14	0.73	0.72
query7	0.03	0.01	0.02
query8	0.04	0.04	0.03
query9	0.57	0.50	0.52
query10	0.55	0.55	0.60
query11	0.16	0.10	0.11
query12	0.13	0.11	0.12
query13	0.61	0.60	0.60
query14	0.79	0.78	0.81
query15	0.84	0.84	0.82
query16	0.40	0.35	0.39
query17	1.09	1.00	1.07
query18	0.24	0.22	0.22
query19	1.99	1.87	1.80
query20	0.02	0.02	0.01
query21	15.39	0.57	0.55
query22	2.44	1.68	2.02
query23	16.93	1.00	0.88
query24	3.07	1.05	0.90
query25	0.17	0.21	0.09
query26	0.33	0.14	0.14
query27	0.05	0.05	0.04
query28	10.62	0.48	0.54
query29	12.60	3.19	3.17
query30	0.34	0.16	0.15
query31	2.80	0.40	0.39
query32	3.23	0.47	0.46
query33	2.99	2.98	3.01
query34	17.15	4.47	4.53
query35	4.61	4.56	4.59
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.06	0.04	0.05
query39	0.04	0.02	0.03
query40	0.18	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 105.02 s
Total hot run time: 30.11 s

@morrySnow morrySnow changed the title [fix](nereids) Fix the expr id are same but different expr when agg table with random distribute (#52993) branch-3.1: [fix](nereids) Fix the expr id are same but different expr when agg table with random distribute #52993 Jul 21, 2025
@morrySnow morrySnow merged commit d781dd1 into apache:branch-3.1 Jul 21, 2025
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@seawinde @Thearas @doris-robot @morrySnow