Skip to content

branch-3.0: [fix](nereids) Fix the expr id are same but different expr when agg table with random distribute #52993 #53619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
Aug 12, 2025
Merged

branch-3.0: [fix](nereids) Fix the expr id are same but different expr when agg table with random distribute #52993 #53619

dataroaring merged 2 commits into from
Aug 12, 2025

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Jul 21, 2025 •
edited by morrySnow
Loading

picked from #52993

@seawinde
[fix](nereids) Fix the expr id are same but different expr when agg t…
19d5237 
…able with random distribute (apache#52993)

If agg table is random hash distribute, would add aggregate node on
scan.
The aggregate function alias expr id is same to the child expr id of
alias.

such as query sql is `select * from db1.tagg`
the query plan is as following, and the `sum(b#1) AS `b`#1`, alias expr
id is same to the child expr id of alias, the id is 1
this would cause hidden problems
```sql
LogicalResultSink[30] ( outputExprs=[a#0, b#1] )
+--LogicalAggregate[29] ( groupByExpr=[a#0], outputExpr=[a#0, sum(b#1) AS `b`#1], hasRepeat=false, stats=1 )
   +--LogicalOlapScan ( qualified=internal.db1.tagg, indexName=<index_not_selected>, selectedIndexId=1752042452160, preAgg=ON, operativeCol=[a#0, b#1], stats=1 )
```

the pr fix this, and the expression  change to `sum(b#1) AS `b`apache#4`  
```sql
LogicalResultSink[30] ( outputExprs=[a#0, b#4] )
+--LogicalAggregate[29] ( groupByExpr=[a#0], outputExpr=[a#0, sum(b#1) AS `b`apache#4], hasRepeat=false, stats=1 )
   +--LogicalOlapScan ( qualified=internal.db1.tagg, indexName=<index_not_selected>, selectedIndexId=1752042065062, preAgg=ON, operativeCol=[a#0, b#1], stats=1 )
```
@seawinde seawinde requested a review from dataroaring as a code owner July 21, 2025 03:57
@Thearas
Copy link
Contributor

Thearas commented Jul 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

seawinde commented Jul 21, 2025

run buildall

@seawinde
Copy link
Contributor Author

seawinde commented Jul 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 21, 2025

TPC-H: Total hot run time: 39569 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 347be419bd62ad90b83c04667fb760d226cc1121, data reload: false

------ Round 1 ----------------------------------
q1	17616	7199	6657	6657
q2	2061	164	153	153
q3	10552	1131	1126	1126
q4	10204	785	683	683
q5	7714	2896	2802	2802
q6	220	130	129	129
q7	956	622	596	596
q8	9349	1946	1999	1946
q9	6666	6351	6423	6351
q10	6977	2245	2246	2245
q11	460	261	260	260
q12	397	209	213	209
q13	17787	2944	2986	2944
q14	223	204	205	204
q15	501	463	468	463
q16	478	390	370	370
q17	968	577	580	577
q18	7333	6540	6621	6540
q19	1391	1091	1011	1011
q20	502	197	200	197
q21	3926	3111	3121	3111
q22	1133	995	1017	995
Total cold run time: 107414 ms
Total hot run time: 39569 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6583	6505	6559	6505
q2	331	236	225	225
q3	2940	2900	2926	2900
q4	2052	1735	1838	1735
q5	5674	5679	5692	5679
q6	204	129	125	125
q7	2221	1803	1836	1803
q8	3329	3485	3530	3485
q9	8673	8884	8784	8784
q10	3538	3527	3533	3527
q11	585	499	476	476
q12	828	597	609	597
q13	10167	3218	3075	3075
q14	294	285	271	271
q15	500	471	463	463
q16	501	447	437	437
q17	1859	1642	1629	1629
q18	8261	7871	7775	7775
q19	1733	1555	1485	1485
q20	2038	1828	1838	1828
q21	5169	5102	5037	5037
q22	1177	1096	1075	1075
Total cold run time: 68657 ms
Total hot run time: 58916 ms

@doris-robot
Copy link

doris-robot commented Jul 21, 2025

TPC-DS: Total hot run time: 194909 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 347be419bd62ad90b83c04667fb760d226cc1121, data reload: false

query1	1304	887	904	887
query2	6282	2004	1943	1943
query3	10998	4380	4707	4380
query4	33135	23879	23199	23199
query5	4310	452	449	449
query6	269	171	178	171
query7	3987	303	309	303
query8	274	219	215	215
query9	9360	2576	2582	2576
query10	487	261	256	256
query11	17961	15114	15717	15114
query12	159	101	98	98
query13	1542	425	430	425
query14	8662	7600	7168	7168
query15	262	184	184	184
query16	8061	512	511	511
query17	1643	579	596	579
query18	2183	326	325	325
query19	321	161	166	161
query20	128	117	125	117
query21	214	107	105	105
query22	5023	4592	4604	4592
query23	35131	34131	33648	33648
query24	11690	2997	2940	2940
query25	696	430	415	415
query26	1887	168	172	168
query27	2854	365	349	349
query28	7641	2212	2173	2173
query29	1072	461	447	447
query30	256	182	157	157
query31	1017	810	799	799
query32	88	51	52	51
query33	761	298	287	287
query34	1152	501	515	501
query35	873	716	710	710
query36	1105	932	958	932
query37	189	68	69	68
query38	4079	3955	3896	3896
query39	1533	1478	1456	1456
query40	268	100	97	97
query41	53	48	51	48
query42	107	104	103	103
query43	535	490	479	479
query44	1328	837	814	814
query45	177	169	169	169
query46	1170	746	730	730
query47	2059	1960	1930	1930
query48	476	386	380	380
query49	1085	384	425	384
query50	851	434	432	432
query51	7443	7249	7300	7249
query52	100	90	91	90
query53	254	183	180	180
query54	1295	486	461	461
query55	81	82	79	79
query56	275	261	266	261
query57	1342	1265	1198	1198
query58	229	206	222	206
query59	3247	3159	3135	3135
query60	302	255	279	255
query61	146	125	157	125
query62	860	680	702	680
query63	226	192	185	185
query64	4980	665	671	665
query65	3385	3266	3220	3220
query66	1284	285	281	281
query67	15995	15450	15520	15450
query68	4793	583	563	563
query69	432	296	257	257
query70	1190	1090	1114	1090
query71	323	258	253	253
query72	6184	2510	3865	2510
query73	739	344	349	344
query74	10705	8968	9311	8968
query75	3362	2620	2653	2620
query76	2965	1001	1060	1001
query77	366	274	258	258
query78	10539	9682	9661	9661
query79	2308	607	606	606
query80	1133	420	418	418
query81	556	221	221	221
query82	891	90	88	88
query83	235	144	144	144
query84	227	75	76	75
query85	1371	301	294	294
query86	431	292	290	290
query87	4452	4201	4307	4201
query88	4122	2427	2405	2405
query89	413	290	285	285
query90	2093	183	178	178
query91	183	147	149	147
query92	69	51	51	51
query93	2371	564	555	555
query94	1003	267	305	267
query95	354	255	253	253
query96	609	276	284	276
query97	3300	3156	3123	3123
query98	215	210	187	187
query99	1530	1279	1310	1279
Total cold run time: 307606 ms
Total hot run time: 194909 ms

@doris-robot
Copy link

doris-robot commented Jul 21, 2025

ClickBench: Total hot run time: 29.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 347be419bd62ad90b83c04667fb760d226cc1121, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.11
query5	0.52	0.53	0.53
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.51
query10	0.54	0.57	0.60
query11	0.14	0.11	0.10
query12	0.14	0.11	0.10
query13	0.60	0.60	0.58
query14	0.78	0.80	0.82
query15	0.82	0.83	0.82
query16	0.39	0.37	0.39
query17	1.08	1.06	1.05
query18	0.24	0.21	0.22
query19	1.99	1.80	1.86
query20	0.01	0.01	0.01
query21	15.37	0.58	0.56
query22	2.28	1.84	1.50
query23	17.05	1.08	0.79
query24	2.92	2.00	1.00
query25	0.19	0.08	0.07
query26	0.61	0.13	0.14
query27	0.03	0.05	0.04
query28	9.61	0.52	0.52
query29	12.59	3.20	3.17
query30	0.25	0.06	0.07
query31	2.85	0.38	0.39
query32	3.24	0.45	0.45
query33	2.97	2.99	2.97
query34	17.12	4.50	4.52
query35	4.63	4.59	4.57
query36	0.66	0.48	0.46
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.16	0.12	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.81 s
Total hot run time: 29.9 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 21, 2025

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jul 23, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 23, 2025

PR approved by anyone and no changes requested.

@morrySnow morrySnow changed the title [fix](nereids) Fix the expr id are same but different expr when agg table with random distribute (#52993) branch-3.0: [fix](nereids) Fix the expr id are same but different expr when agg table with random distribute #52993 Jul 25, 2025
dataroaring
dataroaring approved these changes Aug 12, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 5c6e2a6 into apache:branch-3.0 Aug 12, 2025
24 of 25 checks passed
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@seawinde @Thearas @doris-robot @hello-stephen @dataroaring @morrySnow