Skip to content

[feature](nereids) support common sub expression by multi-layer projections (fe part) #33087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
englefly merged 2 commits into from
Apr 2, 2024
Merged

[feature](nereids) support common sub expression by multi-layer projections (fe part) #33087

englefly merged 2 commits into from
Apr 2, 2024

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Apr 1, 2024

Proposed changes

BE part: #32673
Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Apr 1, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

englefly commented Apr 1, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Apr 1, 2024

TPC-H: Total hot run time: 38425 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e76948cbbccb1114240bfa1e5649577539baff07, data reload: false

------ Round 1 ----------------------------------
q1	17620	4053	4022	4022
q2	2023	191	188	188
q3	10476	1240	1361	1240
q4	10207	835	1021	835
q5	7479	2944	2911	2911
q6	216	135	133	133
q7	1074	623	605	605
q8	9397	2029	1997	1997
q9	6643	6155	6140	6140
q10	8452	3509	3486	3486
q11	425	238	246	238
q12	381	221	218	218
q13	17777	2886	2903	2886
q14	278	243	241	241
q15	530	490	478	478
q16	528	391	380	380
q17	956	912	890	890
q18	7243	6490	6368	6368
q19	1593	1532	1519	1519
q20	601	328	304	304
q21	3532	3050	3070	3050
q22	353	297	296	296
Total cold run time: 107784 ms
Total hot run time: 38425 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4055	4008	4026	4008
q2	327	224	224	224
q3	2940	2940	2942	2940
q4	1858	1800	1796	1796
q5	5215	5208	5213	5208
q6	207	123	123	123
q7	2243	1771	1810	1771
q8	3178	3257	3237	3237
q9	8473	8465	8471	8465
q10	3737	3860	3996	3860
q11	552	455	455	455
q12	745	596	573	573
q13	16220	3112	3151	3112
q14	313	275	271	271
q15	552	477	490	477
q16	518	440	462	440
q17	1770	1742	1699	1699
q18	8127	7597	7700	7597
q19	2246	1677	1683	1677
q20	2005	1812	1860	1812
q21	5069	4939	4938	4938
q22	492	430	450	430
Total cold run time: 70842 ms
Total hot run time: 55113 ms

@doris-robot
Copy link

doris-robot commented Apr 1, 2024

TPC-DS: Total hot run time: 182172 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e76948cbbccb1114240bfa1e5649577539baff07, data reload: false

query1	1219	1120	1114	1114
query2	6324	1976	1877	1877
query3	6667	215	203	203
query4	24190	21414	21458	21414
query5	4189	402	403	402
query6	282	214	189	189
query7	4608	307	299	299
query8	230	177	181	177
query9	8474	2224	2241	2224
query10	471	260	257	257
query11	14968	14630	14510	14510
query12	150	93	96	93
query13	1670	394	396	394
query14	8463	7033	6986	6986
query15	204	176	176	176
query16	6772	276	279	276
query17	968	613	582	582
query18	1842	283	281	281
query19	208	171	168	168
query20	99	98	96	96
query21	208	133	137	133
query22	4916	4927	4824	4824
query23	33641	32625	32422	32422
query24	11898	3250	3260	3250
query25	668	409	446	409
query26	1555	171	173	171
query27	3125	380	395	380
query28	7084	1906	1870	1870
query29	1057	605	585	585
query30	315	156	154	154
query31	1053	758	752	752
query32	99	69	69	69
query33	695	254	246	246
query34	1098	507	513	507
query35	865	757	730	730
query36	997	858	876	858
query37	146	84	85	84
query38	3745	3542	3592	3542
query39	1666	1581	1605	1581
query40	252	151	147	147
query41	50	46	52	46
query42	116	112	107	107
query43	458	405	423	405
query44	1161	745	737	737
query45	269	275	266	266
query46	1091	831	818	818
query47	2033	1893	1910	1893
query48	401	326	317	317
query49	907	381	376	376
query50	842	423	424	423
query51	6981	6808	6848	6808
query52	123	107	109	107
query53	370	308	299	299
query54	302	253	260	253
query55	97	81	83	81
query56	264	251	243	243
query57	1266	1182	1178	1178
query58	268	246	241	241
query59	2813	2647	2428	2428
query60	269	252	242	242
query61	109	108	106	106
query62	677	453	442	442
query63	312	291	289	289
query64	5185	3308	3232	3232
query65	3025	3017	2984	2984
query66	954	318	313	313
query67	15457	15179	14721	14721
query68	8469	578	576	576
query69	558	326	327	326
query70	1263	1068	1069	1068
query71	525	273	267	267
query72	6379	2584	2418	2418
query73	811	333	323	323
query74	6765	6379	6346	6346
query75	3492	2297	2291	2291
query76	5263	1157	1201	1157
query77	582	248	252	248
query78	10967	10127	10059	10059
query79	9741	537	535	535
query80	1565	439	421	421
query81	492	222	221	221
query82	680	104	102	102
query83	215	165	168	165
query84	262	91	85	85
query85	1332	294	279	279
query86	457	258	279	258
query87	3682	3480	3432	3432
query88	4137	2272	2287	2272
query89	568	373	367	367
query90	1912	179	181	179
query91	131	106	103	103
query92	59	52	54	52
query93	6859	530	536	530
query94	1082	196	204	196
query95	444	330	328	328
query96	598	275	270	270
query97	2645	2490	2458	2458
query98	223	223	214	214
query99	1288	839	840	839
Total cold run time: 296327 ms
Total hot run time: 182172 ms

@doris-robot
Copy link

doris-robot commented Apr 1, 2024

ClickBench: Total hot run time: 30.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e76948cbbccb1114240bfa1e5649577539baff07, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.07
query5	0.48	0.48	0.49
query6	1.13	0.66	0.66
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.59	0.52	0.51
query10	0.58	0.58	0.56
query11	0.15	0.12	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	0.76	0.79	0.80
query15	0.88	0.83	0.84
query16	0.35	0.35	0.35
query17	1.02	1.02	0.98
query18	0.25	0.24	0.25
query19	1.81	1.80	1.73
query20	0.01	0.01	0.02
query21	15.41	0.73	0.74
query22	2.98	5.69	2.31
query23	17.54	1.24	1.15
query24	1.50	0.22	0.22
query25	0.13	0.09	0.09
query26	0.28	0.17	0.19
query27	0.08	0.09	0.09
query28	13.83	0.97	0.92
query29	12.76	3.53	3.43
query30	0.26	0.06	0.05
query31	2.86	0.39	0.39
query32	3.26	0.46	0.47
query33	2.80	2.89	2.85
query34	15.54	4.34	4.32
query35	4.36	4.36	4.36
query36	0.68	0.47	0.47
query37	0.18	0.18	0.16
query38	0.18	0.16	0.17
query39	0.04	0.04	0.04
query40	0.18	0.14	0.15
query41	0.09	0.06	0.06
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 105.88 s
Total hot run time: 30.85 s

@doris-robot
Copy link

doris-robot commented Apr 1, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e76948cbbccb1114240bfa1e5649577539baff07 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       15.3 seconds inserted 10000000 Rows, about 653K ops/s

@github-actions
Copy link
Contributor

github-actions bot commented Apr 1, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 1, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Apr 1, 2024

PR approved by anyone and no changes requested.

@englefly englefly merged commit ac5c9c6 into apache:master Apr 2, 2024
@englefly englefly deleted the cse-fe branch April 2, 2024 00:52
yiguolei pushed a commit that referenced this pull request Apr 10, 2024
@englefly @Doris-Extras
[feature](nereids) support common sub expression by multi-layer proje…
c3ffbdb 
…ctions (fe part) (#33087)

* cse fe part
yiguolei pushed a commit that referenced this pull request Apr 10, 2024
@englefly @Doris-Extras
[feature](nereids) support common sub expression by multi-layer proje…
a7c8abe 
…ctions (fe part) (#33087)

* cse fe part
@morrySnow morrySnow mentioned this pull request Mar 17, 2025
Merged
16 tasks
morrySnow added a commit that referenced this pull request Mar 18, 2025
@morrySnow
[fix](Nereids) cse extract wrong expression from lambda expressions (#…
9d4faa5 
…49166)

### What problem does this PR solve?

Related PR: #33087

Problem Summary:

CSE should not extract ArrayItemSlot and ArrayItemReference. Because
they could not be computed out of Lambda expression.
NOTICE: currently, we could not extract common expression from Lambda,
because ArrayItemSlot in Lambda are not same.
924060929 pushed a commit to 924060929/incubator-doris that referenced this pull request Apr 10, 2025
@morrySnow @924060929
[fix](Nereids) cse extract wrong expression from lambda expressions (a…
883a50d 
…pache#49166)

### What problem does this PR solve?

Related PR: apache#33087

Problem Summary:

CSE should not extract ArrayItemSlot and ArrayItemReference. Because
they could not be computed out of Lambda expression.
NOTICE: currently, we could not extract common expression from Lambda,
because ArrayItemSlot in Lambda are not same.

(cherry picked from commit 9d4faa5)
924060929 pushed a commit to 924060929/incubator-doris that referenced this pull request Apr 10, 2025
@morrySnow @924060929
[fix](Nereids) cse extract wrong expression from lambda expressions (a…
608660b 
…pache#49166)

### What problem does this PR solve?

Related PR: apache#33087

Problem Summary:

CSE should not extract ArrayItemSlot and ArrayItemReference. Because
they could not be computed out of Lambda expression.
NOTICE: currently, we could not extract common expression from Lambda,
because ArrayItemSlot in Lambda are not same.

(cherry picked from commit 9d4faa5)
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@morrySnow @koarz
[fix](Nereids) cse extract wrong expression from lambda expressions (a…
30720d6 
…pache#49166)

### What problem does this PR solve?

Related PR: apache#33087

Problem Summary:

CSE should not extract ArrayItemSlot and ArrayItemReference. Because
they could not be computed out of Lambda expression.
NOTICE: currently, we could not extract common expression from Lambda,
because ArrayItemSlot in Lambda are not same.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

+1 more reviewer

@Mryange Mryange Mryange approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@englefly @doris-robot @starocean999 @Mryange