Skip to content

[fix](virtual slot) Fix complex type and lambda function #55869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 5 commits into from
Sep 12, 2025
Merged

[fix](virtual slot) Fix complex type and lambda function #55869

morrySnow merged 5 commits into from
Sep 12, 2025

Conversation

@zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Sep 10, 2025 •
edited
Loading

What problem does this PR solve?

Related: #52701

  1. Functions that processing comprehensive type is too complicated, and may have many unexpected problem, eg. return array<null_type>, so do not process them by using virtual slot.
  2. lambda function can not be processed by virtual column.

So stop removing common sub expression if we meet above cases.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhiqiang-hhhh
Copy link
Contributor Author

zhiqiang-hhhh commented Sep 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 10, 2025

TPC-H: Total hot run time: 34612 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5c5d1585f7edc46006be8975b834edd4d53503ef, data reload: false

------ Round 1 ----------------------------------
q1	17599	5290	5083	5083
q2	1988	317	216	216
q3	10245	1275	709	709
q4	10238	1030	523	523
q5	7516	2422	2379	2379
q6	198	165	139	139
q7	944	792	651	651
q8	9348	1343	1099	1099
q9	6825	5104	5161	5104
q10	6967	2398	1976	1976
q11	479	293	279	279
q12	362	366	224	224
q13	17799	3706	3058	3058
q14	247	240	219	219
q15	561	499	483	483
q16	1028	1010	945	945
q17	627	873	354	354
q18	7348	7220	7001	7001
q19	1464	941	599	599
q20	358	337	228	228
q21	3834	3247	2354	2354
q22	1078	1017	989	989
Total cold run time: 107053 ms
Total hot run time: 34612 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5231	5136	5141	5136
q2	248	328	225	225
q3	2157	2725	2282	2282
q4	1388	1798	1339	1339
q5	4204	4541	4647	4541
q6	223	170	133	133
q7	2098	2059	1851	1851
q8	2694	2555	2495	2495
q9	7314	7420	7318	7318
q10	3158	3337	2882	2882
q11	581	524	521	521
q12	716	784	643	643
q13	3595	3970	3385	3385
q14	301	315	264	264
q15	542	480	483	480
q16	1111	1102	1073	1073
q17	1215	1578	1337	1337
q18	7952	7822	7393	7393
q19	827	810	1015	810
q20	1912	2016	1819	1819
q21	4829	4362	4300	4300
q22	1071	1032	1004	1004
Total cold run time: 53367 ms
Total hot run time: 51231 ms

@doris-robot
Copy link

doris-robot commented Sep 10, 2025

TPC-DS: Total hot run time: 189989 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5c5d1585f7edc46006be8975b834edd4d53503ef, data reload: false

query1	1063	441	410	410
query2	6606	1697	1712	1697
query3	6752	224	230	224
query4	26897	23268	22827	22827
query5	4393	670	507	507
query6	353	262	235	235
query7	4657	542	312	312
query8	338	266	258	258
query9	8654	3025	2872	2872
query10	513	359	309	309
query11	16046	15310	14948	14948
query12	183	128	121	121
query13	1691	581	445	445
query14	11755	9442	9279	9279
query15	218	213	176	176
query16	7699	659	497	497
query17	1680	814	653	653
query18	2077	444	352	352
query19	266	203	175	175
query20	140	126	121	121
query21	217	138	111	111
query22	4043	4206	4034	4034
query23	34022	32996	33020	32996
query24	8028	2426	2473	2426
query25	574	512	457	457
query26	1240	281	177	177
query27	2667	509	372	372
query28	4328	2244	2222	2222
query29	800	632	505	505
query30	294	221	204	204
query31	923	787	785	785
query32	97	85	87	85
query33	564	399	391	391
query34	806	858	549	549
query35	814	840	769	769
query36	954	1038	902	902
query37	150	118	91	91
query38	3605	3579	3571	3571
query39	1499	1431	1460	1431
query40	233	143	139	139
query41	68	72	61	61
query42	136	123	129	123
query43	520	488	499	488
query44	1407	905	895	895
query45	195	186	172	172
query46	923	1066	715	715
query47	1806	1865	1851	1851
query48	400	446	332	332
query49	753	523	429	429
query50	689	732	421	421
query51	4029	4085	4045	4045
query52	121	124	117	117
query53	262	298	210	210
query54	647	624	573	573
query55	97	95	96	95
query56	358	337	337	337
query57	1226	1207	1125	1125
query58	304	293	289	289
query59	2654	2680	2550	2550
query60	385	363	337	337
query61	164	161	191	161
query62	806	749	758	749
query63	246	204	200	200
query64	4448	1155	849	849
query65	4080	3946	3978	3946
query66	1076	462	353	353
query67	15319	15312	15079	15079
query68	9542	952	590	590
query69	503	330	303	303
query70	1332	1301	1290	1290
query71	585	366	327	327
query72	5530	4973	5001	4973
query73	745	640	364	364
query74	8975	9244	8982	8982
query75	4492	3232	2773	2773
query76	3751	1175	741	741
query77	887	416	337	337
query78	9615	9853	8853	8853
query79	1940	840	585	585
query80	683	597	531	531
query81	481	270	240	240
query82	428	168	146	146
query83	300	268	265	265
query84	300	110	98	98
query85	937	459	439	439
query86	346	319	312	312
query87	3739	3777	3664	3664
query88	2868	2233	2232	2232
query89	398	325	299	299
query90	1950	227	233	227
query91	166	164	134	134
query92	89	76	73	73
query93	1145	1005	651	651
query94	694	418	308	308
query95	421	343	331	331
query96	496	590	280	280
query97	2949	2985	2928	2928
query98	256	220	218	218
query99	1463	1420	1358	1358
Total cold run time: 278964 ms
Total hot run time: 189989 ms

@doris-robot
Copy link

doris-robot commented Sep 10, 2025

ClickBench: Total hot run time: 30.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5c5d1585f7edc46006be8975b834edd4d53503ef, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.06
query3	0.26	0.09	0.09
query4	1.61	0.12	0.12
query5	0.29	0.26	0.25
query6	1.16	0.64	0.63
query7	0.04	0.03	0.02
query8	0.06	0.05	0.05
query9	0.63	0.55	0.52
query10	0.58	0.57	0.56
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.64	0.62	0.61
query14	1.07	1.03	1.03
query15	0.86	0.87	0.86
query16	0.40	0.41	0.40
query17	1.07	1.06	1.10
query18	0.22	0.20	0.20
query19	1.97	1.80	1.78
query20	0.02	0.01	0.01
query21	15.40	0.96	0.60
query22	0.80	1.17	0.74
query23	14.82	1.41	0.64
query24	6.57	1.71	1.09
query25	0.55	0.14	0.15
query26	0.69	0.17	0.15
query27	0.08	0.05	0.06
query28	10.46	0.92	0.42
query29	12.56	3.96	3.28
query30	0.30	0.14	0.11
query31	2.82	0.59	0.38
query32	3.23	0.56	0.47
query33	3.26	3.11	3.13
query34	16.10	5.55	4.85
query35	4.94	4.90	4.88
query36	0.71	0.51	0.52
query37	0.10	0.07	0.07
query38	0.06	0.05	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.04	0.03
query43	0.05	0.04	0.03
Total cold run time: 105.19 s
Total hot run time: 30.32 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 10, 2025

FE UT Coverage Report

Increment line coverage 60.00% (3/5) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 10, 2025

FE Regression Coverage Report

Increment line coverage 60.00% (3/5) 🎉
Increment coverage report
Complete coverage report

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as ready for review September 11, 2025 02:19
@zhiqiang-hhhh
Copy link
Contributor Author

zhiqiang-hhhh commented Sep 11, 2025

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

zhiqiang-hhhh commented Sep 11, 2025

run buildall

@zhiqiang-hhhh zhiqiang-hhhh changed the title [fix](virtual slot) Fix array type and lambda function [fix](virtual slot) Fix complex type and lambda function Sep 11, 2025
@doris-robot
Copy link

doris-robot commented Sep 11, 2025

TPC-H: Total hot run time: 34590 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e8dc35d28a6ce5a986d2b2eec3d2a168f3e573c7, data reload: false

------ Round 1 ----------------------------------
q1	17609	5200	5039	5039
q2	2007	333	209	209
q3	10221	1261	692	692
q4	10247	1007	519	519
q5	7528	2411	2297	2297
q6	185	164	139	139
q7	914	757	624	624
q8	9335	1312	1106	1106
q9	6957	5208	5138	5138
q10	6927	2388	1981	1981
q11	490	324	268	268
q12	359	365	224	224
q13	17769	3692	3003	3003
q14	239	234	218	218
q15	586	477	500	477
q16	1011	996	949	949
q17	588	845	384	384
q18	7367	7251	7218	7218
q19	1292	945	539	539
q20	335	332	225	225
q21	3803	2581	2323	2323
q22	1044	1029	1018	1018
Total cold run time: 106813 ms
Total hot run time: 34590 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5147	5063	5090	5063
q2	244	326	228	228
q3	2146	2669	2306	2306
q4	1294	1798	1336	1336
q5	4205	4429	4521	4429
q6	216	179	130	130
q7	2037	1994	1914	1914
q8	2700	2681	2591	2591
q9	7242	7475	7345	7345
q10	3100	3404	2869	2869
q11	566	527	509	509
q12	693	753	606	606
q13	3505	3913	3396	3396
q14	356	311	281	281
q15	532	484	476	476
q16	1086	1160	1061	1061
q17	1334	1504	1402	1402
q18	7919	7652	7805	7652
q19	795	785	827	785
q20	2088	2076	1841	1841
q21	4789	4342	4318	4318
q22	1074	1075	1011	1011
Total cold run time: 53068 ms
Total hot run time: 51549 ms

@doris-robot
Copy link

doris-robot commented Sep 11, 2025

TPC-DS: Total hot run time: 187950 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e8dc35d28a6ce5a986d2b2eec3d2a168f3e573c7, data reload: false

query1	1073	446	399	399
query2	6568	1711	1645	1645
query3	6748	231	233	231
query4	26088	23593	22995	22995
query5	4412	660	514	514
query6	352	239	231	231
query7	4642	554	308	308
query8	293	251	248	248
query9	8631	2984	2999	2984
query10	501	365	301	301
query11	15990	15409	14902	14902
query12	184	123	123	123
query13	1683	576	437	437
query14	11382	9585	9515	9515
query15	228	201	177	177
query16	7715	654	518	518
query17	1251	800	678	678
query18	2072	445	357	357
query19	218	209	187	187
query20	142	129	126	126
query21	217	139	120	120
query22	4154	4321	4078	4078
query23	33963	32954	33133	32954
query24	8164	2430	2532	2430
query25	576	522	441	441
query26	1237	273	167	167
query27	2731	519	356	356
query28	4417	2297	2276	2276
query29	776	642	493	493
query30	291	226	199	199
query31	914	808	729	729
query32	92	76	82	76
query33	666	398	380	380
query34	810	859	522	522
query35	846	838	765	765
query36	977	1014	927	927
query37	130	116	93	93
query38	3544	3504	3498	3498
query39	1460	1416	1450	1416
query40	233	138	127	127
query41	64	66	59	59
query42	130	120	125	120
query43	524	491	462	462
query44	1381	877	868	868
query45	181	177	175	175
query46	863	1026	676	676
query47	1799	1791	1734	1734
query48	384	420	318	318
query49	748	504	439	439
query50	671	706	411	411
query51	4004	3946	3811	3811
query52	113	119	116	116
query53	249	275	205	205
query54	619	618	559	559
query55	98	93	94	93
query56	347	344	320	320
query57	1197	1211	1131	1131
query58	292	287	285	285
query59	2582	2626	2541	2541
query60	356	363	367	363
query61	188	177	166	166
query62	833	729	673	673
query63	232	202	212	202
query64	4435	1136	838	838
query65	4033	3973	4001	3973
query66	1113	463	417	417
query67	15617	15462	15044	15044
query68	9039	935	585	585
query69	499	355	317	317
query70	1359	1228	1263	1228
query71	590	354	329	329
query72	5907	5173	3466	3466
query73	802	706	368	368
query74	8939	9105	8745	8745
query75	4261	3229	2849	2849
query76	3760	1131	743	743
query77	850	410	357	357
query78	9590	9807	8862	8862
query79	2569	845	605	605
query80	702	591	524	524
query81	480	271	233	233
query82	467	178	137	137
query83	300	260	258	258
query84	376	115	92	92
query85	874	458	431	431
query86	345	318	294	294
query87	3834	3757	3610	3610
query88	3105	2234	2267	2234
query89	408	333	320	320
query90	1956	234	236	234
query91	165	165	137	137
query92	92	89	72	72
query93	1157	995	659	659
query94	700	427	337	337
query95	420	347	337	337
query96	487	590	286	286
query97	2948	3002	2880	2880
query98	247	223	223	223
query99	1448	1402	1276	1276
Total cold run time: 278100 ms
Total hot run time: 187950 ms

@doris-robot
Copy link

doris-robot commented Sep 11, 2025

ClickBench: Total hot run time: 30.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e8dc35d28a6ce5a986d2b2eec3d2a168f3e573c7, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.06
query3	0.26	0.08	0.08
query4	1.61	0.11	0.11
query5	0.28	0.26	0.25
query6	1.16	0.67	0.64
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.62	0.52	0.53
query10	0.58	0.58	0.58
query11	0.15	0.12	0.11
query12	0.16	0.12	0.12
query13	0.65	0.64	0.61
query14	1.03	1.05	1.04
query15	0.86	0.85	0.85
query16	0.40	0.40	0.40
query17	1.07	1.08	1.07
query18	0.22	0.20	0.20
query19	1.99	1.80	1.87
query20	0.01	0.01	0.01
query21	15.40	0.96	0.57
query22	0.80	1.19	0.78
query23	14.79	1.38	0.63
query24	7.04	0.99	0.82
query25	0.49	0.25	0.10
query26	0.62	0.15	0.14
query27	0.06	0.05	0.06
query28	9.46	0.93	0.43
query29	12.56	3.96	3.30
query30	0.30	0.15	0.11
query31	2.84	0.58	0.39
query32	3.26	0.55	0.48
query33	3.06	3.18	3.09
query34	16.17	5.52	4.87
query35	4.92	4.93	4.87
query36	0.71	0.53	0.50
query37	0.11	0.08	0.07
query38	0.07	0.05	0.05
query39	0.04	0.03	0.02
query40	0.20	0.15	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 104.33 s
Total hot run time: 30.08 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 11, 2025

FE UT Coverage Report

Increment line coverage 60.00% (3/5) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 11, 2025

FE Regression Coverage Report

Increment line coverage 60.00% (3/5) 🎉
Increment coverage report
Complete coverage report

2 similar comments
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 11, 2025

FE Regression Coverage Report

Increment line coverage 60.00% (3/5) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 11, 2025

FE Regression Coverage Report

Increment line coverage 60.00% (3/5) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 11, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 11, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 11, 2025

PR approved by anyone and no changes requested.

airborne12
airborne12 approved these changes Sep 12, 2025
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morrySnow morrySnow merged commit d090ba2 into apache:master Sep 12, 2025
32 of 33 checks passed
@zhiqiang-hhhh zhiqiang-hhhh deleted the fix-array-vir branch September 12, 2025 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@englefly englefly englefly approved these changes

@airborne12 airborne12 airborne12 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zhiqiang-hhhh @hello-stephen @doris-robot @englefly @airborne12 @morrySnow