Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](funcs) map struct construct funcs #39699

Merged
merged 4 commits into from
Aug 27, 2024
Merged

[fix](funcs) map struct construct funcs #39699

merged 4 commits into from
Aug 27, 2024

Conversation

amorynan
Copy link
Contributor

Proposed changes

we should not change the column which is passed from a block when we execute a function , because this block may be used in other expression or other exec node
Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@amorynan
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Aug 21, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/functions/function.h Outdated Show resolved Hide resolved
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@amorynan
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38465 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 04df395bd13baf07e2c7c555876aa85118b5c7da, data reload: false

------ Round 1 ----------------------------------
q1	18235	4559	4375	4375
q2	2060	209	216	209
q3	11483	995	1204	995
q4	10539	811	719	719
q5	7781	2846	2847	2846
q6	269	161	161	161
q7	1038	663	662	662
q8	9600	2146	2186	2146
q9	7862	6615	6579	6579
q10	7056	2233	2221	2221
q11	506	282	291	282
q12	422	248	255	248
q13	17785	3008	3041	3008
q14	307	243	251	243
q15	561	517	522	517
q16	531	419	404	404
q17	1022	702	660	660
q18	7457	6913	6727	6727
q19	6827	1185	1140	1140
q20	717	352	351	351
q21	3916	2957	3015	2957
q22	1118	1026	1015	1015
Total cold run time: 117092 ms
Total hot run time: 38465 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4544	4314	4353	4314
q2	407	292	294	292
q3	2855	2673	2731	2673
q4	1996	1789	1700	1700
q5	5702	5698	5664	5664
q6	253	163	163	163
q7	2273	1805	1803	1803
q8	3334	3538	3502	3502
q9	8855	8877	8688	8688
q10	3629	3383	3308	3308
q11	598	536	527	527
q12	862	671	675	671
q13	15864	3254	3186	3186
q14	322	308	291	291
q15	560	522	509	509
q16	510	455	461	455
q17	1873	1578	1521	1521
q18	8280	7926	7789	7789
q19	3559	1602	1668	1602
q20	2193	1942	1877	1877
q21	13913	5409	5494	5409
q22	1199	1060	1068	1060
Total cold run time: 83581 ms
Total hot run time: 57004 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196642 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 04df395bd13baf07e2c7c555876aa85118b5c7da, data reload: false

query1	1304	934	909	909
query2	6579	1954	1904	1904
query3	10644	3904	3814	3814
query4	57874	28014	23394	23394
query5	6134	754	726	726
query6	554	216	215	215
query7	6441	333	333	333
query8	558	442	436	436
query9	9438	2578	2531	2531
query10	591	364	344	344
query11	18833	15131	15820	15131
query12	193	133	136	133
query13	1700	434	441	434
query14	12103	7292	7213	7213
query15	265	198	198	198
query16	7583	532	531	531
query17	1190	641	648	641
query18	2125	354	345	345
query19	314	173	170	170
query20	157	141	137	137
query21	247	149	146	146
query22	4448	4247	4211	4211
query23	34613	33749	33924	33749
query24	5705	3021	3026	3021
query25	592	436	433	433
query26	758	187	193	187
query27	1910	297	304	297
query28	4011	2207	2174	2174
query29	742	455	447	447
query30	220	185	184	184
query31	1008	840	823	823
query32	103	77	78	77
query33	553	350	333	333
query34	917	508	506	506
query35	876	783	761	761
query36	1125	981	975	975
query37	162	101	101	101
query38	3953	3875	3987	3875
query39	1535	1511	1481	1481
query40	241	159	154	154
query41	137	160	136	136
query42	144	117	118	117
query43	540	494	496	494
query44	1129	788	800	788
query45	224	200	198	198
query46	1139	783	775	775
query47	1896	1835	1850	1835
query48	409	340	337	337
query49	918	581	583	581
query50	869	472	482	472
query51	7188	7123	7117	7117
query52	119	106	113	106
query53	303	226	233	226
query54	620	508	496	496
query55	89	90	90	90
query56	341	309	312	309
query57	1191	1140	1097	1097
query58	306	314	298	298
query59	3009	2690	2825	2690
query60	346	322	322	322
query61	152	148	149	148
query62	791	702	719	702
query63	262	230	224	224
query64	4271	2398	1876	1876
query65	3296	3172	3186	3172
query66	1051	679	669	669
query67	15399	15067	15006	15006
query68	5000	632	583	583
query69	544	320	313	313
query70	1227	1187	1084	1084
query71	493	316	318	316
query72	7418	2292	2095	2095
query73	827	357	358	357
query74	9291	8767	8884	8767
query75	3514	2723	2743	2723
query76	2534	1106	1005	1005
query77	852	467	451	451
query78	10678	9209	9080	9080
query79	12823	561	553	553
query80	1616	611	609	609
query81	621	262	263	262
query82	728	161	154	154
query83	416	222	210	210
query84	294	98	98	98
query85	927	360	360	360
query86	420	338	330	330
query87	4367	4175	4231	4175
query88	5506	2490	2503	2490
query89	552	328	325	325
query90	2213	237	232	232
query91	155	132	132	132
query92	91	76	76	76
query93	6683	561	557	557
query94	870	329	324	324
query95	381	331	355	331
query96	638	291	281	281
query97	3273	3070	3067	3067
query98	257	228	229	228
query99	1716	1319	1326	1319
Total cold run time: 341363 ms
Total hot run time: 196642 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04df395bd13baf07e2c7c555876aa85118b5c7da, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.64	0.08	0.09
query5	0.51	0.48	0.50
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.06	0.05	0.06
query9	0.55	0.48	0.49
query10	0.55	0.56	0.55
query11	0.16	0.12	0.12
query12	0.16	0.13	0.13
query13	0.63	0.61	0.60
query14	0.78	0.84	0.79
query15	0.84	0.82	0.84
query16	0.37	0.36	0.38
query17	1.05	1.05	1.06
query18	0.21	0.20	0.21
query19	1.90	1.70	1.75
query20	0.01	0.02	0.01
query21	15.39	0.67	0.66
query22	4.21	7.96	1.51
query23	18.26	1.38	1.24
query24	2.13	0.24	0.23
query25	0.17	0.10	0.09
query26	0.27	0.18	0.18
query27	0.08	0.09	0.09
query28	13.23	1.03	1.02
query29	12.68	3.34	3.28
query30	0.39	0.19	0.19
query31	2.79	0.41	0.39
query32	3.27	0.49	0.49
query33	2.97	2.98	3.04
query34	16.88	4.34	4.35
query35	4.47	4.41	4.45
query36	0.67	0.51	0.48
query37	0.20	0.17	0.17
query38	0.18	0.17	0.17
query39	0.06	0.05	0.05
query40	0.18	0.15	0.16
query41	0.12	0.07	0.07
query42	0.08	0.07	0.07
query43	0.07	0.07	0.06
Total cold run time: 109.68 s
Total hot run time: 30.61 s

@wm1581066 wm1581066 added the usercase Important user case type label label Aug 21, 2024
@amorynan
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38441 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5ab7b07dcc3f41550ea9dea05e98b8f8f9e747e8, data reload: false

------ Round 1 ----------------------------------
q1	17616	4481	4358	4358
q2	2007	190	181	181
q3	11884	990	1134	990
q4	10519	723	810	723
q5	7772	2875	2823	2823
q6	222	139	138	138
q7	973	615	592	592
q8	9304	2077	2119	2077
q9	7276	6544	6607	6544
q10	6991	2219	2252	2219
q11	442	243	239	239
q12	400	231	226	226
q13	18723	3102	3056	3056
q14	280	235	235	235
q15	538	495	501	495
q16	493	394	397	394
q17	1008	745	784	745
q18	7469	6952	6959	6952
q19	1389	1088	1008	1008
q20	668	322	328	322
q21	3944	3102	3114	3102
q22	1126	1023	1022	1022
Total cold run time: 111044 ms
Total hot run time: 38441 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4367	4347	4285	4285
q2	376	289	265	265
q3	2928	2657	2652	2652
q4	1918	1701	1640	1640
q5	5665	5687	5756	5687
q6	230	135	134	134
q7	2215	1854	1818	1818
q8	3332	3482	3574	3482
q9	8910	8845	8811	8811
q10	3646	3439	3343	3343
q11	611	514	511	511
q12	823	643	679	643
q13	15365	3207	3238	3207
q14	332	280	298	280
q15	535	498	472	472
q16	514	458	443	443
q17	1871	1550	1540	1540
q18	8183	7950	8025	7950
q19	1794	1685	1601	1601
q20	2148	1954	1928	1928
q21	5841	5627	5473	5473
q22	1139	1056	1050	1050
Total cold run time: 72743 ms
Total hot run time: 57215 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191859 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5ab7b07dcc3f41550ea9dea05e98b8f8f9e747e8, data reload: false

query1	1256	877	853	853
query2	6347	2001	1892	1892
query3	10748	4017	4012	4012
query4	59246	24016	23239	23239
query5	5294	514	517	514
query6	399	166	171	166
query7	5736	295	296	295
query8	294	223	220	220
query9	8651	2497	2495	2495
query10	453	276	275	275
query11	16483	15155	14997	14997
query12	159	112	104	104
query13	1541	393	389	389
query14	10730	7487	7887	7487
query15	218	184	194	184
query16	7048	556	492	492
query17	1146	574	589	574
query18	1921	297	315	297
query19	268	160	148	148
query20	114	110	110	110
query21	200	106	106	106
query22	4517	4490	4564	4490
query23	34639	33612	33243	33243
query24	5905	2832	2743	2743
query25	533	404	413	404
query26	660	164	163	163
query27	1741	284	281	281
query28	3841	2098	2082	2082
query29	666	419	423	419
query30	244	151	156	151
query31	946	766	747	747
query32	81	56	57	56
query33	453	293	292	292
query34	859	488	485	485
query35	824	741	725	725
query36	1125	934	941	934
query37	143	80	81	80
query38	3949	3803	3984	3803
query39	1448	1376	1398	1376
query40	197	119	114	114
query41	45	46	43	43
query42	122	100	98	98
query43	498	459	468	459
query44	1084	738	749	738
query45	195	163	169	163
query46	1086	740	716	716
query47	1847	1776	1826	1776
query48	366	292	285	285
query49	761	427	440	427
query50	815	412	419	412
query51	7182	7067	6988	6988
query52	101	96	90	90
query53	256	179	179	179
query54	565	452	456	452
query55	76	77	77	77
query56	281	264	261	261
query57	1206	1090	1060	1060
query58	220	226	244	226
query59	2948	2698	2805	2698
query60	291	263	271	263
query61	101	96	101	96
query62	755	660	670	660
query63	223	182	180	180
query64	2668	1757	3035	1757
query65	3212	3180	3141	3141
query66	671	337	335	335
query67	15478	15277	15218	15218
query68	2987	582	586	582
query69	407	295	290	290
query70	1176	1081	1071	1071
query71	341	284	288	284
query72	2624	2068	2054	2054
query73	695	322	318	318
query74	9195	8925	8801	8801
query75	3334	2704	2712	2704
query76	1450	1061	965	965
query77	541	329	323	323
query78	9634	9129	8995	8995
query79	1050	558	538	538
query80	696	539	520	520
query81	456	234	229	229
query82	292	135	134	134
query83	173	148	151	148
query84	248	82	79	79
query85	666	287	281	281
query86	293	320	298	298
query87	4448	4314	4213	4213
query88	3390	2305	2306	2305
query89	383	285	284	284
query90	2056	206	199	199
query91	123	99	101	99
query92	63	52	52	52
query93	1065	534	541	534
query94	681	302	303	302
query95	352	341	262	262
query96	586	272	268	268
query97	3200	3021	3093	3021
query98	233	203	194	194
query99	1509	1277	1289	1277
Total cold run time: 298423 ms
Total hot run time: 191859 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5ab7b07dcc3f41550ea9dea05e98b8f8f9e747e8, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.51	0.51	0.49
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.54	0.48	0.49
query10	0.55	0.55	0.54
query11	0.16	0.12	0.12
query12	0.16	0.12	0.13
query13	0.63	0.60	0.58
query14	0.77	0.81	0.78
query15	0.84	0.83	0.84
query16	0.37	0.38	0.37
query17	0.98	1.02	0.98
query18	0.21	0.21	0.21
query19	1.88	1.80	1.87
query20	0.01	0.01	0.01
query21	15.39	0.67	0.66
query22	4.44	6.32	1.97
query23	18.29	1.44	1.37
query24	2.10	0.24	0.23
query25	0.15	0.09	0.07
query26	0.27	0.18	0.19
query27	0.07	0.07	0.08
query28	13.28	1.03	1.00
query29	12.61	3.39	3.35
query30	0.24	0.05	0.05
query31	2.88	0.40	0.39
query32	3.26	0.51	0.47
query33	3.03	2.99	3.01
query34	17.10	4.42	4.42
query35	4.48	4.53	4.48
query36	0.66	0.46	0.47
query37	0.19	0.17	0.16
query38	0.17	0.15	0.15
query39	0.05	0.03	0.04
query40	0.17	0.13	0.12
query41	0.09	0.06	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.05
Total cold run time: 109.89 s
Total hot run time: 31.05 s

HappenLee
HappenLee approved these changes Aug 27, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 27, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

eldenmoon
eldenmoon approved these changes Aug 27, 2024
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit 13bd053 into apache:master Aug 27, 2024
27 of 30 checks passed
@amorynan amorynan mentioned this pull request Aug 27, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 27, 2024
@amorynan
[fix](funcs) fix map struct construct funcs (#39973)
fc9936d 
## Proposed changes
backport: #39699
Issue Number: close #xxx

<!--Describe your changes.-->
dataroaring pushed a commit that referenced this pull request Sep 3, 2024
@amorynan @dataroaring
[fix](funcs) map struct construct funcs (#39699)
21f557a 
we should not change the column which is passed from a block when we
execute a function , because this block may be used in other expression
or other exec node
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
xiaokang pushed a commit to xiaokang/doris that referenced this pull request Sep 7, 2024
@amorynan @xiaokang
[fix](funcs) map struct construct funcs (apache#39699)
d231779 
we should not change the column which is passed from a block when we
execute a function , because this block may be used in other expression
or other exec node
@xiaokang xiaokang mentioned this pull request Sep 7, 2024
Merged
xiaokang added a commit that referenced this pull request Sep 9, 2024
@xiaokang @amorynan
[fix](funcs) map struct construct funcs #39699 (#40502)
5dc0196 
Co-authored-by: amory <wangqiannan@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions GitHub Actions GitHub Actions left review comments

@HappenLee HappenLee HappenLee approved these changes

@eldenmoon eldenmoon eldenmoon approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.2-merged p0_b reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@amorynan @doris-robot @HappenLee @eldenmoon @xiaokang @yiguolei @wm1581066 @dataroaring