Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvement](nereids) Support * EXCEPT, * REPLACE and other expressions to appear in select statement at the same time #40014

Merged
merged 5 commits into from
Aug 29, 2024

Conversation

liutang123
Copy link
Contributor

@liutang123 liutang123 commented Aug 28, 2024

Follow #36777

Sometimes, users want to use * EXCEPT and other expressions in select clause.
For example:
SELECT * EXCEPT (value), sum(value) FROM tbl GROUP BY 1

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@liutang123
Copy link
Contributor Author

run buildall

@liutang123
Copy link
Contributor Author

Will add ut and rt later.

@doris-robot
Copy link

TPC-H: Total hot run time: 38189 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4acbc1a0fe027d802d991135d32fde6fc1c0cdb, data reload: false

------ Round 1 ----------------------------------
q1	17744	4766	4332	4332
q2	2019	183	178	178
q3	11792	972	1126	972
q4	10513	757	771	757
q5	7746	2834	2810	2810
q6	228	138	135	135
q7	963	616	603	603
q8	9518	2079	2046	2046
q9	7638	6542	6543	6542
q10	6990	2289	2255	2255
q11	444	250	248	248
q12	406	221	223	221
q13	17978	3050	3054	3050
q14	275	242	233	233
q15	532	492	472	472
q16	623	521	502	502
q17	985	629	668	629
q18	7239	6841	6868	6841
q19	1385	1077	1084	1077
q20	685	328	321	321
q21	3970	3074	2966	2966
q22	1117	1019	999	999
Total cold run time: 110790 ms
Total hot run time: 38189 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4367	4323	4312	4312
q2	372	277	272	272
q3	2929	2680	2610	2610
q4	1951	1639	1647	1639
q5	5664	5665	5689	5665
q6	224	136	142	136
q7	2236	1860	1847	1847
q8	3336	3482	3496	3482
q9	8899	8867	8864	8864
q10	3626	3418	3345	3345
q11	603	517	514	514
q12	814	659	665	659
q13	14166	3193	3252	3193
q14	343	292	293	292
q15	535	500	498	498
q16	618	559	567	559
q17	1854	1548	1564	1548
q18	8128	7927	7837	7837
q19	1747	1615	1657	1615
q20	2179	1900	1921	1900
q21	5888	5475	5498	5475
q22	1119	1036	1018	1018
Total cold run time: 71598 ms
Total hot run time: 57280 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193816 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c4acbc1a0fe027d802d991135d32fde6fc1c0cdb, data reload: false

query1	1266	885	870	870
query2	6408	2031	1961	1961
query3	10622	3981	3925	3925
query4	59336	25751	23425	23425
query5	5343	499	492	492
query6	409	158	168	158
query7	5825	303	297	297
query8	289	213	203	203
query9	9041	2524	2525	2524
query10	494	278	275	275
query11	16737	15139	15402	15139
query12	156	104	107	104
query13	1570	413	382	382
query14	11364	7442	7555	7442
query15	227	179	178	178
query16	7426	501	461	461
query17	1168	609	593	593
query18	1332	292	296	292
query19	292	149	141	141
query20	121	110	120	110
query21	212	108	109	108
query22	4808	4315	4565	4315
query23	34473	33225	33793	33225
query24	5950	2896	2895	2895
query25	532	389	391	389
query26	684	153	156	153
query27	1808	275	287	275
query28	3767	2129	2120	2120
query29	696	405	412	405
query30	236	150	158	150
query31	943	755	734	734
query32	85	54	57	54
query33	465	286	292	286
query34	880	475	493	475
query35	846	715	727	715
query36	1064	954	941	941
query37	144	98	98	98
query38	4017	3983	3909	3909
query39	1470	1411	1374	1374
query40	213	117	116	116
query41	48	45	44	44
query42	118	96	96	96
query43	513	483	487	483
query44	1086	750	748	748
query45	198	167	166	166
query46	1104	771	774	771
query47	1891	1789	1798	1789
query48	364	294	293	293
query49	757	441	447	441
query50	818	429	418	418
query51	7238	7084	7079	7079
query52	100	89	87	87
query53	255	189	187	187
query54	568	474	459	459
query55	80	76	81	76
query56	291	273	253	253
query57	1185	1061	1111	1061
query58	240	235	239	235
query59	3003	2849	2767	2767
query60	329	289	288	288
query61	124	121	120	120
query62	742	650	692	650
query63	226	188	191	188
query64	2947	767	767	767
query65	3258	3122	3148	3122
query66	638	357	347	347
query67	15396	15382	15322	15322
query68	3302	597	581	581
query69	409	285	278	278
query70	1148	1105	1175	1105
query71	352	282	281	281
query72	6432	4328	4094	4094
query73	757	334	337	334
query74	9115	8873	8951	8873
query75	3494	2702	2770	2702
query76	1440	1014	1021	1014
query77	508	326	320	320
query78	9744	9136	8966	8966
query79	2098	532	550	532
query80	991	518	504	504
query81	569	236	237	236
query82	252	148	150	148
query83	226	169	154	154
query84	264	79	79	79
query85	917	302	284	284
query86	404	299	283	283
query87	4527	4284	4243	4243
query88	3598	2316	2319	2316
query89	397	288	286	286
query90	1837	221	192	192
query91	123	101	101	101
query92	68	51	53	51
query93	2120	541	527	527
query94	790	308	295	295
query95	358	265	264	264
query96	627	265	266	265
query97	3219	3062	3065	3062
query98	236	202	213	202
query99	1537	1295	1278	1278
Total cold run time: 308095 ms
Total hot run time: 193816 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c4acbc1a0fe027d802d991135d32fde6fc1c0cdb, data reload: false

query1	0.04	0.05	0.04
query2	0.07	0.05	0.04
query3	0.23	0.05	0.04
query4	1.67	0.07	0.07
query5	0.51	0.50	0.49
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.55	0.53	0.53
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.59	0.59
query14	2.02	2.04	2.10
query15	0.86	0.82	0.81
query16	0.37	0.36	0.40
query17	1.07	1.05	1.03
query18	0.21	0.20	0.21
query19	1.96	1.87	1.85
query20	0.01	0.01	0.01
query21	15.40	0.67	0.66
query22	4.25	6.08	3.22
query23	18.33	1.46	1.28
query24	2.09	0.23	0.22
query25	0.16	0.09	0.09
query26	0.27	0.18	0.19
query27	0.08	0.07	0.07
query28	13.25	1.04	1.01
query29	12.62	3.29	3.35
query30	0.24	0.05	0.05
query31	2.87	0.40	0.40
query32	3.26	0.50	0.48
query33	2.97	3.02	3.00
query34	17.14	4.37	4.36
query35	4.45	4.43	4.44
query36	0.65	0.48	0.50
query37	0.20	0.16	0.16
query38	0.16	0.16	0.15
query39	0.04	0.04	0.04
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.05	0.06	0.05
query43	0.05	0.05	0.04
Total cold run time: 111.04 s
Total hot run time: 33.43 s

@liutang123
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39065 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ac6db5ac1f2b1e4727a3ac44d7d5db8e52bde434, data reload: false

------ Round 1 ----------------------------------
q1	17613	4502	4357	4357
q2	2009	206	207	206
q3	10945	1267	1086	1086
q4	10543	819	810	810
q5	8054	2889	2834	2834
q6	235	147	145	145
q7	995	647	638	638
q8	9360	2104	2108	2104
q9	7569	6653	6655	6653
q10	7456	2322	2261	2261
q11	471	256	259	256
q12	401	235	236	235
q13	17759	3066	3046	3046
q14	286	232	236	232
q15	513	490	500	490
q16	599	512	502	502
q17	1003	669	676	669
q18	7489	7013	7031	7013
q19	1387	1072	1071	1071
q20	674	353	332	332
q21	3962	3103	3213	3103
q22	1164	1022	1055	1022
Total cold run time: 110487 ms
Total hot run time: 39065 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4416	4325	4294	4294
q2	390	280	280	280
q3	2909	2718	2628	2628
q4	1947	1682	1719	1682
q5	5683	5790	5628	5628
q6	227	134	141	134
q7	2218	1805	1845	1805
q8	3407	3459	3454	3454
q9	8890	8756	8851	8756
q10	3578	3384	3320	3320
q11	620	528	529	528
q12	853	632	642	632
q13	10499	3185	3201	3185
q14	321	306	298	298
q15	525	506	519	506
q16	614	583	585	583
q17	1875	1583	1511	1511
q18	8309	8082	7896	7896
q19	1757	1552	1586	1552
q20	2156	1851	1874	1851
q21	5658	5540	5474	5474
q22	1189	1092	1059	1059
Total cold run time: 68041 ms
Total hot run time: 57056 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193498 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ac6db5ac1f2b1e4727a3ac44d7d5db8e52bde434, data reload: false

query1	1267	878	873	873
query2	6096	2036	1934	1934
query3	10730	4157	4045	4045
query4	59015	26238	23162	23162
query5	5363	510	500	500
query6	408	172	187	172
query7	5755	295	292	292
query8	285	204	207	204
query9	9187	2510	2492	2492
query10	485	275	267	267
query11	18244	15041	15344	15041
query12	167	103	105	103
query13	1562	382	378	378
query14	11046	7556	7412	7412
query15	214	191	183	183
query16	7339	484	434	434
query17	1120	587	575	575
query18	1885	305	297	297
query19	272	153	144	144
query20	130	113	116	113
query21	219	109	106	106
query22	4571	4410	4348	4348
query23	34284	33871	33414	33414
query24	5952	2828	2826	2826
query25	506	375	377	375
query26	680	166	167	166
query27	1755	287	279	279
query28	3641	2125	2153	2125
query29	658	410	403	403
query30	241	154	158	154
query31	950	743	781	743
query32	78	54	60	54
query33	437	294	297	294
query34	851	468	479	468
query35	852	717	728	717
query36	1035	937	897	897
query37	147	91	98	91
query38	3964	3879	3801	3801
query39	1448	1409	1414	1409
query40	200	118	120	118
query41	47	46	45	45
query42	118	101	103	101
query43	530	473	482	473
query44	1078	752	755	752
query45	198	164	163	163
query46	1087	753	779	753
query47	1902	1796	1804	1796
query48	382	299	307	299
query49	780	441	468	441
query50	809	420	430	420
query51	7129	7027	7124	7027
query52	102	93	94	93
query53	256	183	184	183
query54	569	458	463	458
query55	80	77	78	77
query56	298	278	273	273
query57	1206	1081	1086	1081
query58	233	230	233	230
query59	3060	2765	2716	2716
query60	305	282	286	282
query61	123	118	121	118
query62	746	653	671	653
query63	224	191	186	186
query64	2511	767	745	745
query65	3248	3123	3154	3123
query66	687	351	352	351
query67	15597	15527	15467	15467
query68	3055	595	606	595
query69	414	285	291	285
query70	1203	1116	1112	1112
query71	368	281	278	278
query72	6519	4304	4028	4028
query73	741	334	337	334
query74	9014	8671	8872	8671
query75	3342	2710	2640	2640
query76	1527	1009	1007	1007
query77	573	328	323	323
query78	10139	9158	9034	9034
query79	1084	550	534	534
query80	896	509	505	505
query81	569	240	231	231
query82	247	191	155	155
query83	257	150	153	150
query84	266	80	73	73
query85	933	289	287	287
query86	338	303	310	303
query87	4466	4304	4354	4304
query88	3402	2343	2340	2340
query89	388	284	295	284
query90	1849	200	197	197
query91	132	104	105	104
query92	63	54	54	54
query93	1659	584	544	544
query94	777	311	303	303
query95	350	273	275	273
query96	607	269	273	269
query97	3272	3107	3095	3095
query98	214	203	196	196
query99	1581	1294	1275	1275
Total cold run time: 306589 ms
Total hot run time: 193498 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ac6db5ac1f2b1e4727a3ac44d7d5db8e52bde434, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.68	0.09	0.09
query5	0.52	0.49	0.51
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.54	0.49	0.48
query10	0.53	0.53	0.53
query11	0.15	0.12	0.11
query12	0.15	0.13	0.12
query13	0.61	0.59	0.59
query14	2.06	2.14	2.05
query15	0.90	0.82	0.82
query16	0.36	0.38	0.38
query17	1.05	1.08	1.08
query18	0.22	0.20	0.22
query19	1.92	1.76	1.83
query20	0.01	0.02	0.01
query21	15.40	0.68	0.67
query22	3.88	8.47	1.69
query23	18.20	1.37	1.27
query24	2.09	0.22	0.24
query25	0.14	0.08	0.09
query26	0.27	0.17	0.17
query27	0.09	0.07	0.08
query28	13.26	1.04	1.01
query29	12.60	3.32	3.32
query30	0.24	0.05	0.05
query31	2.90	0.41	0.41
query32	3.23	0.49	0.49
query33	2.99	2.99	3.04
query34	17.21	4.41	4.39
query35	4.52	4.46	4.47
query36	0.67	0.47	0.51
query37	0.21	0.15	0.16
query38	0.17	0.15	0.15
query39	0.05	0.03	0.03
query40	0.16	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.73 s
Total hot run time: 31.97 s

@924060929
Copy link
Contributor

add test for

SELECT * EXCEPT (value), sum(value) FROM tbl GROUP BY 1

morrySnow
morrySnow previously approved these changes Aug 28, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 28, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 28, 2024
@liutang123
Copy link
Contributor Author

run buildall

@liutang123
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38648 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 74ff98de7435108c0afbcf8d3813653cdf90434b, data reload: false

------ Round 1 ----------------------------------
q1	18162	4467	4407	4407
q2	2849	188	180	180
q3	11098	1217	1106	1106
q4	10208	808	751	751
q5	8396	2899	2859	2859
q6	235	145	146	145
q7	999	642	599	599
q8	9336	2059	2068	2059
q9	7069	6558	6558	6558
q10	7005	2222	2193	2193
q11	433	241	250	241
q12	397	226	221	221
q13	17783	3045	3058	3045
q14	264	242	239	239
q15	507	475	491	475
q16	588	508	508	508
q17	973	690	668	668
q18	7281	6979	6899	6899
q19	1388	1099	1120	1099
q20	668	328	347	328
q21	3874	3073	3058	3058
q22	1132	1025	1010	1010
Total cold run time: 110645 ms
Total hot run time: 38648 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4396	4300	4403	4300
q2	379	283	288	283
q3	2937	2682	2626	2626
q4	1893	1665	1655	1655
q5	5430	5380	5434	5380
q6	221	136	136	136
q7	2125	1709	1768	1709
q8	3191	3353	3407	3353
q9	8493	8432	8407	8407
q10	3470	3240	3156	3156
q11	604	494	505	494
q12	793	611	612	611
q13	10637	3065	3090	3065
q14	303	272	280	272
q15	534	481	494	481
q16	594	564	559	559
q17	1791	1504	1458	1458
q18	7759	7419	7619	7419
q19	1662	1611	1652	1611
q20	2054	1825	1845	1825
q21	5420	5216	5289	5216
q22	1148	1063	1042	1042
Total cold run time: 65834 ms
Total hot run time: 55058 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188210 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 74ff98de7435108c0afbcf8d3813653cdf90434b, data reload: false

query1	915	370	368	368
query2	6445	1985	1868	1868
query3	6638	214	224	214
query4	34060	23288	23025	23025
query5	4151	509	492	492
query6	252	161	161	161
query7	4591	300	295	295
query8	259	210	213	210
query9	8561	2477	2483	2477
query10	439	290	258	258
query11	17642	14842	14962	14842
query12	150	102	99	99
query13	1628	383	383	383
query14	9721	6738	7011	6738
query15	230	178	172	172
query16	7725	493	477	477
query17	1598	560	561	560
query18	1494	302	304	302
query19	193	151	150	150
query20	118	113	112	112
query21	208	108	104	104
query22	4620	4167	4400	4167
query23	34080	33677	33674	33674
query24	11214	2877	2866	2866
query25	651	408	414	408
query26	1277	164	162	162
query27	2646	292	283	283
query28	7473	2140	2121	2121
query29	824	432	426	426
query30	305	164	153	153
query31	987	743	801	743
query32	96	58	62	58
query33	769	285	290	285
query34	1001	485	490	485
query35	853	727	722	722
query36	1101	939	948	939
query37	154	92	88	88
query38	3914	3877	3967	3877
query39	1446	1425	1391	1391
query40	278	123	119	119
query41	52	47	49	47
query42	120	100	100	100
query43	531	475	487	475
query44	1245	767	755	755
query45	202	165	171	165
query46	1108	726	738	726
query47	1907	1841	1835	1835
query48	374	309	299	299
query49	1104	444	439	439
query50	805	405	421	405
query51	7255	7041	7004	7004
query52	99	88	93	88
query53	256	185	184	184
query54	1104	463	456	456
query55	89	81	79	79
query56	287	272	275	272
query57	1224	1086	1103	1086
query58	253	235	234	234
query59	2997	2798	2916	2798
query60	317	304	290	290
query61	129	117	121	117
query62	844	677	659	659
query63	231	187	187	187
query64	5316	785	661	661
query65	3223	3138	3108	3108
query66	1293	362	328	328
query67	15647	15269	15528	15269
query68	4801	574	559	559
query69	563	362	302	302
query70	1191	1052	1113	1052
query71	429	278	272	272
query72	6669	4044	4012	4012
query73	779	340	350	340
query74	9363	8832	8683	8683
query75	3563	2691	2665	2665
query76	3417	986	1013	986
query77	682	330	316	316
query78	9793	10078	9434	9434
query79	1992	534	551	534
query80	901	510	503	503
query81	579	236	233	233
query82	603	145	151	145
query83	176	161	155	155
query84	277	82	73	73
query85	739	280	279	279
query86	455	302	298	298
query87	4440	4316	4200	4200
query88	3354	2335	2333	2333
query89	382	286	288	286
query90	1842	195	191	191
query91	124	102	99	99
query92	64	55	51	51
query93	1132	553	541	541
query94	782	308	297	297
query95	357	265	257	257
query96	596	315	267	267
query97	3252	3132	3050	3050
query98	230	214	195	195
query99	1582	1262	1298	1262
Total cold run time: 293828 ms
Total hot run time: 188210 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.44 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 74ff98de7435108c0afbcf8d3813653cdf90434b, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.05	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.51	0.52	0.50
query6	1.13	0.74	0.73
query7	0.02	0.02	0.02
query8	0.05	0.05	0.05
query9	0.56	0.49	0.48
query10	0.54	0.54	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.59	0.58
query14	2.04	2.05	2.06
query15	0.83	0.83	0.80
query16	0.38	0.39	0.37
query17	1.08	1.03	1.06
query18	0.21	0.20	0.21
query19	1.96	1.86	1.78
query20	0.01	0.01	0.01
query21	15.56	0.66	0.66
query22	4.63	6.36	2.30
query23	18.23	1.39	1.32
query24	2.08	0.24	0.22
query25	0.16	0.10	0.08
query26	0.26	0.17	0.18
query27	0.08	0.08	0.08
query28	13.26	1.01	1.00
query29	12.61	3.34	3.28
query30	0.24	0.07	0.05
query31	2.87	0.40	0.39
query32	3.25	0.48	0.48
query33	2.96	2.99	2.98
query34	16.93	4.43	4.39
query35	4.43	4.40	4.46
query36	0.64	0.49	0.47
query37	0.19	0.16	0.16
query38	0.16	0.16	0.15
query39	0.05	0.03	0.04
query40	0.16	0.12	0.12
query41	0.10	0.04	0.04
query42	0.07	0.05	0.06
query43	0.05	0.05	0.05
Total cold run time: 111.25 s
Total hot run time: 32.44 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 29, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit 9ce6671 into apache:master Aug 29, 2024
28 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants