Skip to content

[feat](nereids) add rewrite rule :EliminateGroupByKeyByUniform (#43391) #45075

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Dec 10, 2024
Merged

[feat](nereids) add rewrite rule :EliminateGroupByKeyByUniform (#43391) #45075

morrySnow merged 2 commits into from
Dec 10, 2024

Conversation

@feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Dec 6, 2024

cherry-pick #43391 to branch-3.0

@feiniaofeiafei
[feat](nereids) add rewrite rule :EliminateGroupByKeyByUniform (apach…
0bcf46c 
…e#43391)

This PR introduces two main changes:
1. Adds an optional constant value to the uniform attribute in
DataTrait. A slot with a constant value that is not null will be
considered uniform and not null.
2. Introduces a new transform rule: EliminateGroupByKeyByUniform, which
utilizes the newly added part of the uniform attribute. Following is
example transformation:

from

 +--aggregate(group by a,b output a,b,max(c))
(a is uniform and not null: e.g. a is projection 2 as a in
logicalProject)

to

+--aggregate(group by b output b,any_value(a) as a,max(c))
@doris-robot
Copy link

doris-robot commented Dec 6, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Dec 6, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Dec 6, 2024

TPC-H: Total hot run time: 40614 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0bcf46c3cceed1150821fe8e9c3c8b7c83efe502, data reload: false

------ Round 1 ----------------------------------
q1	17567	7526	7268	7268
q2	2070	168	167	167
q3	10835	1111	1200	1111
q4	10561	760	724	724
q5	7783	2821	2765	2765
q6	238	148	149	148
q7	982	609	614	609
q8	9350	1962	1972	1962
q9	6615	6427	6421	6421
q10	6999	2276	2341	2276
q11	460	260	251	251
q12	415	223	211	211
q13	17785	2941	2971	2941
q14	262	208	209	208
q15	558	517	577	517
q16	676	602	611	602
q17	969	526	573	526
q18	7253	6573	6585	6573
q19	3585	1040	1010	1010
q20	487	196	193	193
q21	3981	3207	3167	3167
q22	1056	976	964	964
Total cold run time: 110487 ms
Total hot run time: 40614 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7365	7203	7192	7192
q2	331	239	224	224
q3	2971	2898	2866	2866
q4	2059	1786	1740	1740
q5	5690	5698	5720	5698
q6	219	141	145	141
q7	2187	1760	1782	1760
q8	3311	3552	3420	3420
q9	8946	8900	8853	8853
q10	3553	3545	3547	3545
q11	608	500	487	487
q12	830	634	588	588
q13	16414	3108	3133	3108
q14	309	276	272	272
q15	574	534	512	512
q16	704	659	678	659
q17	1849	1614	1611	1611
q18	8255	7731	7592	7592
q19	4478	1643	1539	1539
q20	2080	1865	1867	1865
q21	5795	5290	5267	5267
q22	1125	1032	995	995
Total cold run time: 79653 ms
Total hot run time: 59934 ms

@doris-robot
Copy link

doris-robot commented Dec 6, 2024

TPC-DS: Total hot run time: 194843 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0bcf46c3cceed1150821fe8e9c3c8b7c83efe502, data reload: false

query1	1275	929	912	912
query2	6227	2142	2080	2080
query3	10836	4181	4308	4181
query4	66657	29396	23451	23451
query5	5387	433	428	428
query6	416	179	174	174
query7	5652	315	315	315
query8	318	240	232	232
query9	9504	2689	2658	2658
query10	522	267	255	255
query11	17834	15117	15817	15117
query12	165	103	107	103
query13	1580	442	435	435
query14	10708	6644	6858	6644
query15	215	178	184	178
query16	8040	500	466	466
query17	1051	583	592	583
query18	2144	319	326	319
query19	207	159	154	154
query20	115	107	111	107
query21	208	105	104	104
query22	4701	4296	4536	4296
query23	34606	33796	34230	33796
query24	6175	2955	2892	2892
query25	530	413	410	410
query26	690	182	173	173
query27	1727	305	306	305
query28	4196	2550	2563	2550
query29	711	455	437	437
query30	249	165	179	165
query31	1015	822	848	822
query32	64	52	53	52
query33	428	269	266	266
query34	901	498	510	498
query35	850	742	734	734
query36	1057	967	961	961
query37	117	73	71	71
query38	4115	4076	3963	3963
query39	1544	1520	1467	1467
query40	208	96	96	96
query41	47	46	45	45
query42	113	95	93	93
query43	536	511	515	511
query44	1192	821	810	810
query45	182	164	167	164
query46	1135	702	724	702
query47	1981	1879	1916	1879
query48	480	379	374	374
query49	729	388	375	375
query50	832	416	415	415
query51	7380	7204	7072	7072
query52	97	88	84	84
query53	247	179	177	177
query54	557	450	442	442
query55	76	72	75	72
query56	239	231	224	224
query57	1197	1096	1115	1096
query58	202	195	199	195
query59	3137	2981	2832	2832
query60	272	250	243	243
query61	108	102	100	100
query62	771	654	649	649
query63	204	178	191	178
query64	1807	635	649	635
query65	3280	3161	3201	3161
query66	716	296	297	296
query67	15951	15327	15258	15258
query68	4563	545	561	545
query69	418	255	257	255
query70	1202	1122	1159	1122
query71	392	260	255	255
query72	6257	3898	4002	3898
query73	753	342	335	335
query74	10152	8886	8915	8886
query75	3312	2593	2637	2593
query76	2020	1026	933	933
query77	513	258	265	258
query78	10658	9696	9534	9534
query79	6829	585	591	585
query80	1857	427	422	422
query81	540	246	241	241
query82	1102	121	112	112
query83	261	146	138	138
query84	281	82	75	75
query85	1746	305	287	287
query86	470	296	304	296
query87	4578	4218	4216	4216
query88	5534	2479	2348	2348
query89	421	296	289	289
query90	2046	183	181	181
query91	177	143	143	143
query92	66	50	47	47
query93	5967	534	533	533
query94	895	275	290	275
query95	354	248	247	247
query96	630	276	272	272
query97	3360	3098	3141	3098
query98	221	212	198	198
query99	1625	1294	1311	1294
Total cold run time: 335613 ms
Total hot run time: 194843 ms

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Dec 6, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Dec 6, 2024

TPC-H: Total hot run time: 40643 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9de5d5c54c7cced91988bc7a72def1261c4aed4f, data reload: false

------ Round 1 ----------------------------------
q1	17616	7440	7368	7368
q2	2048	179	157	157
q3	10663	1113	1175	1113
q4	10550	743	698	698
q5	7737	2850	2795	2795
q6	233	148	148	148
q7	972	606	611	606
q8	9346	1971	1968	1968
q9	6637	6449	6442	6442
q10	7005	2305	2304	2304
q11	457	262	264	262
q12	404	221	218	218
q13	17789	2956	3005	2956
q14	232	213	207	207
q15	563	515	522	515
q16	692	594	602	594
q17	982	581	544	544
q18	7326	6596	6456	6456
q19	2820	1037	1152	1037
q20	484	199	187	187
q21	4170	3226	3106	3106
q22	1091	996	962	962
Total cold run time: 109817 ms
Total hot run time: 40643 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7436	7300	7247	7247
q2	326	234	227	227
q3	2964	2865	2867	2865
q4	1994	1825	1737	1737
q5	5684	5760	5778	5760
q6	239	143	145	143
q7	2191	1801	1770	1770
q8	3335	3528	3426	3426
q9	8895	8951	8850	8850
q10	3572	3504	3486	3486
q11	596	506	500	500
q12	840	619	612	612
q13	16531	3141	3118	3118
q14	305	267	278	267
q15	559	518	538	518
q16	700	652	677	652
q17	1861	1623	1592	1592
q18	8318	7882	7389	7389
q19	1752	1537	1605	1537
q20	2070	1897	1844	1844
q21	5510	5360	5333	5333
q22	1083	995	996	995
Total cold run time: 76761 ms
Total hot run time: 59868 ms

@doris-robot
Copy link

doris-robot commented Dec 6, 2024

TPC-DS: Total hot run time: 195818 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9de5d5c54c7cced91988bc7a72def1261c4aed4f, data reload: false

query1	1240	958	919	919
query2	6239	2141	2086	2086
query3	10858	4157	4081	4081
query4	66329	29556	23332	23332
query5	5520	447	424	424
query6	447	179	175	175
query7	5683	316	316	316
query8	324	229	238	229
query9	9278	2705	2677	2677
query10	467	283	250	250
query11	17916	15344	15750	15344
query12	151	103	115	103
query13	1550	426	417	417
query14	10112	7543	7180	7180
query15	210	184	176	176
query16	7744	519	523	519
query17	1060	584	588	584
query18	1944	319	322	319
query19	239	159	160	159
query20	115	110	113	110
query21	214	112	105	105
query22	4713	4397	4444	4397
query23	34355	34132	34238	34132
query24	6125	2892	2884	2884
query25	561	410	436	410
query26	696	184	173	173
query27	2036	300	307	300
query28	4248	2515	2532	2515
query29	723	470	458	458
query30	248	164	172	164
query31	1046	831	846	831
query32	66	55	57	55
query33	444	284	297	284
query34	932	498	515	498
query35	870	743	720	720
query36	1119	979	956	956
query37	119	68	70	68
query38	4031	4002	3960	3960
query39	1545	1472	1459	1459
query40	194	94	99	94
query41	47	46	46	46
query42	109	97	98	97
query43	548	481	504	481
query44	1177	805	789	789
query45	186	163	172	163
query46	1175	756	738	738
query47	1970	1914	1929	1914
query48	453	376	376	376
query49	754	366	384	366
query50	863	425	427	425
query51	7364	7331	6957	6957
query52	93	93	88	88
query53	269	181	182	181
query54	577	432	439	432
query55	79	76	75	75
query56	238	234	230	230
query57	1221	1083	1104	1083
query58	200	199	219	199
query59	3279	3089	3018	3018
query60	271	238	259	238
query61	105	106	107	106
query62	759	637	646	637
query63	214	192	201	192
query64	1752	637	623	623
query65	3246	3172	3160	3160
query66	713	296	299	296
query67	15707	15302	15204	15204
query68	4467	552	551	551
query69	429	253	250	250
query70	1111	1068	1125	1068
query71	357	250	243	243
query72	6448	3926	3875	3875
query73	797	344	335	335
query74	10231	8853	8944	8853
query75	3341	2619	2635	2619
query76	2280	1048	1041	1041
query77	474	256	264	256
query78	10718	9761	9433	9433
query79	8066	590	587	587
query80	1931	427	417	417
query81	560	246	243	243
query82	1181	115	114	114
query83	259	143	146	143
query84	289	77	81	77
query85	1640	306	295	295
query86	447	306	281	281
query87	4389	4285	4214	4214
query88	5696	2335	2371	2335
query89	486	300	295	295
query90	2108	184	186	184
query91	197	141	145	141
query92	63	47	48	47
query93	6254	533	524	524
query94	881	267	276	267
query95	344	247	251	247
query96	651	280	277	277
query97	3370	3152	3218	3152
query98	219	200	195	195
query99	1583	1282	1286	1282
Total cold run time: 336347 ms
Total hot run time: 195818 ms

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Dec 6, 2024

run cloud_p0

@morrySnow morrySnow merged commit 8a61eb9 into apache:branch-3.0 Dec 10, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@feiniaofeiafei @doris-robot @morrySnow