Skip to content

Conversation

@feiniaofeiafei
Copy link
Contributor

pick #41731 to branch-3.0

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41211 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d88d3f6a842113662fca09e7a85b6f7261cdac63, data reload: false

------ Round 1 ----------------------------------
q1	17602	7475	7403	7403
q2	2026	281	274	274
q3	13626	1087	1145	1087
q4	10545	749	733	733
q5	7756	2911	2725	2725
q6	240	150	148	148
q7	976	617	614	614
q8	9894	1930	1996	1930
q9	6563	6442	6466	6442
q10	7131	2359	2356	2356
q11	465	256	246	246
q12	400	213	217	213
q13	18448	3150	3164	3150
q14	240	222	224	222
q15	573	546	537	537
q16	843	630	632	630
q17	1097	517	599	517
q18	9413	6673	6768	6673
q19	5027	1023	1165	1023
q20	455	197	187	187
q21	3885	3165	3123	3123
q22	1075	983	978	978
Total cold run time: 118280 ms
Total hot run time: 41211 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7433	7201	7395	7201
q2	336	238	236	236
q3	2967	2971	2909	2909
q4	2018	1753	1762	1753
q5	5927	5690	5700	5690
q6	221	140	136	136
q7	2187	1790	1752	1752
q8	3284	3443	3485	3443
q9	8839	8779	8777	8777
q10	3546	3518	3499	3499
q11	573	485	481	481
q12	852	613	606	606
q13	16589	3120	3176	3120
q14	323	293	269	269
q15	576	522	523	522
q16	693	665	654	654
q17	1846	1648	1571	1571
q18	8204	7854	7446	7446
q19	1754	1587	1650	1587
q20	2083	1866	1851	1851
q21	7968	5349	5326	5326
q22	1121	1044	1016	1016
Total cold run time: 79340 ms
Total hot run time: 59845 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189481 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d88d3f6a842113662fca09e7a85b6f7261cdac63, data reload: false

query1	816	387	376	376
query2	6231	2128	2034	2034
query3	8698	211	212	211
query4	33477	23398	23316	23316
query5	3801	447	432	432
query6	279	182	160	160
query7	4194	313	302	302
query8	292	227	242	227
query9	9219	2677	2634	2634
query10	472	258	251	251
query11	18025	15227	15148	15148
query12	143	103	100	100
query13	1529	399	399	399
query14	8527	6176	6960	6176
query15	215	170	173	170
query16	7687	469	491	469
query17	1309	556	558	556
query18	1966	332	340	332
query19	211	161	167	161
query20	118	119	111	111
query21	205	113	116	113
query22	4798	4505	4477	4477
query23	35013	35035	34021	34021
query24	12227	2867	2793	2793
query25	542	399	409	399
query26	1053	168	181	168
query27	2861	301	300	300
query28	8097	2471	2460	2460
query29	694	442	434	434
query30	263	162	156	156
query31	1024	794	808	794
query32	98	54	57	54
query33	781	290	279	279
query34	1007	523	499	499
query35	831	719	693	693
query36	1067	928	939	928
query37	195	73	72	72
query38	3895	3879	3836	3836
query39	1466	1461	1446	1446
query40	254	99	100	99
query41	52	48	51	48
query42	115	111	102	102
query43	538	489	480	480
query44	1150	769	781	769
query45	196	164	165	164
query46	1137	724	704	704
query47	1928	1822	1821	1821
query48	467	368	356	356
query49	1009	394	398	394
query50	791	395	424	395
query51	6998	6899	6963	6899
query52	99	99	98	98
query53	254	182	181	181
query54	1067	441	472	441
query55	82	79	77	77
query56	276	250	240	240
query57	1205	1126	1109	1109
query58	236	211	207	207
query59	3108	3024	2866	2866
query60	279	246	253	246
query61	103	104	103	103
query62	824	665	681	665
query63	213	183	185	183
query64	4839	631	606	606
query65	3276	3155	3143	3143
query66	1025	331	302	302
query67	15834	15595	15386	15386
query68	5643	545	536	536
query69	419	264	263	263
query70	1221	1146	1146	1146
query71	406	257	255	255
query72	5813	3938	3931	3931
query73	759	351	339	339
query74	10088	8789	8931	8789
query75	3346	2652	2622	2622
query76	3458	874	1179	874
query77	446	278	268	268
query78	10710	9797	9439	9439
query79	7120	589	579	579
query80	2384	434	454	434
query81	588	238	243	238
query82	1045	115	117	115
query83	267	163	141	141
query84	303	79	75	75
query85	2344	303	283	283
query86	489	280	298	280
query87	4450	4409	4329	4329
query88	5518	2375	2378	2375
query89	563	289	292	289
query90	2310	185	188	185
query91	184	144	142	142
query92	64	49	49	49
query93	6717	537	523	523
query94	1128	290	271	271
query95	346	247	263	247
query96	632	280	276	276
query97	3362	3110	3152	3110
query98	229	201	194	194
query99	1704	1313	1318	1313
Total cold run time: 314737 ms
Total hot run time: 189481 ms

@feiniaofeiafei feiniaofeiafei force-pushed the pick_extend_infer_predicates branch from d88d3f6 to 6bc303a Compare October 25, 2024 08:02
@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40241 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6bc303a26ff81c12b9cea872b44b16618bb8ef95, data reload: false

------ Round 1 ----------------------------------
q1	17557	7577	7238	7238
q2	2030	268	273	268
q3	11919	1037	1176	1037
q4	10566	747	700	700
q5	7730	2800	2783	2783
q6	232	148	147	147
q7	964	602	603	602
q8	9583	1893	1930	1893
q9	7995	6407	6365	6365
q10	6948	2306	2299	2299
q11	433	245	249	245
q12	401	214	209	209
q13	17768	2980	2920	2920
q14	238	204	210	204
q15	569	513	516	513
q16	677	613	595	595
q17	956	563	563	563
q18	7049	6612	6536	6536
q19	1369	1110	928	928
q20	465	192	199	192
q21	3878	3044	3104	3044
q22	1058	982	960	960
Total cold run time: 110385 ms
Total hot run time: 40241 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7285	7175	7203	7175
q2	323	223	222	222
q3	2936	2860	2785	2785
q4	2000	1757	1714	1714
q5	5613	5622	5683	5622
q6	225	139	145	139
q7	2153	1761	1752	1752
q8	3267	3459	3456	3456
q9	8729	8805	8708	8708
q10	3525	3495	3511	3495
q11	581	466	480	466
q12	787	591	630	591
q13	16530	3099	3117	3099
q14	299	284	266	266
q15	567	506	502	502
q16	717	679	649	649
q17	1817	1597	1556	1556
q18	8158	7830	7318	7318
q19	4867	1595	1629	1595
q20	2075	1866	1826	1826
q21	5189	5175	5108	5108
q22	1121	1038	1023	1023
Total cold run time: 78764 ms
Total hot run time: 59067 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188798 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bc303a26ff81c12b9cea872b44b16618bb8ef95, data reload: false

query1	900	393	410	393
query2	6265	2079	2028	2028
query3	8690	206	206	206
query4	33985	23348	23371	23348
query5	3806	433	444	433
query6	290	193	169	169
query7	4189	321	307	307
query8	321	234	227	227
query9	9606	2664	2659	2659
query10	453	256	253	253
query11	18025	15043	15136	15043
query12	147	100	103	100
query13	1536	411	400	400
query14	9593	6895	6111	6111
query15	248	174	168	168
query16	7976	487	477	477
query17	1589	550	604	550
query18	2136	327	322	322
query19	209	162	166	162
query20	125	115	113	113
query21	211	109	113	109
query22	4708	4523	4576	4523
query23	35792	33932	33816	33816
query24	12022	2820	2833	2820
query25	564	401	401	401
query26	1180	159	168	159
query27	2757	285	292	285
query28	7873	2505	2454	2454
query29	696	430	430	430
query30	259	157	157	157
query31	1043	789	790	789
query32	97	53	60	53
query33	765	276	289	276
query34	979	501	491	491
query35	916	718	722	718
query36	1099	937	953	937
query37	217	72	72	72
query38	3978	3884	3852	3852
query39	1472	1419	1443	1419
query40	253	100	98	98
query41	51	48	48	48
query42	115	99	101	99
query43	508	480	466	466
query44	1263	781	774	774
query45	182	166	168	166
query46	1125	730	728	728
query47	1905	1832	1818	1818
query48	473	364	369	364
query49	1050	386	397	386
query50	810	404	402	402
query51	7088	6920	6851	6851
query52	102	91	87	87
query53	283	184	181	181
query54	1159	438	433	433
query55	74	78	74	74
query56	248	227	230	227
query57	1239	1112	1085	1085
query58	220	218	200	200
query59	3062	2850	2999	2850
query60	279	236	244	236
query61	116	121	99	99
query62	857	653	648	648
query63	204	181	185	181
query64	4767	621	589	589
query65	3232	3198	3192	3192
query66	1384	315	303	303
query67	15661	15323	15179	15179
query68	5481	561	536	536
query69	426	244	258	244
query70	1155	1144	1133	1133
query71	459	265	256	256
query72	5823	3887	3907	3887
query73	756	337	331	331
query74	10377	8909	8895	8895
query75	3397	2620	2620	2620
query76	3380	907	988	907
query77	454	270	265	265
query78	10640	9892	9460	9460
query79	7388	600	592	592
query80	2182	420	411	411
query81	570	240	235	235
query82	1302	115	110	110
query83	263	138	134	134
query84	288	74	74	74
query85	2368	301	275	275
query86	500	294	300	294
query87	4408	4419	4198	4198
query88	5858	2362	2416	2362
query89	514	290	292	290
query90	2119	183	181	181
query91	182	138	139	138
query92	69	47	46	46
query93	7324	524	526	524
query94	1077	266	267	266
query95	357	247	260	247
query96	628	283	288	283
query97	3321	3139	3157	3139
query98	220	189	194	189
query99	1750	1282	1292	1282
Total cold run time: 319463 ms
Total hot run time: 188798 ms

@feiniaofeiafei
Copy link
Contributor Author

run p0

@feiniaofeiafei
Copy link
Contributor Author

run cloud_p0

@feiniaofeiafei feiniaofeiafei force-pushed the pick_extend_infer_predicates branch 2 times, most recently from e2bb10d to b6eb963 Compare October 28, 2024 00:06
@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40555 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b6eb96382ca7e0cbcac17b843789592108fd9697, data reload: false

------ Round 1 ----------------------------------
q1	17564	7384	7226	7226
q2	2005	282	272	272
q3	12341	1064	1148	1064
q4	10563	729	739	729
q5	7732	2877	2801	2801
q6	234	148	144	144
q7	980	604	598	598
q8	9574	1931	1929	1929
q9	8121	6365	6325	6325
q10	7002	2249	2280	2249
q11	440	245	247	245
q12	405	207	209	207
q13	17782	3061	3113	3061
q14	251	209	222	209
q15	564	505	512	505
q16	703	595	601	595
q17	995	562	584	562
q18	7241	6563	6545	6545
q19	4233	1075	1065	1065
q20	478	193	193	193
q21	3914	3337	3067	3067
q22	1047	1015	964	964
Total cold run time: 114169 ms
Total hot run time: 40555 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7359	7249	7343	7249
q2	353	229	234	229
q3	3034	2866	2922	2866
q4	2045	1758	1779	1758
q5	5618	5708	5760	5708
q6	223	146	146	146
q7	2148	1748	1759	1748
q8	3228	3415	3455	3415
q9	8845	8864	8812	8812
q10	3507	3509	3490	3490
q11	573	481	483	481
q12	816	592	608	592
q13	16686	3117	3161	3117
q14	305	273	280	273
q15	571	517	515	515
q16	698	661	643	643
q17	1821	1608	1620	1608
q18	7986	7707	7609	7609
q19	2311	1548	1549	1548
q20	2075	1869	1840	1840
q21	5353	5248	5243	5243
q22	1081	1006	1056	1006
Total cold run time: 76636 ms
Total hot run time: 59896 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188649 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b6eb96382ca7e0cbcac17b843789592108fd9697, data reload: false

query1	846	393	402	393
query2	6247	2136	2049	2049
query3	8698	205	216	205
query4	33868	23427	23308	23308
query5	3856	439	447	439
query6	262	161	160	160
query7	4191	301	308	301
query8	295	234	240	234
query9	9561	2661	2674	2661
query10	475	251	266	251
query11	17800	15195	14962	14962
query12	146	99	105	99
query13	1535	414	404	404
query14	9434	6545	7054	6545
query15	220	173	170	170
query16	7711	481	453	453
query17	1479	556	565	556
query18	2017	346	311	311
query19	218	159	164	159
query20	122	116	118	116
query21	209	117	106	106
query22	4658	4521	4527	4521
query23	34803	33972	33664	33664
query24	11198	2865	2777	2777
query25	608	399	416	399
query26	731	159	165	159
query27	2102	299	308	299
query28	6147	2521	2461	2461
query29	816	428	433	428
query30	253	152	151	151
query31	1036	793	779	779
query32	99	54	53	53
query33	666	275	285	275
query34	899	515	508	508
query35	832	730	716	716
query36	1083	970	923	923
query37	114	69	69	69
query38	3998	3828	3833	3828
query39	1486	1435	1441	1435
query40	204	98	100	98
query41	51	50	48	48
query42	113	102	100	100
query43	519	483	505	483
query44	1144	766	770	766
query45	183	166	167	166
query46	1151	711	678	678
query47	1906	1852	1797	1797
query48	447	370	371	370
query49	903	402	389	389
query50	800	400	401	400
query51	7119	6917	6850	6850
query52	103	89	94	89
query53	259	183	182	182
query54	1196	450	435	435
query55	77	80	77	77
query56	265	259	249	249
query57	1190	1130	1087	1087
query58	237	224	214	214
query59	3220	2928	3054	2928
query60	274	244	251	244
query61	101	102	105	102
query62	851	655	669	655
query63	211	186	181	181
query64	3607	619	599	599
query65	3212	3200	3185	3185
query66	846	309	298	298
query67	15793	15424	15271	15271
query68	4388	536	551	536
query69	418	246	249	246
query70	1181	1083	1116	1083
query71	368	262	276	262
query72	6609	3893	3637	3637
query73	756	340	343	340
query74	10153	8910	8754	8754
query75	3355	2593	2593	2593
query76	2303	877	947	877
query77	399	257	271	257
query78	10639	9604	9569	9569
query79	8733	590	592	590
query80	2500	406	423	406
query81	582	243	242	242
query82	1686	106	122	106
query83	275	140	137	137
query84	292	81	77	77
query85	2164	308	273	273
query86	477	301	299	299
query87	4353	4173	4250	4173
query88	5577	2358	2393	2358
query89	558	296	287	287
query90	2003	186	180	180
query91	174	140	163	140
query92	66	49	47	47
query93	7624	531	530	530
query94	887	264	284	264
query95	348	245	246	245
query96	649	278	285	278
query97	3308	3149	3140	3140
query98	223	195	205	195
query99	1620	1301	1260	1260
Total cold run time: 311369 ms
Total hot run time: 188649 ms

@feiniaofeiafei
Copy link
Contributor Author

run p0

@feiniaofeiafei
Copy link
Contributor Author

run cloud_p0

@feiniaofeiafei feiniaofeiafei force-pushed the pick_extend_infer_predicates branch from b6eb963 to f1bca63 Compare October 28, 2024 03:18
feiniaofeiafei and others added 2 commits October 28, 2024 11:19
This pr refactors the PredicatePropagation module and adds support for
predicate deduction, including:

Support for predicate deduction of like, not in, !=;
Support for predicate deduction of abs(b)=1 for a=b and abs(a)=1;
Support for transitive deduction of non-equivalent relations, for example, a>b b>1 leads to a>1.
Deleted useless predicates.
But still has something to do in predicate inference:
support expr in infer predicate, e.g. abs(t1.c1)>abs(t2.c2) and abs(t1.c1)<1
need to add expr qualifier info, to determine whether abs(t1.c1) and abs(t2.c2) is from same table.
@feiniaofeiafei feiniaofeiafei force-pushed the pick_extend_infer_predicates branch from f1bca63 to 780edef Compare October 28, 2024 03:19
@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40306 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 780edef9856576d4ac59a0261c2d3dd8257171e8, data reload: false

------ Round 1 ----------------------------------
q1	17559	7354	7242	7242
q2	2035	279	262	262
q3	12277	1037	1212	1037
q4	10560	737	779	737
q5	7724	2821	2773	2773
q6	233	148	141	141
q7	974	607	603	603
q8	9585	1927	2049	1927
q9	8126	6423	6386	6386
q10	6981	2282	2313	2282
q11	434	245	248	245
q12	395	214	208	208
q13	17773	2962	2952	2952
q14	233	207	221	207
q15	566	528	508	508
q16	679	608	606	606
q17	952	504	563	504
q18	7106	6479	6473	6473
q19	1361	1041	954	954
q20	455	193	187	187
q21	3817	3256	3120	3120
q22	1078	996	952	952
Total cold run time: 110903 ms
Total hot run time: 40306 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7290	7182	7187	7182
q2	319	224	229	224
q3	2962	2862	2877	2862
q4	2032	1737	1745	1737
q5	5647	5665	5663	5663
q6	245	140	141	140
q7	2210	1784	1732	1732
q8	3330	3449	3328	3328
q9	8805	8759	8775	8759
q10	3504	3504	3483	3483
q11	583	479	468	468
q12	822	613	561	561
q13	16633	3155	3165	3155
q14	288	264	267	264
q15	562	534	509	509
q16	702	664	678	664
q17	1792	1641	1562	1562
q18	8045	7845	7479	7479
q19	1694	1640	1549	1549
q20	2061	1818	1874	1818
q21	5222	5208	5258	5208
q22	1058	1050	1026	1026
Total cold run time: 75806 ms
Total hot run time: 59373 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189291 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 780edef9856576d4ac59a0261c2d3dd8257171e8, data reload: false

query1	848	394	397	394
query2	6249	2113	2114	2113
query3	8687	210	211	210
query4	33704	23420	23368	23368
query5	3728	426	413	413
query6	272	166	157	157
query7	4188	299	297	297
query8	301	245	215	215
query9	9480	2662	2632	2632
query10	471	244	252	244
query11	17985	15318	15126	15126
query12	149	101	101	101
query13	1524	413	395	395
query14	8847	7236	7130	7130
query15	229	170	173	170
query16	7332	478	478	478
query17	1558	590	587	587
query18	1519	340	322	322
query19	211	157	153	153
query20	129	116	113	113
query21	200	117	103	103
query22	4782	4490	4378	4378
query23	34590	33379	33741	33379
query24	11422	2837	2833	2833
query25	604	398	392	392
query26	953	163	163	163
query27	2474	286	290	286
query28	7541	2498	2455	2455
query29	754	439	424	424
query30	253	163	151	151
query31	1053	803	780	780
query32	94	56	56	56
query33	767	275	286	275
query34	920	487	493	487
query35	881	744	735	735
query36	1110	952	935	935
query37	122	69	69	69
query38	3974	3821	3913	3821
query39	1487	1459	1406	1406
query40	203	100	98	98
query41	53	50	48	48
query42	109	104	98	98
query43	534	484	485	484
query44	1187	780	777	777
query45	180	166	168	166
query46	1138	718	709	709
query47	1920	1850	1814	1814
query48	455	370	362	362
query49	925	393	393	393
query50	801	406	412	406
query51	7031	6913	6913	6913
query52	102	90	94	90
query53	250	187	189	187
query54	1144	447	440	440
query55	78	79	80	79
query56	256	253	237	237
query57	1219	1087	1103	1087
query58	227	202	204	202
query59	3107	2806	3018	2806
query60	313	247	248	247
query61	98	97	115	97
query62	845	664	663	663
query63	203	190	186	186
query64	4047	614	581	581
query65	3265	3164	3151	3151
query66	882	303	309	303
query67	15987	15270	15275	15270
query68	4442	549	548	548
query69	422	251	251	251
query70	1140	1135	1105	1105
query71	428	256	259	256
query72	6101	3957	3834	3834
query73	751	340	334	334
query74	10131	8843	8839	8839
query75	3420	2612	2606	2606
query76	2865	923	1054	923
query77	391	267	267	267
query78	10735	9763	9476	9476
query79	9132	592	586	586
query80	2460	411	420	411
query81	601	235	241	235
query82	1519	113	116	113
query83	254	137	137	137
query84	291	83	92	83
query85	1703	287	288	287
query86	478	305	299	299
query87	4413	4260	4154	4154
query88	5858	2383	2348	2348
query89	553	286	291	286
query90	2108	186	182	182
query91	170	140	140	140
query92	68	49	47	47
query93	8340	523	527	523
query94	943	290	273	273
query95	351	243	242	242
query96	633	286	292	286
query97	3331	3111	3134	3111
query98	215	194	192	192
query99	1616	1277	1306	1277
Total cold run time: 313814 ms
Total hot run time: 189291 ms

@feiniaofeiafei
Copy link
Contributor Author

run p0

1 similar comment
@feiniaofeiafei
Copy link
Contributor Author

run p0

@morrySnow morrySnow merged commit 5174c9a into apache:branch-3.0 Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants