Skip to content

[opt](scanner) Control the degree of parallelism of scanner when only limit involved #39927 #42079

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 3 commits into from
Nov 11, 2024
Merged

[opt](scanner) Control the degree of parallelism of scanner when only limit involved #39927 #42079

dataroaring merged 3 commits into from
Nov 11, 2024

Conversation

@zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Oct 18, 2024

cherry pick from #39927

@zhiqiang-hhhh
Copy link
Contributor Author

zhiqiang-hhhh commented Oct 18, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 18, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

doris-robot commented Oct 18, 2024

TPC-H: Total hot run time: 39605 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45008b3e148f0db43b8aa1150d556dad29f06188, data reload: false

------ Round 1 ----------------------------------
q1	17627	7416	7274	7274
q2	2057	155	147	147
q3	10737	1053	1141	1053
q4	10549	706	754	706
q5	7745	2721	2728	2721
q6	249	147	144	144
q7	988	630	627	627
q8	9484	1864	1914	1864
q9	6546	6374	6390	6374
q10	7013	2245	2278	2245
q11	445	247	248	247
q12	410	213	205	205
q13	17766	2957	2944	2944
q14	230	206	211	206
q15	550	509	521	509
q16	492	408	407	407
q17	959	607	502	502
q18	7126	6405	6486	6405
q19	3045	976	868	868
q20	547	277	266	266
q21	3833	3164	2914	2914
q22	1081	977	978	977
Total cold run time: 109479 ms
Total hot run time: 39605 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7320	7221	7203	7203
q2	319	224	234	224
q3	2847	2805	2819	2805
q4	2038	1792	1852	1792
q5	5628	5642	5654	5642
q6	224	143	143	143
q7	2189	1766	1776	1766
q8	3314	3529	3465	3465
q9	8762	8806	8825	8806
q10	3517	3535	3481	3481
q11	593	475	479	475
q12	808	582	599	582
q13	16509	3118	3113	3113
q14	304	284	265	265
q15	556	522	534	522
q16	534	467	467	467
q17	1806	1638	1574	1574
q18	8150	7692	7612	7612
q19	3988	1458	1315	1315
q20	2123	1849	1824	1824
q21	5344	5304	5229	5229
q22	1150	1016	984	984
Total cold run time: 78023 ms
Total hot run time: 59289 ms

@doris-robot
Copy link

doris-robot commented Oct 18, 2024

TPC-DS: Total hot run time: 192974 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45008b3e148f0db43b8aa1150d556dad29f06188, data reload: false

query1	1256	897	894	894
query2	6290	1953	1912	1912
query3	10781	3741	3539	3539
query4	67451	26804	23224	23224
query5	5980	465	456	456
query6	497	173	172	172
query7	6476	305	297	297
query8	310	227	212	212
query9	9644	2659	2646	2646
query10	543	276	281	276
query11	18567	15228	15708	15228
query12	162	98	101	98
query13	1655	483	451	451
query14	10469	7386	7271	7271
query15	220	173	178	173
query16	7345	491	486	486
query17	1063	559	552	552
query18	1914	298	308	298
query19	190	155	148	148
query20	117	112	105	105
query21	208	103	101	101
query22	4754	4478	4284	4284
query23	34215	33851	33945	33851
query24	5559	2835	2800	2800
query25	518	423	396	396
query26	690	162	169	162
query27	1699	296	296	296
query28	3918	2208	2191	2191
query29	661	431	429	429
query30	230	152	157	152
query31	985	782	809	782
query32	71	52	55	52
query33	471	281	282	281
query34	883	501	490	490
query35	850	725	713	713
query36	1055	951	940	940
query37	135	75	77	75
query38	3915	3870	3821	3821
query39	1467	1417	1422	1417
query40	206	110	98	98
query41	47	43	44	43
query42	111	98	98	98
query43	520	478	454	454
query44	1104	764	765	764
query45	204	170	173	170
query46	1128	708	707	707
query47	1888	1839	1850	1839
query48	423	339	341	339
query49	713	402	384	384
query50	839	390	405	390
query51	7115	6954	6930	6930
query52	104	91	119	91
query53	264	180	175	175
query54	559	472	451	451
query55	82	74	74	74
query56	275	247	256	247
query57	1204	1114	1117	1114
query58	229	224	237	224
query59	2960	2804	2700	2700
query60	281	269	269	269
query61	104	113	101	101
query62	754	651	637	637
query63	208	179	177	177
query64	1864	617	606	606
query65	3254	3122	3134	3122
query66	750	311	299	299
query67	15738	15576	15365	15365
query68	3788	543	539	539
query69	609	288	283	283
query70	1168	1110	1127	1110
query71	473	276	264	264
query72	7824	3875	3866	3866
query73	774	338	334	334
query74	10352	8967	8900	8900
query75	4400	2661	2666	2661
query76	3048	900	936	900
query77	774	287	279	279
query78	9829	9245	9154	9154
query79	7605	577	587	577
query80	1108	433	462	433
query81	550	226	223	223
query82	881	126	126	126
query83	387	135	135	135
query84	290	83	76	76
query85	1501	289	281	281
query86	435	284	297	284
query87	4349	4129	4245	4129
query88	5612	2397	2392	2392
query89	490	288	285	285
query90	2117	180	182	180
query91	138	107	107	107
query92	59	47	47	47
query93	6353	523	546	523
query94	1038	272	271	271
query95	341	242	245	242
query96	637	274	285	274
query97	3307	3071	3110	3071
query98	211	201	226	201
query99	1589	1288	1298	1288
Total cold run time: 338179 ms
Total hot run time: 192974 ms

@yiguolei yiguolei force-pushed the pick_39927_to_upstream_branch-3.0 branch from 45008b3 to 8105f95 Compare October 19, 2024 08:56
@yiguolei
Copy link
Contributor

yiguolei commented Oct 19, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 19, 2024

TPC-H: Total hot run time: 40989 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8105f953ce7e5a1f30665799219bfe53b82cfbd2, data reload: false

------ Round 1 ----------------------------------
q1	17600	7498	7360	7360
q2	2043	156	175	156
q3	10753	1077	1111	1077
q4	10559	715	713	713
q5	7967	2874	2807	2807
q6	234	151	147	147
q7	975	648	627	627
q8	9345	1895	1916	1895
q9	6643	6478	6453	6453
q10	7276	2372	2306	2306
q11	445	254	257	254
q12	435	226	231	226
q13	17806	3005	2986	2986
q14	246	224	221	221
q15	557	530	527	527
q16	660	607	610	607
q17	969	564	606	564
q18	7277	6530	6645	6530
q19	4351	958	1043	958
q20	584	282	268	268
q21	4079	3316	3420	3316
q22	1097	995	991	991
Total cold run time: 111901 ms
Total hot run time: 40989 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7510	7351	7384	7351
q2	350	256	248	248
q3	3202	3055	3126	3055
q4	2113	1815	1803	1803
q5	5891	5968	5870	5870
q6	233	172	159	159
q7	2170	1760	1769	1760
q8	3313	3589	3502	3502
q9	8917	8983	8821	8821
q10	3550	3524	3480	3480
q11	598	513	478	478
q12	805	607	616	607
q13	16416	3137	3134	3134
q14	308	272	272	272
q15	578	520	528	520
q16	722	669	666	666
q17	1868	1635	1605	1605
q18	8201	7701	7672	7672
q19	5153	1437	1532	1437
q20	2152	1873	1832	1832
q21	5356	5261	5361	5261
q22	1149	1010	994	994
Total cold run time: 80555 ms
Total hot run time: 60527 ms

@doris-robot
Copy link

doris-robot commented Oct 19, 2024

TPC-DS: Total hot run time: 194394 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8105f953ce7e5a1f30665799219bfe53b82cfbd2, data reload: false

query1	1281	919	893	893
query2	6247	2184	2085	2085
query3	10753	3890	4003	3890
query4	69637	25155	23418	23418
query5	5402	455	451	451
query6	475	165	193	165
query7	6291	306	298	298
query8	327	204	218	204
query9	9576	2642	2637	2637
query10	503	280	274	274
query11	17836	15242	15809	15242
query12	163	99	102	99
query13	1628	456	461	456
query14	10853	7396	7501	7396
query15	229	178	182	178
query16	7179	483	492	483
query17	1075	562	562	562
query18	1238	303	304	303
query19	191	149	149	149
query20	118	107	104	104
query21	211	105	100	100
query22	4576	4260	4144	4144
query23	34021	33792	33911	33792
query24	5694	2911	2827	2827
query25	516	414	401	401
query26	670	164	163	163
query27	1689	298	297	297
query28	4158	2515	2472	2472
query29	668	433	439	433
query30	252	159	162	159
query31	997	799	806	799
query32	73	54	57	54
query33	434	303	289	289
query34	903	498	506	498
query35	865	727	721	721
query36	1069	905	926	905
query37	142	85	85	85
query38	4029	3913	3940	3913
query39	1478	1410	1425	1410
query40	211	103	100	100
query41	47	45	45	45
query42	118	96	93	93
query43	525	480	498	480
query44	1128	769	765	765
query45	196	165	167	165
query46	1139	721	733	721
query47	1896	1854	1801	1801
query48	434	341	335	335
query49	712	379	392	379
query50	816	418	402	402
query51	7052	6848	6861	6848
query52	108	88	100	88
query53	264	186	188	186
query54	587	472	459	459
query55	77	77	75	75
query56	288	252	265	252
query57	1203	1161	1135	1135
query58	224	241	233	233
query59	3216	3223	2919	2919
query60	294	265	272	265
query61	105	107	148	107
query62	771	646	664	646
query63	213	181	180	180
query64	1464	644	608	608
query65	3275	3166	3190	3166
query66	751	328	313	313
query67	15589	15337	15309	15309
query68	3881	534	547	534
query69	681	285	282	282
query70	1086	1126	1089	1089
query71	448	266	276	266
query72	7879	3810	3831	3810
query73	756	330	328	328
query74	10291	8927	9030	8927
query75	4180	2615	2626	2615
query76	3487	926	932	926
query77	762	283	282	282
query78	9976	9265	9075	9075
query79	2813	593	592	592
query80	2802	446	446	446
query81	587	240	250	240
query82	684	137	138	137
query83	320	139	141	139
query84	287	83	75	75
query85	1821	294	298	294
query86	455	291	311	291
query87	4395	4320	4293	4293
query88	4653	2351	2362	2351
query89	404	289	282	282
query90	2165	189	181	181
query91	136	104	106	104
query92	64	49	47	47
query93	4884	554	539	539
query94	1123	279	301	279
query95	346	254	256	254
query96	636	275	275	275
query97	3331	3162	3130	3130
query98	220	195	197	195
query99	1717	1300	1292	1292
Total cold run time: 333420 ms
Total hot run time: 194394 ms

@zhiqiang-hhhh zhiqiang-hhhh force-pushed the pick_39927_to_upstream_branch-3.0 branch from 8105f95 to dcbb094 Compare October 25, 2024 03:09
@zhiqiang-hhhh
Copy link
Contributor Author

zhiqiang-hhhh commented Oct 25, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 25, 2024

TPC-H: Total hot run time: 40537 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dcbb0944a1a9dc83b2f95bda5f02a45e15c8ce3c, data reload: false

------ Round 1 ----------------------------------
q1	18133	7456	7401	7401
q2	2681	179	156	156
q3	10972	1084	1124	1084
q4	10218	699	711	699
q5	7729	2792	2813	2792
q6	234	147	149	147
q7	1000	625	614	614
q8	9364	1877	1997	1877
q9	6653	6415	6399	6399
q10	6944	2267	2331	2267
q11	440	246	249	246
q12	395	219	218	218
q13	17773	2955	2967	2955
q14	244	206	214	206
q15	549	499	500	499
q16	656	601	594	594
q17	968	512	547	512
q18	7367	6678	6506	6506
q19	2728	1030	997	997
q20	490	195	195	195
q21	3997	3189	3228	3189
q22	1112	984	1009	984
Total cold run time: 110647 ms
Total hot run time: 40537 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7346	7280	7237	7237
q2	325	230	226	226
q3	2835	2684	2724	2684
q4	1911	1623	1660	1623
q5	5397	5418	5410	5410
q6	217	137	139	137
q7	2040	1649	1629	1629
q8	3250	3355	3392	3355
q9	8505	8487	8478	8478
q10	3437	3418	3386	3386
q11	569	467	474	467
q12	738	613	583	583
q13	16887	2985	2960	2960
q14	298	259	258	258
q15	554	510	509	509
q16	704	675	658	658
q17	1808	1573	1546	1546
q18	7548	7652	7362	7362
q19	1625	1535	1547	1535
q20	2000	1775	1786	1775
q21	5202	5104	5088	5088
q22	1067	987	955	955
Total cold run time: 74263 ms
Total hot run time: 57861 ms

@doris-robot
Copy link

doris-robot commented Oct 25, 2024

TPC-DS: Total hot run time: 189569 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dcbb0944a1a9dc83b2f95bda5f02a45e15c8ce3c, data reload: false

query1	984	362	372	362
query2	6531	2107	2038	2038
query3	6698	221	225	221
query4	33705	23242	23259	23242
query5	4377	459	456	456
query6	269	173	166	166
query7	4613	319	314	314
query8	280	228	225	225
query9	9762	2671	2671	2671
query10	457	265	286	265
query11	18039	15187	14957	14957
query12	149	110	100	100
query13	1627	425	405	405
query14	9365	7017	6727	6727
query15	229	172	174	172
query16	7201	482	460	460
query17	1427	571	587	571
query18	1838	305	318	305
query19	202	156	161	156
query20	118	106	107	106
query21	210	103	103	103
query22	4406	4284	4151	4151
query23	34360	33836	33863	33836
query24	12461	2837	2883	2837
query25	701	399	404	399
query26	1689	167	165	165
query27	2814	296	296	296
query28	7783	2501	2470	2470
query29	1062	441	434	434
query30	334	157	153	153
query31	1011	811	798	798
query32	102	59	58	58
query33	770	285	278	278
query34	1017	503	509	503
query35	866	716	733	716
query36	1118	950	958	950
query37	213	69	72	69
query38	3964	3847	3783	3783
query39	1498	1509	1465	1465
query40	294	99	104	99
query41	52	48	49	48
query42	104	101	101	101
query43	537	507	491	491
query44	1260	776	789	776
query45	187	165	165	165
query46	1141	734	728	728
query47	1903	1830	1847	1830
query48	455	368	365	365
query49	1272	397	387	387
query50	813	407	414	407
query51	7021	7098	6879	6879
query52	102	92	100	92
query53	261	190	189	189
query54	1197	445	441	441
query55	78	76	82	76
query56	265	257	248	248
query57	1222	1104	1100	1100
query58	245	243	219	219
query59	3219	2781	2889	2781
query60	296	278	268	268
query61	149	99	102	99
query62	862	662	661	661
query63	217	183	182	182
query64	5245	642	603	603
query65	3253	3198	3176	3176
query66	1432	307	301	301
query67	15779	15297	15398	15297
query68	4753	546	535	535
query69	421	258	260	258
query70	1197	1154	1165	1154
query71	426	272	262	262
query72	6341	3907	3904	3904
query73	763	347	341	341
query74	10328	8934	8893	8893
query75	3354	2638	2605	2605
query76	2926	1027	1116	1027
query77	416	286	272	272
query78	10610	9657	9550	9550
query79	6686	577	598	577
query80	1744	413	413	413
query81	569	250	247	247
query82	1177	112	111	111
query83	249	140	134	134
query84	289	78	78	78
query85	2331	283	274	274
query86	504	292	296	292
query87	4420	4273	4246	4246
query88	5513	2369	2373	2369
query89	427	289	289	289
query90	2053	177	182	177
query91	171	139	150	139
query92	66	49	46	46
query93	5524	528	532	528
query94	961	280	288	280
query95	341	244	243	243
query96	617	282	280	280
query97	3357	3155	3180	3155
query98	212	195	194	194
query99	1634	1304	1306	1304
Total cold run time: 313491 ms
Total hot run time: 189569 ms

@zhiqiang-hhhh
Copy link
Contributor Author

zhiqiang-hhhh commented Oct 25, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 25, 2024

TPC-H: Total hot run time: 40900 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7c1e27156d0584daf8f5985ca5186153643aeff1, data reload: false

------ Round 1 ----------------------------------
q1	17585	7476	7393	7393
q2	2066	154	148	148
q3	10581	1140	1167	1140
q4	10215	769	771	769
q5	7780	2843	2768	2768
q6	231	144	141	141
q7	951	610	592	592
q8	9509	1964	1946	1946
q9	7106	6446	6447	6446
q10	7073	2301	2285	2285
q11	446	243	247	243
q12	473	208	210	208
q13	17789	2993	2963	2963
q14	238	218	218	218
q15	562	522	508	508
q16	688	604	599	599
q17	997	580	543	543
q18	7320	6699	6669	6669
q19	1352	1056	1032	1032
q20	492	217	194	194
q21	3923	3245	3112	3112
q22	1104	992	983	983
Total cold run time: 108481 ms
Total hot run time: 40900 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7302	7267	7269	7267
q2	331	240	230	230
q3	2919	2741	2671	2671
q4	1894	1618	1612	1612
q5	5360	5424	5406	5406
q6	215	138	140	138
q7	2107	1660	1624	1624
q8	3226	3404	3420	3404
q9	8553	8467	8462	8462
q10	3473	3393	3403	3393
q11	563	475	481	475
q12	767	575	585	575
q13	16879	2977	2970	2970
q14	279	253	257	253
q15	560	499	501	499
q16	689	641	670	641
q17	1801	1584	1603	1584
q18	7504	7534	7384	7384
q19	1649	1571	1581	1571
q20	1975	1827	1782	1782
q21	5180	5025	4982	4982
q22	1075	1001	1003	1001
Total cold run time: 74301 ms
Total hot run time: 57924 ms

@doris-robot
Copy link

doris-robot commented Oct 25, 2024

TPC-DS: Total hot run time: 188673 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7c1e27156d0584daf8f5985ca5186153643aeff1, data reload: false

query1	969	373	371	371
query2	6476	2128	2014	2014
query3	6646	220	226	220
query4	33826	23744	23361	23361
query5	4277	437	425	425
query6	260	170	164	164
query7	4566	317	304	304
query8	284	221	229	221
query9	9637	2656	2639	2639
query10	468	261	282	261
query11	18148	15197	15184	15184
query12	154	99	103	99
query13	1610	419	393	393
query14	9593	6603	6576	6576
query15	215	172	171	171
query16	7633	499	488	488
query17	1624	574	546	546
query18	1773	310	304	304
query19	200	144	147	144
query20	113	105	106	105
query21	204	102	98	98
query22	4486	4419	4021	4021
query23	34654	33726	33984	33726
query24	12209	2819	2793	2793
query25	710	388	405	388
query26	1867	162	160	160
query27	2963	287	296	287
query28	8033	2473	2463	2463
query29	1093	451	436	436
query30	333	153	151	151
query31	1019	788	796	788
query32	93	56	58	56
query33	783	289	266	266
query34	1004	491	512	491
query35	856	750	722	722
query36	1103	939	920	920
query37	258	72	73	72
query38	3966	4000	3782	3782
query39	1512	1408	1443	1408
query40	285	99	99	99
query41	52	53	49	49
query42	108	100	98	98
query43	531	492	490	490
query44	1277	779	766	766
query45	183	162	168	162
query46	1139	705	715	705
query47	1908	1834	1805	1805
query48	450	365	362	362
query49	1229	371	382	371
query50	805	409	407	407
query51	6918	6844	6805	6805
query52	104	94	93	93
query53	262	188	187	187
query54	1213	439	446	439
query55	78	74	77	74
query56	262	250	235	235
query57	1193	1107	1104	1104
query58	240	214	235	214
query59	3212	2862	3100	2862
query60	293	255	249	249
query61	101	99	103	99
query62	848	674	658	658
query63	204	187	178	178
query64	5111	642	590	590
query65	3266	3155	3195	3155
query66	1355	302	308	302
query67	15986	15290	15232	15232
query68	5259	540	541	540
query69	420	249	258	249
query70	1159	1144	1133	1133
query71	426	270	250	250
query72	6251	3926	3910	3910
query73	765	346	333	333
query74	10413	8923	8924	8923
query75	3387	2598	2604	2598
query76	3307	954	886	886
query77	395	272	265	265
query78	10789	9701	9468	9468
query79	8937	592	582	582
query80	2218	434	410	410
query81	562	241	247	241
query82	1450	115	111	111
query83	253	138	134	134
query84	288	74	78	74
query85	2316	281	275	275
query86	507	293	288	288
query87	4465	4260	4216	4216
query88	5985	2372	2387	2372
query89	573	288	292	288
query90	2228	182	181	181
query91	184	167	138	138
query92	60	46	47	46
query93	7187	542	525	525
query94	1000	281	253	253
query95	342	241	242	241
query96	623	277	279	277
query97	3359	3171	3172	3171
query98	214	192	190	190
query99	1852	1301	1297	1297
Total cold run time: 321405 ms
Total hot run time: 188673 ms

@zhiqiang-hhhh
[opt](scanner) Control the degree of parallelism of scanner when only…
2ad0b3e 
… limit involved (apache#39927)

For queries like `select * from tbl limit 100;`, we should limit the
parallelism to 1.

We already have strategies in our code, but we can not control its
behavior. So add session variables to control them.

Previous pr like apache#33888 apache#36535 already has done something similar, this
pr integrates them.
@zhiqiang-hhhh zhiqiang-hhhh force-pushed the pick_39927_to_upstream_branch-3.0 branch from 7c1e271 to 2ad0b3e Compare November 8, 2024 12:37
@zhiqiang-hhhh
Copy link
Contributor Author

zhiqiang-hhhh commented Nov 8, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Nov 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Nov 8, 2024

TPC-H: Total hot run time: 40442 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ad0b3e175f6d339c4cfe726386f1710e98283bb, data reload: false

------ Round 1 ----------------------------------
q1	17583	7410	7261	7261
q2	2066	167	166	166
q3	10694	1044	1166	1044
q4	10544	717	693	693
q5	7756	2793	2802	2793
q6	234	147	141	141
q7	974	617	600	600
q8	9597	1939	2002	1939
q9	6826	6402	6398	6398
q10	6986	2270	2280	2270
q11	457	263	253	253
q12	414	208	201	201
q13	18047	2997	2965	2965
q14	230	210	216	210
q15	571	521	509	509
q16	653	609	606	606
q17	981	575	613	575
q18	7143	6533	6551	6533
q19	2458	1086	1042	1042
q20	476	195	191	191
q21	4255	3353	3092	3092
q22	1061	960	970	960
Total cold run time: 110006 ms
Total hot run time: 40442 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7329	7376	7179	7179
q2	389	315	303	303
q3	2920	2841	2811	2811
q4	2032	1806	1761	1761
q5	5697	5722	5722	5722
q6	230	142	138	138
q7	2169	1924	1766	1766
q8	3365	3532	3417	3417
q9	8854	8878	8867	8867
q10	3558	3484	3521	3484
q11	610	494	501	494
q12	835	630	590	590
q13	16420	3130	3131	3130
q14	292	278	285	278
q15	580	546	554	546
q16	725	676	687	676
q17	1864	1605	1585	1585
q18	8222	7840	7433	7433
q19	6128	1640	1496	1496
q20	2081	1838	1831	1831
q21	5427	5339	5302	5302
q22	1101	995	1007	995
Total cold run time: 80828 ms
Total hot run time: 59804 ms

@doris-robot
Copy link

doris-robot commented Nov 8, 2024

TPC-DS: Total hot run time: 192934 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ad0b3e175f6d339c4cfe726386f1710e98283bb, data reload: false

query1	2420	2259	2244	2244
query2	6255	2101	2045	2045
query3	13938	10115	229	229
query4	33147	23657	23627	23627
query5	3741	458	420	420
query6	285	187	179	179
query7	3972	310	308	308
query8	291	237	232	232
query9	9522	2685	2660	2660
query10	494	253	258	253
query11	17788	15098	15204	15098
query12	155	103	100	100
query13	1539	418	396	396
query14	9964	6957	7427	6957
query15	230	184	181	181
query16	7740	555	487	487
query17	1555	582	566	566
query18	2276	625	620	620
query19	262	193	191	191
query20	127	111	109	109
query21	214	102	101	101
query22	4822	4470	4008	4008
query23	34612	33855	33761	33761
query24	12018	3431	3330	3330
query25	611	403	407	403
query26	777	180	182	180
query27	2117	295	294	294
query28	6762	2489	2474	2474
query29	810	437	435	435
query30	412	302	314	302
query31	1039	785	814	785
query32	97	54	58	54
query33	778	288	275	275
query34	939	487	509	487
query35	912	716	705	705
query36	1090	958	931	931
query37	115	77	71	71
query38	4012	3825	3816	3816
query39	1474	1438	1426	1426
query40	206	98	99	98
query41	52	53	52	52
query42	112	100	98	98
query43	525	494	463	463
query44	1206	773	796	773
query45	181	164	165	164
query46	1142	702	723	702
query47	1884	1802	1838	1802
query48	459	366	368	366
query49	936	394	390	390
query50	804	404	411	404
query51	7176	7118	7085	7085
query52	101	92	95	92
query53	261	189	181	181
query54	1273	446	442	442
query55	73	73	74	73
query56	250	248	232	232
query57	1177	1089	1101	1089
query58	212	226	214	214
query59	3026	2919	2874	2874
query60	271	264	253	253
query61	113	110	116	110
query62	854	670	659	659
query63	214	192	185	185
query64	3846	638	640	638
query65	3259	3217	3192	3192
query66	864	298	308	298
query67	15752	15552	15245	15245
query68	4583	568	570	568
query69	432	256	249	249
query70	1144	1140	1128	1128
query71	402	258	260	258
query72	6282	3907	3915	3907
query73	770	346	342	342
query74	10070	8838	8940	8838
query75	3322	2610	2621	2610
query76	2225	1053	956	956
query77	429	275	287	275
query78	10569	9738	9523	9523
query79	9652	601	587	587
query80	2431	428	430	428
query81	583	249	247	247
query82	1676	114	112	112
query83	286	162	167	162
query84	290	80	77	77
query85	2104	302	286	286
query86	495	297	292	292
query87	4353	4456	4209	4209
query88	5867	2390	2389	2389
query89	554	285	290	285
query90	1996	180	179	179
query91	179	143	141	141
query92	67	46	46	46
query93	7528	523	527	523
query94	857	268	281	268
query95	343	241	250	241
query96	610	301	282	282
query97	3377	3130	3152	3130
query98	213	205	200	200
query99	1982	1282	1293	1282
Total cold run time: 321212 ms
Total hot run time: 192934 ms

@zhiqiang-hhhh
Copy link
Contributor Author

zhiqiang-hhhh commented Nov 8, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Nov 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Nov 8, 2024

TPC-H: Total hot run time: 40375 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47d11c8aaa5484d299f0a52eda8d44633521a840, data reload: false

------ Round 1 ----------------------------------
q1	17735	8022	7302	7302
q2	2070	169	172	169
q3	10695	1068	1179	1068
q4	10559	730	736	730
q5	7765	2853	2818	2818
q6	238	151	149	149
q7	973	623	617	617
q8	9382	1962	2014	1962
q9	6511	6366	6414	6366
q10	6983	2322	2293	2293
q11	452	257	257	257
q12	408	203	216	203
q13	17785	2990	3023	2990
q14	237	203	220	203
q15	561	523	506	506
q16	659	592	604	592
q17	969	569	627	569
q18	7306	6609	6400	6400
q19	2806	1066	998	998
q20	454	200	195	195
q21	3877	3093	3005	3005
q22	1075	1007	983	983
Total cold run time: 109500 ms
Total hot run time: 40375 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7470	7298	7313	7298
q2	415	311	313	311
q3	2966	2881	2891	2881
q4	2064	1826	1745	1745
q5	5661	5751	5730	5730
q6	230	142	141	141
q7	2227	1781	1728	1728
q8	3427	3531	3425	3425
q9	8884	8922	8842	8842
q10	3540	3514	3528	3514
q11	598	498	487	487
q12	819	629	596	596
q13	16356	3168	3102	3102
q14	315	293	274	274
q15	596	547	552	547
q16	723	674	662	662
q17	1836	1618	1592	1592
q18	8220	7754	7523	7523
q19	6170	1628	1589	1589
q20	2102	1874	1842	1842
q21	8467	5293	5272	5272
q22	1102	1047	1009	1009
Total cold run time: 84188 ms
Total hot run time: 60110 ms

@doris-robot
Copy link

doris-robot commented Nov 8, 2024

TPC-DS: Total hot run time: 192358 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 47d11c8aaa5484d299f0a52eda8d44633521a840, data reload: false

query1	2679	2355	2240	2240
query2	6234	2138	2042	2042
query3	13999	9916	233	233
query4	32995	23493	23458	23458
query5	3802	450	426	426
query6	269	181	185	181
query7	3979	317	307	307
query8	295	268	233	233
query9	9463	2693	2684	2684
query10	481	270	256	256
query11	17854	15103	15120	15103
query12	150	102	99	99
query13	1543	416	404	404
query14	9304	7173	6609	6609
query15	246	187	189	187
query16	8048	517	497	497
query17	1607	584	568	568
query18	2361	622	637	622
query19	269	190	184	184
query20	129	118	113	113
query21	209	105	113	105
query22	4670	4443	4018	4018
query23	34480	33851	33624	33624
query24	10800	3407	3469	3407
query25	581	403	409	403
query26	742	181	186	181
query27	2006	297	295	295
query28	6654	2507	2493	2493
query29	756	440	441	440
query30	384	311	312	311
query31	1023	792	806	792
query32	93	58	58	58
query33	689	286	285	285
query34	885	511	515	511
query35	829	746	722	722
query36	1097	939	940	939
query37	115	70	78	70
query38	3979	3806	3848	3806
query39	1457	1429	1412	1412
query40	200	100	100	100
query41	52	50	48	48
query42	106	99	97	97
query43	533	483	493	483
query44	1142	780	792	780
query45	183	166	167	166
query46	1124	716	706	706
query47	1901	1812	1797	1797
query48	460	377	389	377
query49	913	408	391	391
query50	801	408	413	408
query51	7146	7136	7054	7054
query52	100	95	90	90
query53	265	186	187	186
query54	1175	459	447	447
query55	79	74	77	74
query56	248	225	227	225
query57	1208	1123	1127	1123
query58	213	197	202	197
query59	3098	2905	3107	2905
query60	294	253	259	253
query61	112	107	106	106
query62	846	646	654	646
query63	219	197	188	188
query64	3670	636	612	612
query65	3268	3146	3135	3135
query66	855	291	297	291
query67	16060	15517	15292	15292
query68	4464	580	565	565
query69	422	250	257	250
query70	1165	1141	1112	1112
query71	362	260	256	256
query72	6668	3920	3858	3858
query73	766	339	341	339
query74	10099	8888	8894	8888
query75	3299	2624	2605	2605
query76	2227	925	914	914
query77	368	259	274	259
query78	10665	9603	9457	9457
query79	9409	589	576	576
query80	2365	429	425	425
query81	598	242	249	242
query82	1116	115	112	112
query83	290	153	152	152
query84	294	81	77	77
query85	1725	308	296	296
query86	476	297	294	294
query87	4404	4329	4187	4187
query88	5691	2436	2401	2401
query89	546	296	289	289
query90	1994	187	187	187
query91	185	147	145	145
query92	63	47	47	47
query93	7955	537	535	535
query94	889	302	281	281
query95	352	259	248	248
query96	644	280	283	280
query97	3364	3189	3102	3102
query98	215	198	199	198
query99	1581	1286	1317	1286
Total cold run time: 318088 ms
Total hot run time: 192358 ms

@zhiqiang-hhhh
Copy link
Contributor Author

zhiqiang-hhhh commented Nov 11, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Nov 11, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Nov 11, 2024

TPC-H: Total hot run time: 40747 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6053c92c9d90a87a395dd49a41fef1c6435eb720, data reload: false

------ Round 1 ----------------------------------
q1	9432	7410	7391	7391
q2	1758	166	161	161
q3	3069	1343	1391	1343
q4	4364	718	807	718
q5	3619	2799	2785	2785
q6	242	147	149	147
q7	939	604	617	604
q8	4026	2016	1977	1977
q9	6447	6424	6388	6388
q10	7013	2289	2295	2289
q11	461	259	254	254
q12	392	205	205	205
q13	17529	2968	2981	2968
q14	234	206	205	205
q15	571	504	513	504
q16	654	595	599	595
q17	949	589	625	589
q18	7333	6763	6447	6447
q19	1206	984	998	984
q20	464	208	192	192
q21	4716	3206	3017	3017
q22	1076	986	984	984
Total cold run time: 76494 ms
Total hot run time: 40747 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7360	7277	7293	7277
q2	425	313	297	297
q3	2897	2713	2707	2707
q4	1948	1649	1662	1649
q5	5401	5417	5402	5402
q6	210	134	134	134
q7	2063	1647	1748	1647
q8	3417	3644	3602	3602
q9	8937	8915	9047	8915
q10	3576	3510	3808	3510
q11	594	472	488	472
q12	730	565	576	565
q13	10704	3222	3149	3149
q14	296	280	268	268
q15	587	536	545	536
q16	688	681	661	661
q17	1858	1619	1601	1601
q18	8172	7811	7693	7693
q19	1690	1564	1474	1474
q20	2097	1935	1847	1847
q21	5512	5180	5179	5179
q22	1105	1028	1037	1028
Total cold run time: 70267 ms
Total hot run time: 59613 ms

@doris-robot
Copy link

doris-robot commented Nov 11, 2024

TPC-DS: Total hot run time: 193288 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6053c92c9d90a87a395dd49a41fef1c6435eb720, data reload: false

query1	2417	2374	2222	2222
query2	6426	2052	1975	1975
query3	14342	10749	235	235
query4	33769	23366	23339	23339
query5	5635	450	440	440
query6	297	189	190	189
query7	3975	318	318	318
query8	297	227	235	227
query9	9434	2703	2665	2665
query10	530	252	257	252
query11	18256	15157	15038	15038
query12	157	99	98	98
query13	1544	428	396	396
query14	9704	7287	7111	7111
query15	231	185	184	184
query16	7753	512	499	499
query17	1570	588	568	568
query18	2396	597	592	592
query19	241	183	184	183
query20	125	119	110	110
query21	208	108	101	101
query22	4738	4420	4489	4420
query23	34443	33910	33720	33720
query24	7246	3327	3302	3302
query25	563	401	405	401
query26	736	180	178	178
query27	1730	291	299	291
query28	4235	2509	2493	2493
query29	746	443	434	434
query30	368	305	298	298
query31	974	778	805	778
query32	80	56	57	56
query33	485	289	274	274
query34	911	498	495	495
query35	864	710	697	697
query36	1050	918	937	918
query37	112	73	73	73
query38	3883	3851	3867	3851
query39	1464	1411	1448	1411
query40	202	99	98	98
query41	53	50	49	49
query42	104	103	97	97
query43	526	490	509	490
query44	1122	783	796	783
query45	174	162	159	159
query46	1121	700	694	694
query47	1886	1783	1787	1783
query48	447	371	361	361
query49	713	372	366	366
query50	811	399	401	399
query51	7223	7222	6987	6987
query52	103	92	86	86
query53	249	180	177	177
query54	552	433	431	431
query55	74	72	75	72
query56	258	246	230	230
query57	1176	1081	1065	1065
query58	211	196	196	196
query59	3134	2954	3017	2954
query60	276	242	245	242
query61	109	104	122	104
query62	767	665	676	665
query63	206	184	182	182
query64	3092	636	598	598
query65	3296	3159	3181	3159
query66	721	308	301	301
query67	15774	15185	15311	15185
query68	4677	566	571	566
query69	436	253	257	253
query70	1166	1138	1138	1138
query71	377	255	255	255
query72	6409	4092	3871	3871
query73	752	340	340	340
query74	9721	8803	8858	8803
query75	3334	2615	2659	2615
query76	2472	947	956	947
query77	372	266	253	253
query78	10910	10260	11423	10260
query79	14739	617	581	581
query80	1903	426	409	409
query81	561	250	242	242
query82	1063	117	119	117
query83	319	153	150	150
query84	288	76	74	74
query85	1415	298	287	287
query86	495	292	306	292
query87	4498	4204	4244	4204
query88	5796	2359	2386	2359
query89	484	288	286	286
query90	2099	188	174	174
query91	173	144	140	140
query92	74	48	46	46
query93	6429	524	524	524
query94	852	272	294	272
query95	336	244	240	240
query96	625	279	284	279
query97	3256	3084	3117	3084
query98	211	194	194	194
query99	1711	1288	1290	1288
Total cold run time: 316268 ms
Total hot run time: 193288 ms

@dataroaring dataroaring merged commit 89ff353 into apache:branch-3.0 Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zhiqiang-hhhh @doris-robot @yiguolei @dataroaring