Skip to content

Conversation

@CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Jun 30, 2025

bp #50849

@CalvinKirs CalvinKirs requested a review from morrySnow as a code owner June 30, 2025 03:16
@CalvinKirs
Copy link
Member Author

run buildall

…branch-3.1-50849

# Conflicts:
#	fe/fe-core/src/main/java/org/apache/doris/datasource/LoadScanProvider.java
#	fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java
@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39870 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9508ee5ee9136348c5ea3c7076ccc03bba8b3337, data reload: false

------ Round 1 ----------------------------------
q1	17639	6826	6618	6618
q2	2063	200	170	170
q3	10606	1154	1192	1154
q4	10262	714	717	714
q5	7739	2888	2879	2879
q6	211	132	134	132
q7	998	624	625	624
q8	9370	1940	2053	1940
q9	6602	6367	6402	6367
q10	7027	2236	2311	2236
q11	462	273	263	263
q12	412	223	216	216
q13	17796	3016	3071	3016
q14	252	223	209	209
q15	517	483	474	474
q16	483	396	378	378
q17	981	609	629	609
q18	7320	6601	6770	6601
q19	1328	1027	920	920
q20	497	216	206	206
q21	4240	3168	3165	3165
q22	1098	1011	979	979
Total cold run time: 107903 ms
Total hot run time: 39870 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6651	6575	6540	6540
q2	337	230	235	230
q3	2927	2955	2947	2947
q4	2064	1853	1841	1841
q5	5713	5709	5707	5707
q6	206	128	129	128
q7	2221	1795	1837	1795
q8	3360	3532	3567	3532
q9	8763	8874	8811	8811
q10	3590	3557	3545	3545
q11	610	490	495	490
q12	792	675	600	600
q13	8415	3200	3190	3190
q14	309	271	300	271
q15	518	473	461	461
q16	499	458	436	436
q17	1857	1617	1612	1612
q18	8248	7831	7817	7817
q19	1710	1526	1518	1518
q20	2069	1853	1838	1838
q21	5401	5226	5093	5093
q22	1142	1044	1061	1044
Total cold run time: 67402 ms
Total hot run time: 59446 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198977 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9508ee5ee9136348c5ea3c7076ccc03bba8b3337, data reload: false

query1	1292	961	905	905
query2	6369	2032	1973	1973
query3	10906	4576	4589	4576
query4	33020	23402	23528	23402
query5	3599	478	457	457
query6	275	187	185	185
query7	3988	328	337	328
query8	298	243	236	236
query9	9636	2700	2679	2679
query10	468	262	259	259
query11	17959	15237	15293	15237
query12	153	109	107	107
query13	1581	451	438	438
query14	9454	7336	7305	7305
query15	247	191	194	191
query16	8048	534	511	511
query17	1617	620	587	587
query18	2131	326	316	316
query19	257	160	157	157
query20	131	126	129	126
query21	217	111	110	110
query22	4831	4466	4509	4466
query23	35467	34298	34057	34057
query24	11164	3047	2983	2983
query25	631	405	417	405
query26	1350	183	175	175
query27	2832	355	363	355
query28	7737	2237	2273	2237
query29	883	462	458	458
query30	280	167	170	167
query31	1076	826	839	826
query32	97	60	58	58
query33	785	310	311	310
query34	953	526	512	512
query35	866	763	727	727
query36	1150	960	993	960
query37	141	73	76	73
query38	4122	3965	3968	3965
query39	1570	1503	1467	1467
query40	254	111	115	111
query41	54	58	54	54
query42	129	108	105	105
query43	559	512	516	512
query44	1254	866	846	846
query45	188	176	170	170
query46	1205	744	735	735
query47	2065	1942	1952	1942
query48	449	353	359	353
query49	991	425	429	425
query50	871	434	428	428
query51	7608	7299	7362	7299
query52	109	96	99	96
query53	268	195	187	187
query54	1353	521	507	507
query55	88	82	86	82
query56	305	275	273	273
query57	1356	1238	1194	1194
query58	236	220	228	220
query59	3574	3153	3096	3096
query60	294	274	264	264
query61	116	155	188	155
query62	864	696	723	696
query63	236	199	190	190
query64	4462	699	683	683
query65	3355	3305	3218	3218
query66	1351	297	311	297
query67	16095	15893	15805	15805
query68	4766	608	604	604
query69	458	281	271	271
query70	1250	1126	1173	1126
query71	343	257	258	257
query72	6301	4015	4176	4015
query73	766	361	404	361
query74	10566	9353	8981	8981
query75	3330	2647	2641	2641
query76	2690	974	1051	974
query77	378	281	297	281
query78	10605	9658	9611	9611
query79	1717	613	607	607
query80	1136	436	429	429
query81	542	219	228	219
query82	952	99	92	92
query83	237	150	147	147
query84	231	82	81	81
query85	1353	319	302	302
query86	460	297	308	297
query87	4484	4344	4267	4267
query88	3828	2475	2513	2475
query89	421	295	294	294
query90	1911	188	184	184
query91	147	114	114	114
query92	74	54	52	52
query93	2515	589	581	581
query94	842	310	287	287
query95	373	268	262	262
query96	632	293	290	290
query97	3362	3152	3160	3152
query98	225	203	204	203
query99	1486	1291	1297	1291
Total cold run time: 305651 ms
Total hot run time: 198977 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.87 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9508ee5ee9136348c5ea3c7076ccc03bba8b3337, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.11
query5	0.53	0.51	0.54
query6	1.13	0.74	0.73
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.55	0.48	0.52
query10	0.58	0.58	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.61
query14	0.80	0.78	0.85
query15	0.85	0.83	0.82
query16	0.38	0.38	0.39
query17	1.03	1.10	1.06
query18	0.24	0.22	0.24
query19	2.02	1.90	1.91
query20	0.02	0.02	0.01
query21	15.38	0.59	0.58
query22	2.42	2.61	2.50
query23	16.90	0.98	0.88
query24	3.34	1.70	1.95
query25	0.19	0.18	0.06
query26	0.55	0.14	0.14
query27	0.05	0.04	0.03
query28	8.67	0.49	0.47
query29	12.56	3.26	3.26
query30	0.25	0.06	0.06
query31	2.83	0.41	0.39
query32	3.23	0.46	0.46
query33	3.03	2.96	3.03
query34	17.06	4.55	4.51
query35	4.60	4.54	4.55
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.18	0.12	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.3 s
Total hot run time: 31.87 s

@CalvinKirs
Copy link
Member Author

run external

@morrySnow
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39776 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9ff1717f4ffb7eb9e7b41cd63f56b5f9407ff1b4, data reload: false

------ Round 1 ----------------------------------
q1	17576	6899	6662	6662
q2	2077	203	183	183
q3	10480	1145	1223	1145
q4	10463	765	822	765
q5	7723	2927	2803	2803
q6	216	138	140	138
q7	981	633	618	618
q8	9368	1979	1982	1979
q9	6695	6435	6370	6370
q10	6996	2211	2317	2211
q11	457	272	270	270
q12	402	203	208	203
q13	17794	2960	2974	2960
q14	232	219	200	200
q15	508	465	468	465
q16	460	375	379	375
q17	985	612	600	600
q18	7342	6573	6654	6573
q19	1308	1060	977	977
q20	478	195	195	195
q21	3858	3136	3214	3136
q22	1042	979	948	948
Total cold run time: 107441 ms
Total hot run time: 39776 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6602	6606	6578	6578
q2	322	237	231	231
q3	2917	2969	2942	2942
q4	2028	1820	1771	1771
q5	5680	5765	5711	5711
q6	209	125	130	125
q7	2202	1816	1807	1807
q8	3390	3513	3484	3484
q9	8792	8855	8835	8835
q10	3518	3501	3526	3501
q11	602	494	493	493
q12	809	583	619	583
q13	9379	3135	3159	3135
q14	301	268	279	268
q15	505	479	456	456
q16	510	434	458	434
q17	1851	1621	1621	1621
q18	8233	7877	7717	7717
q19	1708	1591	1555	1555
q20	2083	1845	1812	1812
q21	5357	4937	5133	4937
q22	1106	1027	1095	1027
Total cold run time: 68104 ms
Total hot run time: 59023 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196394 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9ff1717f4ffb7eb9e7b41cd63f56b5f9407ff1b4, data reload: false

query1	1277	903	895	895
query2	6408	1982	1904	1904
query3	10868	4229	4515	4229
query4	32923	23751	23559	23559
query5	4261	461	472	461
query6	281	185	192	185
query7	3999	312	313	312
query8	293	229	230	229
query9	9553	2590	2583	2583
query10	466	261	267	261
query11	17935	15136	15068	15068
query12	163	107	104	104
query13	1569	423	424	423
query14	8592	7433	7088	7088
query15	247	189	200	189
query16	8050	443	466	443
query17	1623	600	614	600
query18	2163	319	313	313
query19	251	166	173	166
query20	130	119	121	119
query21	215	113	122	113
query22	4709	4441	4326	4326
query23	35110	33938	34122	33938
query24	11015	2946	2953	2946
query25	616	413	425	413
query26	735	170	166	166
query27	2328	354	362	354
query28	6807	2203	2153	2153
query29	775	437	440	437
query30	261	162	160	160
query31	1043	816	841	816
query32	95	56	57	56
query33	727	313	306	306
query34	937	516	502	502
query35	867	730	750	730
query36	1126	950	944	944
query37	117	66	66	66
query38	4095	4027	3957	3957
query39	1525	1448	1508	1448
query40	217	108	108	108
query41	49	48	53	48
query42	121	106	114	106
query43	551	486	479	479
query44	1213	815	817	815
query45	186	171	166	166
query46	1151	741	719	719
query47	2024	1899	1892	1892
query48	450	346	341	341
query49	953	395	388	388
query50	839	430	427	427
query51	7461	7242	7156	7156
query52	111	94	91	91
query53	268	185	180	180
query54	1170	468	465	465
query55	84	75	79	75
query56	268	248	238	238
query57	1329	1181	1195	1181
query58	230	221	210	210
query59	3160	3037	3070	3037
query60	302	261	272	261
query61	113	108	143	108
query62	854	691	669	669
query63	214	187	194	187
query64	3876	674	658	658
query65	3350	3354	3283	3283
query66	782	295	301	295
query67	16400	15635	15427	15427
query68	4440	587	573	573
query69	435	261	267	261
query70	1138	1140	1119	1119
query71	346	257	255	255
query72	6424	4138	3914	3914
query73	754	346	359	346
query74	10454	9165	9166	9165
query75	3404	2616	2702	2616
query76	2712	1038	1030	1030
query77	396	267	277	267
query78	10573	9516	9613	9516
query79	2470	604	612	604
query80	1168	434	422	422
query81	549	230	220	220
query82	685	86	86	86
query83	230	141	145	141
query84	240	84	80	80
query85	1501	302	285	285
query86	476	303	290	290
query87	4389	4226	4215	4215
query88	4318	2380	2367	2367
query89	412	293	296	293
query90	1927	189	188	188
query91	141	110	115	110
query92	63	48	53	48
query93	2272	558	549	549
query94	834	301	302	301
query95	362	258	253	253
query96	610	279	282	279
query97	3292	3136	3156	3136
query98	222	198	204	198
query99	1583	1329	1295	1295
Total cold run time: 300701 ms
Total hot run time: 196394 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9ff1717f4ffb7eb9e7b41cd63f56b5f9407ff1b4, data reload: false

query1	0.03	0.03	0.02
query2	0.08	0.03	0.04
query3	0.24	0.06	0.06
query4	1.62	0.10	0.11
query5	0.52	0.50	0.51
query6	1.14	0.73	0.74
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.49
query10	0.57	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.78	0.79	0.80
query15	0.85	0.82	0.83
query16	0.37	0.37	0.40
query17	0.96	1.07	1.05
query18	0.23	0.22	0.21
query19	1.88	1.80	1.77
query20	0.01	0.01	0.01
query21	15.41	0.58	0.58
query22	2.16	2.24	2.56
query23	16.89	1.13	0.89
query24	3.43	1.39	1.62
query25	0.30	0.08	0.14
query26	0.52	0.14	0.14
query27	0.04	0.05	0.05
query28	9.31	0.53	0.46
query29	12.62	3.20	3.23
query30	0.25	0.06	0.07
query31	2.85	0.39	0.36
query32	3.26	0.45	0.45
query33	2.91	2.94	3.01
query34	16.92	4.50	4.57
query35	4.48	4.51	4.49
query36	0.66	0.47	0.47
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.3 s
Total hot run time: 30.97 s

@CalvinKirs
Copy link
Member Author

run buildall

@CalvinKirs
Copy link
Member Author

run external

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40293 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8f14d45fe7cb53fca9597f6bea5b8faf6d883697, data reload: false

------ Round 1 ----------------------------------
q1	17579	6875	6706	6706
q2	2073	216	175	175
q3	10475	1157	1183	1157
q4	10964	770	726	726
q5	8548	2992	2966	2966
q6	220	143	143	143
q7	1019	641	636	636
q8	9461	2027	2053	2027
q9	6872	6557	6486	6486
q10	8263	2346	2385	2346
q11	532	265	265	265
q12	408	216	214	214
q13	17798	2980	2993	2980
q14	237	228	202	202
q15	519	469	486	469
q16	492	379	374	374
q17	1003	573	544	544
q18	7388	6548	6808	6548
q19	1326	1012	957	957
q20	463	198	204	198
q21	3951	3309	3197	3197
q22	1114	991	977	977
Total cold run time: 110705 ms
Total hot run time: 40293 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6668	6678	6626	6626
q2	347	240	239	239
q3	3231	3043	2934	2934
q4	1986	1880	1826	1826
q5	5691	5676	5710	5676
q6	214	129	127	127
q7	2191	1775	1793	1775
q8	3417	3525	3543	3525
q9	8914	8902	8922	8902
q10	3588	3550	3536	3536
q11	613	496	496	496
q12	802	588	623	588
q13	6762	3204	3140	3140
q14	322	260	276	260
q15	512	465	457	457
q16	492	428	435	428
q17	1862	1630	1622	1622
q18	8180	7687	7729	7687
q19	1759	1667	1526	1526
q20	2139	1896	1902	1896
q21	5393	4982	4868	4868
q22	1081	986	983	983
Total cold run time: 66164 ms
Total hot run time: 59117 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190071 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8f14d45fe7cb53fca9597f6bea5b8faf6d883697, data reload: false

query1	971	387	374	374
query2	6513	1997	1972	1972
query3	6705	221	227	221
query4	34353	23479	23311	23311
query5	4349	469	487	469
query6	283	177	192	177
query7	4641	335	313	313
query8	288	234	227	227
query9	9419	2607	2605	2605
query10	500	298	287	287
query11	18228	15434	15582	15434
query12	153	113	101	101
query13	1650	424	419	419
query14	9749	6725	6684	6684
query15	250	176	183	176
query16	7999	464	508	464
query17	1621	596	568	568
query18	2126	310	316	310
query19	254	157	158	157
query20	119	108	110	108
query21	211	105	107	105
query22	4269	4060	4103	4060
query23	34632	33410	33571	33410
query24	12813	2904	2834	2834
query25	695	404	382	382
query26	1875	174	175	174
query27	3045	346	352	346
query28	7615	2112	2103	2103
query29	1057	441	420	420
query30	334	157	157	157
query31	1024	777	812	777
query32	93	60	61	60
query33	776	292	318	292
query34	1000	508	516	508
query35	884	728	718	718
query36	1121	940	922	922
query37	292	71	70	70
query38	4194	3869	3826	3826
query39	1460	1432	1420	1420
query40	287	99	100	99
query41	49	46	48	46
query42	131	104	101	101
query43	546	506	471	471
query44	1183	782	792	782
query45	183	167	168	167
query46	1129	709	708	708
query47	1883	1796	1812	1796
query48	429	343	346	343
query49	1301	398	386	386
query50	822	421	426	421
query51	7222	7155	7212	7155
query52	102	96	97	96
query53	258	182	181	181
query54	1166	475	474	474
query55	82	81	78	78
query56	271	256	253	253
query57	1292	1157	1165	1157
query58	235	212	213	212
query59	3227	2980	2850	2850
query60	280	250	260	250
query61	128	139	113	113
query62	866	681	702	681
query63	217	192	192	192
query64	5207	691	636	636
query65	3308	3193	3196	3193
query66	1417	317	331	317
query67	15803	15633	15418	15418
query68	4500	595	567	567
query69	414	272	264	264
query70	1205	1138	1149	1138
query71	351	265	263	263
query72	6334	4035	4083	4035
query73	751	347	360	347
query74	10389	9013	8982	8982
query75	3412	2640	2647	2640
query76	2743	1132	1238	1132
query77	394	273	285	273
query78	10465	9482	9622	9482
query79	1084	600	597	597
query80	640	438	431	431
query81	499	221	222	221
query82	184	89	87	87
query83	240	147	151	147
query84	240	79	81	79
query85	997	313	295	295
query86	304	301	285	285
query87	4461	4176	4191	4176
query88	3842	2467	2392	2392
query89	408	295	292	292
query90	1956	193	196	193
query91	141	111	112	111
query92	63	54	54	54
query93	1039	543	540	540
query94	867	297	312	297
query95	362	262	265	262
query96	609	285	293	285
query97	3308	3196	3169	3169
query98	216	207	197	197
query99	1488	1311	1301	1301
Total cold run time: 300491 ms
Total hot run time: 190071 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8f14d45fe7cb53fca9597f6bea5b8faf6d883697, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.63	0.10	0.11
query5	0.50	0.50	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.04	0.02
query9	0.56	0.49	0.49
query10	0.54	0.55	0.53
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.60
query14	0.78	0.78	0.79
query15	0.84	0.84	0.83
query16	0.38	0.38	0.40
query17	1.03	1.05	1.03
query18	0.24	0.22	0.22
query19	1.82	1.87	1.84
query20	0.01	0.01	0.02
query21	15.42	0.58	0.58
query22	2.62	1.53	1.74
query23	16.93	0.92	0.81
query24	2.98	1.01	0.96
query25	0.19	0.13	0.15
query26	0.35	0.14	0.13
query27	0.06	0.05	0.05
query28	10.78	0.49	0.48
query29	12.65	3.21	3.20
query30	0.25	0.07	0.06
query31	2.85	0.40	0.39
query32	3.23	0.46	0.46
query33	2.96	3.01	3.02
query34	17.14	4.47	4.48
query35	4.50	4.55	4.50
query36	0.65	0.47	0.47
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 104.76 s
Total hot run time: 29.94 s

@morrySnow morrySnow changed the title branch-3.1: [feat](refactor-param)Integrate New Storage System Suppor… …t for BACKUP/RESTORE/LOAD/TVF (#50849) branch-3.1: [feat](refactor-param)Integrate New Storage System Support for BACKUP/RESTORE/LOAD/TVF #50849 Jul 8, 2025
@morrySnow morrySnow changed the title branch-3.1: [feat](refactor-param)Integrate New Storage System Support for BACKUP/RESTORE/LOAD/TVF #50849 branch-3.1: [feat](catalog-param)Integrate New Storage System Support for BACKUP/RESTORE/LOAD/TVF #50849 Jul 8, 2025
@morrySnow morrySnow merged commit 8d07481 into apache:branch-3.1 Jul 8, 2025
23 of 24 checks passed
morrySnow pushed a commit that referenced this pull request Sep 19, 2025
### What problem does this PR solve?

PR #51219 fixed the issue of S3 load HTTPS access failure, but the
changes were accidentally dropped during the cherry-pick in PR #52479.
Now we need to add those changes back.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants