Skip to content

Conversation

@xy720
Copy link
Member

@xy720 xy720 commented Aug 21, 2025

What problem does this PR solve?

Problem Summary:

support cloud restore from doris version 2.1 to 3.1

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@xy720 xy720 requested a review from morrySnow as a code owner August 21, 2025 07:28
@Thearas
Copy link
Contributor

Thearas commented Aug 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xy720 xy720 changed the title [enhancement](cloud) Fix cloud restore from doris version 2.1 to 3.1 branch-3.1: [enhancement](cloud) Fix cloud restore from doris version 2.1 to 3.1 Aug 21, 2025
@xy720
Copy link
Member Author

xy720 commented Aug 21, 2025

run buildall

@xy720 xy720 requested a review from gavinchou August 21, 2025 07:33
@doris-robot
Copy link

TPC-H: Total hot run time: 32886 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c5c4bfb3c30b104bce8e125a70b22e3d96ffec3, data reload: false

------ Round 1 ----------------------------------
q1	17583	5538	5423	5423
q2	2023	393	276	276
q3	12048	1265	773	773
q4	10272	881	478	478
q5	8635	2421	2140	2140
q6	182	162	132	132
q7	904	774	618	618
q8	9330	1464	1223	1223
q9	5253	5003	4874	4874
q10	6773	2278	1847	1847
q11	465	282	265	265
q12	342	358	206	206
q13	17768	3628	3032	3032
q14	225	231	224	224
q15	534	481	461	461
q16	439	430	371	371
q17	629	893	379	379
q18	7031	6520	6370	6370
q19	1223	969	575	575
q20	340	354	205	205
q21	2992	2182	2028	2028
q22	1058	1023	986	986
Total cold run time: 106049 ms
Total hot run time: 32886 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5592	5574	5522	5522
q2	237	331	236	236
q3	2256	2622	2338	2338
q4	1397	1825	1386	1386
q5	4459	5031	4965	4965
q6	170	163	127	127
q7	2053	1981	1811	1811
q8	2652	2870	2727	2727
q9	7301	7228	7279	7228
q10	3046	3343	2764	2764
q11	575	498	486	486
q12	669	805	588	588
q13	3388	3822	3165	3165
q14	286	321	282	282
q15	523	476	455	455
q16	449	486	434	434
q17	1227	1753	1254	1254
q18	7757	7426	7422	7422
q19	823	1173	1089	1089
q20	2010	2035	1914	1914
q21	5432	4924	4587	4587
q22	1089	1051	1019	1019
Total cold run time: 53391 ms
Total hot run time: 51799 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193746 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c5c4bfb3c30b104bce8e125a70b22e3d96ffec3, data reload: false

query1	964	406	404	404
query2	6075	2030	1951	1951
query3	8684	203	193	193
query4	33683	23805	23447	23447
query5	3886	616	453	453
query6	282	191	175	175
query7	4198	505	333	333
query8	295	249	230	230
query9	9348	2611	2602	2602
query10	484	349	262	262
query11	18156	15333	15292	15292
query12	172	113	104	104
query13	1572	559	422	422
query14	8958	8013	7373	7373
query15	240	191	175	175
query16	8107	668	509	509
query17	1551	779	595	595
query18	2177	422	325	325
query19	224	202	193	193
query20	124	125	119	119
query21	213	132	110	110
query22	4594	4760	4408	4408
query23	35358	34115	34493	34115
query24	7339	2755	2762	2755
query25	522	514	426	426
query26	1186	296	178	178
query27	2038	503	365	365
query28	5000	2218	2175	2175
query29	826	597	489	489
query30	246	196	164	164
query31	1012	937	843	843
query32	96	62	63	62
query33	543	374	320	320
query34	766	854	529	529
query35	777	821	766	766
query36	1007	1074	985	985
query37	124	97	68	68
query38	4001	4003	4032	4003
query39	1521	1491	1476	1476
query40	226	127	103	103
query41	51	51	50	50
query42	130	112	113	112
query43	535	509	497	497
query44	1350	833	839	833
query45	183	178	173	173
query46	924	1062	694	694
query47	1947	1962	1906	1906
query48	405	429	369	369
query49	782	496	447	447
query50	686	696	430	430
query51	7351	7419	7241	7241
query52	112	111	94	94
query53	242	261	194	194
query54	578	565	476	476
query55	81	81	82	81
query56	279	290	274	274
query57	1286	1251	1219	1219
query58	238	236	226	226
query59	3210	3285	3127	3127
query60	290	306	281	281
query61	117	131	121	121
query62	815	783	730	730
query63	250	211	211	211
query64	4527	1002	658	658
query65	3449	3334	3354	3334
query66	975	423	313	313
query67	16503	15968	15577	15577
query68	7907	835	547	547
query69	494	310	276	276
query70	1256	1109	1085	1085
query71	379	307	283	283
query72	5757	3774	3763	3763
query73	636	738	361	361
query74	10026	9130	9366	9130
query75	3216	3164	2656	2656
query76	3020	1153	805	805
query77	530	374	288	288
query78	10343	10535	9731	9731
query79	3953	846	599	599
query80	789	541	450	450
query81	501	257	219	219
query82	575	121	90	90
query83	165	168	149	149
query84	241	104	81	81
query85	793	367	309	309
query86	391	319	289	289
query87	4403	4364	4272	4272
query88	5237	2435	2414	2414
query89	417	336	289	289
query90	1772	196	192	192
query91	144	147	121	121
query92	63	58	54	54
query93	2483	902	559	559
query94	679	426	318	318
query95	350	287	275	275
query96	494	621	292	292
query97	3189	3311	3160	3160
query98	224	207	203	203
query99	1509	1400	1321	1321
Total cold run time: 295807 ms
Total hot run time: 193746 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.25 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c5c4bfb3c30b104bce8e125a70b22e3d96ffec3, data reload: false

query1	0.03	0.03	0.04
query2	0.06	0.03	0.04
query3	0.24	0.07	0.07
query4	1.63	0.11	0.11
query5	0.52	0.52	0.53
query6	1.15	0.74	0.72
query7	0.03	0.02	0.01
query8	0.05	0.03	0.04
query9	0.59	0.52	0.50
query10	0.56	0.57	0.55
query11	0.15	0.12	0.11
query12	0.14	0.10	0.11
query13	0.61	0.60	0.60
query14	0.78	0.80	0.78
query15	0.85	0.81	0.83
query16	0.41	0.39	0.41
query17	1.09	1.04	0.99
query18	0.24	0.22	0.22
query19	1.88	1.79	1.88
query20	0.02	0.01	0.01
query21	15.37	0.94	0.57
query22	0.74	0.75	0.59
query23	15.24	1.39	0.51
query24	3.03	1.96	0.30
query25	0.18	0.07	0.08
query26	0.21	0.16	0.13
query27	0.06	0.07	0.05
query28	12.62	1.04	0.42
query29	12.64	4.09	3.47
query30	0.26	0.09	0.06
query31	2.83	0.58	0.39
query32	3.23	0.54	0.46
query33	3.05	3.01	3.04
query34	16.59	5.19	4.55
query35	4.54	4.55	4.51
query36	0.66	0.49	0.48
query37	0.08	0.06	0.07
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 102.75 s
Total hot run time: 28.25 s

@xy720
Copy link
Member Author

xy720 commented Aug 21, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32625 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e8d09fb9ec915527b3c83b3a7f4ce6830ec01aa0, data reload: false

------ Round 1 ----------------------------------
q1	17577	5467	5477	5467
q2	2028	398	297	297
q3	12112	1238	741	741
q4	10309	861	444	444
q5	9740	2355	2078	2078
q6	196	163	134	134
q7	884	747	610	610
q8	9323	1378	1124	1124
q9	5321	4936	4913	4913
q10	6773	2268	1811	1811
q11	462	284	258	258
q12	340	353	212	212
q13	17795	3576	3030	3030
q14	214	221	216	216
q15	542	466	493	466
q16	433	427	369	369
q17	612	864	367	367
q18	6694	6386	6386	6386
q19	1213	937	550	550
q20	331	350	211	211
q21	2893	2190	1953	1953
q22	1034	1007	988	988
Total cold run time: 106826 ms
Total hot run time: 32625 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5515	5509	5511	5509
q2	239	326	234	234
q3	2267	2615	2296	2296
q4	1457	1866	1425	1425
q5	4467	5058	5054	5054
q6	182	163	129	129
q7	2109	1985	1823	1823
q8	2651	2827	2722	2722
q9	7470	7438	7336	7336
q10	3067	3313	2757	2757
q11	567	506	492	492
q12	710	748	655	655
q13	3422	3769	3168	3168
q14	291	309	263	263
q15	509	468	464	464
q16	433	496	445	445
q17	1196	1722	1253	1253
q18	7745	7357	7321	7321
q19	779	1066	1112	1066
q20	2031	2047	1901	1901
q21	5313	5029	4408	4408
q22	1095	1047	1018	1018
Total cold run time: 53515 ms
Total hot run time: 51739 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192754 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e8d09fb9ec915527b3c83b3a7f4ce6830ec01aa0, data reload: false

query1	952	422	398	398
query2	6201	1924	1896	1896
query3	8689	197	196	196
query4	33620	23618	23569	23569
query5	3631	610	445	445
query6	277	192	188	188
query7	4211	493	308	308
query8	298	238	239	238
query9	9465	2603	2595	2595
query10	479	336	262	262
query11	18052	15601	15074	15074
query12	151	110	108	108
query13	1564	558	427	427
query14	9130	6683	7362	6683
query15	246	193	184	184
query16	7993	650	439	439
query17	1551	785	594	594
query18	2151	418	318	318
query19	238	191	163	163
query20	129	125	122	122
query21	208	129	113	113
query22	4740	4865	4422	4422
query23	35206	34654	34189	34189
query24	7274	2741	2725	2725
query25	553	516	437	437
query26	1155	286	174	174
query27	2701	476	371	371
query28	5526	2194	2147	2147
query29	765	608	462	462
query30	234	191	165	165
query31	1015	914	894	894
query32	87	57	54	54
query33	488	377	312	312
query34	742	844	523	523
query35	775	829	760	760
query36	1041	1096	965	965
query37	103	100	66	66
query38	4037	4081	4046	4046
query39	1538	1480	1465	1465
query40	214	113	102	102
query41	50	49	54	49
query42	132	104	103	103
query43	520	520	494	494
query44	1311	811	816	811
query45	184	178	172	172
query46	908	1075	673	673
query47	1959	2035	1926	1926
query48	421	459	349	349
query49	789	499	401	401
query50	691	721	432	432
query51	7390	7344	7286	7286
query52	108	107	95	95
query53	231	260	199	199
query54	557	535	480	480
query55	86	83	80	80
query56	269	269	259	259
query57	1292	1276	1225	1225
query58	233	231	229	229
query59	3027	3262	3149	3149
query60	297	291	283	283
query61	116	114	115	114
query62	797	761	722	722
query63	229	190	191	190
query64	4478	1005	660	660
query65	3382	3325	3317	3317
query66	931	412	313	313
query67	16113	15973	15627	15627
query68	7737	817	529	529
query69	492	307	268	268
query70	1200	1160	1094	1094
query71	388	302	266	266
query72	5171	3748	3785	3748
query73	644	750	358	358
query74	10268	9479	9267	9267
query75	3249	3106	2692	2692
query76	3218	1117	781	781
query77	587	370	270	270
query78	10310	10342	9632	9632
query79	3916	847	583	583
query80	610	531	436	436
query81	501	257	224	224
query82	594	120	89	89
query83	180	160	144	144
query84	285	102	86	86
query85	802	358	300	300
query86	384	323	297	297
query87	4308	4276	4437	4276
query88	5201	2406	2416	2406
query89	405	333	301	301
query90	1734	190	186	186
query91	136	146	111	111
query92	61	57	54	54
query93	2165	891	534	534
query94	677	406	305	305
query95	347	282	271	271
query96	491	615	296	296
query97	3245	3294	3214	3214
query98	225	209	211	209
query99	1563	1407	1290	1290
Total cold run time: 295025 ms
Total hot run time: 192754 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.38 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e8d09fb9ec915527b3c83b3a7f4ce6830ec01aa0, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.25	0.07	0.07
query4	1.62	0.11	0.10
query5	0.52	0.50	0.52
query6	1.12	0.74	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.56	0.51	0.50
query10	0.55	0.55	0.55
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.60	0.60	0.60
query14	0.77	0.78	0.79
query15	0.85	0.83	0.82
query16	0.40	0.38	0.39
query17	1.05	1.01	1.01
query18	0.25	0.23	0.23
query19	1.81	1.81	1.88
query20	0.01	0.02	0.01
query21	15.39	0.91	0.58
query22	0.74	0.71	0.68
query23	15.17	1.39	0.63
query24	3.03	1.30	1.44
query25	0.18	0.16	0.08
query26	0.32	0.15	0.14
query27	0.05	0.05	0.04
query28	14.09	0.96	0.43
query29	12.60	3.88	3.24
query30	0.26	0.09	0.07
query31	2.80	0.57	0.38
query32	3.23	0.55	0.46
query33	2.99	3.02	3.01
query34	16.70	5.18	4.55
query35	4.56	4.60	4.63
query36	0.64	0.48	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.03 s
Total hot run time: 29.38 s

@morrySnow morrySnow merged commit f72f80d into apache:branch-3.1 Sep 4, 2025
22 checks passed
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants