Skip to content

branch-3.1: [refactor](cloud) separate cloud restore create tablet rpc into multiple batches #54933 #55691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Sep 5, 2025
Merged

branch-3.1: [refactor](cloud) separate cloud restore create tablet rpc into multiple batches #54933 #55691

morrySnow merged 2 commits into from
Sep 5, 2025

Conversation

@xy720
Copy link
Member

@xy720 xy720 commented Sep 4, 2025 •
edited by morrySnow
Loading

pick #54933

@xy720
[refactore](cloud) separate cloud restore create tablet rpc into mult…
2ab804b 
…iple batches (apache#54933)

1、Add a config cloud_restore_create_tablet_batch_size default 256,
separate cloud restore create tablet rpc into multiple batches to
prevent rpc timeout
2、Fix replica's partition id incorrect in restored table.
@xy720 xy720 requested a review from morrySnow as a code owner September 4, 2025 16:28
@xy720
Copy link
Member Author

xy720 commented Sep 4, 2025

run buildall

@Thearas
Copy link
Contributor

Thearas commented Sep 4, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

doris-robot commented Sep 4, 2025

Cloud UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.02% (1218/1485)
Line Coverage 65.84% (21785/33090)
Region Coverage 67.37% (10964/16275)
Branch Coverage 56.90% (5782/10162)

@xy720
Copy link
Member Author

xy720 commented Sep 5, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 5, 2025

Cloud UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 82.02% (1218/1485)
Line Coverage 65.85% (21789/33090)
Region Coverage 67.38% (10966/16275)
Branch Coverage 56.94% (5786/10162)

@doris-robot
Copy link

doris-robot commented Sep 5, 2025

TPC-H: Total hot run time: 32886 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 81566cc1aa1b2ee17fae1859b5a53a500673a073, data reload: false

------ Round 1 ----------------------------------
q1	17604	5549	5492	5492
q2	2030	383	276	276
q3	12197	1243	777	777
q4	10314	887	456	456
q5	9202	2381	2149	2149
q6	187	166	134	134
q7	904	749	619	619
q8	9333	1421	1205	1205
q9	5280	4967	4957	4957
q10	6789	2265	1840	1840
q11	479	278	255	255
q12	344	360	212	212
q13	17809	3588	3029	3029
q14	221	233	202	202
q15	524	460	469	460
q16	413	426	367	367
q17	596	858	355	355
q18	6894	6514	6353	6353
q19	1364	954	582	582
q20	332	337	207	207
q21	2785	2195	1993	1993
q22	1061	1016	966	966
Total cold run time: 106662 ms
Total hot run time: 32886 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5609	5573	5571	5571
q2	233	322	231	231
q3	2275	2595	2338	2338
q4	1410	1797	1399	1399
q5	4416	5019	4983	4983
q6	172	162	129	129
q7	2068	1961	1785	1785
q8	2662	2852	2694	2694
q9	7260	7196	7154	7154
q10	3029	3294	2705	2705
q11	577	540	484	484
q12	664	753	592	592
q13	3452	3811	3184	3184
q14	268	283	275	275
q15	510	479	465	465
q16	434	465	419	419
q17	1248	1726	1276	1276
q18	7616	7462	7466	7462
q19	829	1053	1167	1053
q20	2019	2054	1901	1901
q21	5348	4969	4557	4557
q22	1068	1076	1029	1029
Total cold run time: 53167 ms
Total hot run time: 51686 ms

@doris-robot
Copy link

doris-robot commented Sep 5, 2025

TPC-DS: Total hot run time: 190725 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 81566cc1aa1b2ee17fae1859b5a53a500673a073, data reload: false

query1	977	397	431	397
query2	6188	1928	1860	1860
query3	8679	204	201	201
query4	33044	23684	23194	23194
query5	3696	577	436	436
query6	266	198	188	188
query7	4202	483	327	327
query8	321	248	243	243
query9	9550	2608	2581	2581
query10	474	315	248	248
query11	18095	15407	15187	15187
query12	156	108	103	103
query13	1547	539	414	414
query14	9379	6487	7204	6487
query15	279	188	181	181
query16	8393	645	499	499
query17	1550	772	583	583
query18	2168	447	314	314
query19	219	191	168	168
query20	129	134	124	124
query21	204	125	110	110
query22	4607	4686	4374	4374
query23	34970	34098	33844	33844
query24	7208	2709	2670	2670
query25	567	491	435	435
query26	807	291	175	175
query27	2092	471	364	364
query28	5303	2250	2212	2212
query29	731	637	469	469
query30	254	188	166	166
query31	1008	953	838	838
query32	93	66	58	58
query33	523	369	309	309
query34	759	858	539	539
query35	808	816	768	768
query36	998	1096	988	988
query37	119	102	73	73
query38	4049	4015	3919	3919
query39	1554	1454	1452	1452
query40	206	116	102	102
query41	47	45	44	44
query42	117	105	116	105
query43	521	544	488	488
query44	1350	843	837	837
query45	180	183	177	177
query46	885	1053	669	669
query47	1980	2017	1946	1946
query48	409	425	356	356
query49	734	508	412	412
query50	691	686	436	436
query51	7303	7220	7282	7220
query52	104	104	95	95
query53	232	268	205	205
query54	556	556	469	469
query55	85	77	73	73
query56	272	274	248	248
query57	1279	1276	1199	1199
query58	236	219	212	212
query59	3037	3234	2972	2972
query60	296	283	260	260
query61	114	112	110	110
query62	795	747	687	687
query63	236	196	188	188
query64	3898	1004	641	641
query65	3380	3297	3350	3297
query66	947	415	305	305
query67	16654	15573	15424	15424
query68	8040	827	546	546
query69	498	306	260	260
query70	1199	1150	1092	1092
query71	362	291	261	261
query72	5729	3743	3880	3743
query73	625	746	367	367
query74	10637	9105	9241	9105
query75	3245	3165	2611	2611
query76	3122	1180	776	776
query77	632	358	272	272
query78	10355	10382	9580	9580
query79	3251	856	582	582
query80	678	524	441	441
query81	501	253	218	218
query82	574	119	94	94
query83	182	159	150	150
query84	278	102	84	84
query85	777	357	312	312
query86	343	275	303	275
query87	4249	4347	4193	4193
query88	4944	2412	2397	2397
query89	406	332	296	296
query90	1790	188	187	187
query91	136	152	112	112
query92	65	56	53	53
query93	1829	908	551	551
query94	693	393	302	302
query95	341	273	261	261
query96	499	606	281	281
query97	3181	3286	3150	3150
query98	231	208	208	208
query99	1531	1401	1290	1290
Total cold run time: 293433 ms
Total hot run time: 190725 ms

@doris-robot
Copy link

doris-robot commented Sep 5, 2025

ClickBench: Total hot run time: 28.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 81566cc1aa1b2ee17fae1859b5a53a500673a073, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.02	0.03
query3	0.24	0.07	0.07
query4	1.61	0.11	0.11
query5	0.50	0.52	0.49
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.50	0.51
query10	0.55	0.54	0.56
query11	0.15	0.11	0.10
query12	0.13	0.11	0.10
query13	0.61	0.60	0.59
query14	0.80	0.79	0.80
query15	0.86	0.83	0.82
query16	0.41	0.38	0.41
query17	1.03	1.05	1.01
query18	0.25	0.22	0.22
query19	1.85	1.83	1.80
query20	0.02	0.00	0.02
query21	15.39	0.94	0.59
query22	0.73	0.87	0.70
query23	15.04	1.44	0.58
query24	3.88	1.59	0.32
query25	0.11	0.07	0.10
query26	0.25	0.17	0.14
query27	0.05	0.04	0.05
query28	12.62	1.03	0.44
query29	12.55	3.93	3.27
query30	0.25	0.09	0.06
query31	2.83	0.62	0.39
query32	3.22	0.54	0.47
query33	2.99	3.03	3.03
query34	16.54	5.20	4.50
query35	4.58	4.54	4.50
query36	0.64	0.49	0.47
query37	0.09	0.06	0.05
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.06 s
Total hot run time: 28.19 s

@morrySnow morrySnow changed the title branch-3.1: [refactor](cloud) separate cloud restore create tablet rpc into multiple batches branch-3.1: [refactor](cloud) separate cloud restore create tablet rpc into multiple batches #54933 Sep 5, 2025
@morrySnow morrySnow merged commit d41dd79 into apache:branch-3.1 Sep 5, 2025
24 checks passed
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@xy720 @Thearas @doris-robot @morrySnow