Skip to content

branch-3.1: [enhancement](backup) handle dropped tables and partitions during backup #52935 #54989

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Sep 4, 2025
Merged

branch-3.1: [enhancement](backup) handle dropped tables and partitions during backup #52935 #54989

morrySnow merged 2 commits into from
Sep 4, 2025

Conversation

@peterylh
Copy link
Contributor

@peterylh peterylh commented Aug 19, 2025 •
edited by morrySnow
Loading

picked from #52935

peterylh and others added 2 commits August 19, 2025 14:37
@peterylh
[enhancement](backup) handle dropped tables and partitions during bac…
323747c 
…kup (apache#52935)

### What problem does this PR solve?

Issue Number: close apache#52274

Problem Summary:

When a backup is running, dropping a table, dropping a partition or
truncating a table would cause backup failure because BE cannot find
some tablets. This fix:

1. Add droppedTables and droppedPartitionsByTable fields to track
dropped objects during backup
2.  Filter out dropped tables and partitions during backup

This ensures backup jobs can complete successfully even when tables or
partitions are dropped during the backup process.

Co-authored-by: ylh <ylh@example.com>
@peterylh
[test](backup) fix compile error for earlier branches
c4b1c34
@peterylh peterylh requested a review from morrySnow as a code owner August 19, 2025 07:04
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@peterylh
Copy link
Contributor Author

peterylh commented Aug 19, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-H: Total hot run time: 32684 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4b1c34c172a829181e0bf9121ffb1229b16ef6d, data reload: false

------ Round 1 ----------------------------------
q1	17601	5452	5400	5400
q2	2034	384	276	276
q3	11831	1240	725	725
q4	10290	865	444	444
q5	8999	2365	2141	2141
q6	184	163	136	136
q7	900	755	632	632
q8	9345	1470	1180	1180
q9	5250	4946	5102	4946
q10	6773	2246	1808	1808
q11	472	294	269	269
q12	332	350	216	216
q13	17809	3583	3022	3022
q14	232	229	209	209
q15	549	467	466	466
q16	423	432	369	369
q17	610	857	360	360
q18	6814	6378	6356	6356
q19	1291	944	553	553
q20	331	341	212	212
q21	2894	2176	1967	1967
q22	1057	1050	997	997
Total cold run time: 106021 ms
Total hot run time: 32684 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5511	5525	5486	5486
q2	234	326	237	237
q3	2248	2640	2320	2320
q4	1337	1775	1402	1402
q5	4399	4950	4937	4937
q6	170	162	125	125
q7	2073	1935	1840	1840
q8	2584	2811	2682	2682
q9	7238	7184	7173	7173
q10	3037	3314	2653	2653
q11	557	493	506	493
q12	657	760	615	615
q13	3370	3809	3179	3179
q14	278	306	270	270
q15	513	470	473	470
q16	446	494	445	445
q17	1225	1728	1264	1264
q18	7543	7417	7235	7235
q19	810	1150	1037	1037
q20	2062	2059	1882	1882
q21	5341	4921	4546	4546
q22	1071	1081	983	983
Total cold run time: 52704 ms
Total hot run time: 51274 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-DS: Total hot run time: 192345 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c4b1c34c172a829181e0bf9121ffb1229b16ef6d, data reload: false

query1	933	394	390	390
query2	6237	2015	1922	1922
query3	8694	197	198	197
query4	33358	24168	23817	23817
query5	4667	613	455	455
query6	314	186	195	186
query7	4236	494	306	306
query8	305	252	230	230
query9	9395	2581	2567	2567
query10	493	324	270	270
query11	18237	15424	15107	15107
query12	160	110	108	108
query13	1566	553	419	419
query14	9237	7361	6756	6756
query15	275	191	179	179
query16	8102	689	529	529
query17	1545	774	582	582
query18	2183	419	318	318
query19	217	203	167	167
query20	122	136	132	132
query21	214	126	105	105
query22	4407	4736	4440	4440
query23	35053	34197	34073	34073
query24	7503	2667	2712	2667
query25	572	487	432	432
query26	808	289	177	177
query27	2033	476	364	364
query28	5485	2238	2186	2186
query29	723	610	463	463
query30	252	191	159	159
query31	1015	918	829	829
query32	93	60	57	57
query33	518	379	301	301
query34	759	864	524	524
query35	797	816	733	733
query36	992	1071	960	960
query37	103	93	71	71
query38	4042	4123	4008	4008
query39	1570	1471	1480	1471
query40	213	120	108	108
query41	50	51	52	51
query42	124	107	107	107
query43	506	537	504	504
query44	1360	828	823	823
query45	182	193	172	172
query46	895	1064	675	675
query47	1987	1996	1878	1878
query48	392	413	353	353
query49	741	505	414	414
query50	697	690	430	430
query51	7375	7235	7220	7220
query52	105	103	94	94
query53	234	252	189	189
query54	553	559	478	478
query55	81	81	76	76
query56	267	276	257	257
query57	1279	1256	1221	1221
query58	229	236	216	216
query59	3109	3272	3105	3105
query60	300	286	277	277
query61	125	109	112	109
query62	797	777	720	720
query63	237	211	204	204
query64	3905	989	644	644
query65	3387	3325	3261	3261
query66	880	441	307	307
query67	16686	15758	15737	15737
query68	7891	813	521	521
query69	493	304	262	262
query70	1180	1154	1123	1123
query71	385	293	274	274
query72	5779	3562	3835	3562
query73	635	752	352	352
query74	10502	9428	8975	8975
query75	3186	3176	2692	2692
query76	2964	1196	770	770
query77	492	359	277	277
query78	10371	10462	9629	9629
query79	3245	916	594	594
query80	608	514	434	434
query81	505	257	225	225
query82	593	116	89	89
query83	161	155	142	142
query84	241	94	82	82
query85	775	370	315	315
query86	393	305	295	295
query87	4338	4342	4319	4319
query88	4898	2405	2383	2383
query89	406	331	296	296
query90	1742	187	186	186
query91	136	138	109	109
query92	64	56	54	54
query93	2035	887	532	532
query94	702	423	314	314
query95	337	270	265	265
query96	490	608	295	295
query97	3201	3265	3165	3165
query98	218	208	210	208
query99	1330	1410	1319	1319
Total cold run time: 294207 ms
Total hot run time: 192345 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

ClickBench: Total hot run time: 28.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c4b1c34c172a829181e0bf9121ffb1229b16ef6d, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.04
query3	0.23	0.06	0.06
query4	1.62	0.12	0.10
query5	0.53	0.52	0.53
query6	1.12	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.03
query9	0.57	0.50	0.50
query10	0.56	0.57	0.55
query11	0.14	0.11	0.10
query12	0.13	0.11	0.10
query13	0.61	0.60	0.59
query14	0.79	0.80	0.79
query15	0.84	0.83	0.83
query16	0.42	0.38	0.38
query17	1.07	1.09	1.03
query18	0.23	0.20	0.22
query19	1.91	1.82	1.80
query20	0.01	0.02	0.01
query21	15.37	0.94	0.56
query22	0.74	0.85	0.67
query23	15.04	1.42	0.56
query24	3.15	1.88	0.96
query25	0.16	0.16	0.11
query26	0.34	0.14	0.14
query27	0.08	0.05	0.04
query28	13.55	0.96	0.43
query29	12.62	4.00	3.28
query30	0.25	0.09	0.07
query31	2.81	0.58	0.37
query32	3.23	0.54	0.46
query33	3.01	3.02	3.05
query34	16.61	5.19	4.55
query35	4.62	4.55	4.50
query36	0.62	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.16	0.14	0.13
query41	0.08	0.02	0.03
query42	0.04	0.04	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.64 s
Total hot run time: 28.9 s

@morrySnow morrySnow changed the title close issue 52274 for branch-3.1 branch-3.1: [enhancement](backup) handle dropped tables and partitions during backup #52935 Aug 19, 2025
@morrySnow morrySnow merged commit 41a2797 into apache:branch-3.1 Sep 4, 2025
25 checks passed
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@peterylh @hello-stephen @doris-robot @morrySnow