Skip to content

[fix](restore) Clear restored table/partition/resource to reduce overhead #55757

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
w41ter merged 1 commit into from
Sep 8, 2025
Merged

[fix](restore) Clear restored table/partition/resource to reduce overhead #55757

w41ter merged 1 commit into from
Sep 8, 2025

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Sep 8, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Sep 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter
Copy link
Contributor Author

w41ter commented Sep 8, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 8, 2025

TPC-H: Total hot run time: 34574 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 20fd4f424614ca9697de1a7faf60cb59c84fc8f3, data reload: false

------ Round 1 ----------------------------------
q1	17682	5239	5062	5062
q2	2020	377	209	209
q3	10216	1293	730	730
q4	10221	1020	527	527
q5	7542	2422	2304	2304
q6	189	165	138	138
q7	909	771	657	657
q8	9348	1325	1147	1147
q9	6900	5149	5063	5063
q10	6953	2388	1987	1987
q11	495	304	281	281
q12	350	353	219	219
q13	17773	3613	3043	3043
q14	239	239	218	218
q15	575	501	477	477
q16	981	990	965	965
q17	610	869	368	368
q18	7415	7071	7114	7071
q19	1469	947	557	557
q20	344	333	231	231
q21	3786	2555	2328	2328
q22	1076	1030	992	992
Total cold run time: 107093 ms
Total hot run time: 34574 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5198	5038	5074	5038
q2	247	332	228	228
q3	2177	2638	2337	2337
q4	1342	1736	1346	1346
q5	4197	4373	4523	4373
q6	216	172	132	132
q7	2042	2001	1802	1802
q8	2683	2551	2614	2551
q9	7317	7341	7427	7341
q10	3106	3277	2904	2904
q11	621	511	503	503
q12	722	951	626	626
q13	3485	3789	3405	3405
q14	317	309	295	295
q15	508	485	505	485
q16	1063	1124	1068	1068
q17	1179	1566	1377	1377
q18	7976	7658	7649	7649
q19	821	806	839	806
q20	1931	1950	1888	1888
q21	4815	4356	4279	4279
q22	1093	1074	1055	1055
Total cold run time: 53056 ms
Total hot run time: 51488 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 8, 2025

FE UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 8, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (3/3) 🎉
Increment coverage report
Complete coverage report

dataroaring
dataroaring approved these changes Sep 8, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 8, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 8, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 8, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Sep 8, 2025

TPC-H: Total hot run time: 34831 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 20fd4f424614ca9697de1a7faf60cb59c84fc8f3, data reload: false

------ Round 1 ----------------------------------
q1	17647	5326	5073	5073
q2	2000	356	230	230
q3	10205	1342	740	740
q4	10221	1060	530	530
q5	7541	2349	2402	2349
q6	187	173	138	138
q7	934	779	636	636
q8	9337	1363	1157	1157
q9	6849	5113	5070	5070
q10	6917	2411	1984	1984
q11	502	322	283	283
q12	353	365	235	235
q13	17766	3739	3081	3081
q14	243	247	217	217
q15	569	506	498	498
q16	1005	1018	950	950
q17	605	878	379	379
q18	7957	7230	7086	7086
q19	1255	973	582	582
q20	346	341	238	238
q21	3767	2568	2405	2405
q22	1068	1057	970	970
Total cold run time: 107274 ms
Total hot run time: 34831 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5163	5116	5092	5092
q2	255	344	233	233
q3	2209	2719	2347	2347
q4	1360	1822	1411	1411
q5	4306	4687	4651	4651
q6	214	171	139	139
q7	2089	1992	1862	1862
q8	2692	2597	2523	2523
q9	7433	7435	7320	7320
q10	3168	3297	2903	2903
q11	579	513	525	513
q12	696	789	642	642
q13	3576	4099	3270	3270
q14	281	293	307	293
q15	605	500	489	489
q16	1100	1131	1060	1060
q17	1202	1626	1403	1403
q18	7831	7915	7511	7511
q19	826	842	1112	842
q20	2037	2014	1992	1992
q21	5202	4623	4378	4378
q22	1075	1039	984	984
Total cold run time: 53899 ms
Total hot run time: 51858 ms

@doris-robot
Copy link

doris-robot commented Sep 8, 2025

TPC-DS: Total hot run time: 189429 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 20fd4f424614ca9697de1a7faf60cb59c84fc8f3, data reload: false

query1	1070	437	412	412
query2	6556	1716	1703	1703
query3	6751	223	220	220
query4	26001	23272	22907	22907
query5	4441	674	546	546
query6	359	246	231	231
query7	4649	516	303	303
query8	326	267	283	267
query9	8621	2938	2909	2909
query10	476	361	316	316
query11	15912	14989	14850	14850
query12	184	119	122	119
query13	1672	572	438	438
query14	11692	9231	9306	9231
query15	217	195	176	176
query16	7678	675	487	487
query17	1299	753	629	629
query18	2052	444	356	356
query19	211	224	192	192
query20	135	142	136	136
query21	214	137	119	119
query22	4090	4081	4079	4079
query23	33940	33148	32916	32916
query24	8117	2350	2423	2350
query25	575	528	450	450
query26	1236	276	166	166
query27	2691	529	360	360
query28	4374	2330	2359	2330
query29	770	621	490	490
query30	293	235	206	206
query31	920	817	736	736
query32	91	86	89	86
query33	583	399	372	372
query34	804	864	526	526
query35	814	830	737	737
query36	961	1021	924	924
query37	134	122	104	104
query38	3582	3596	3454	3454
query39	1504	1437	1470	1437
query40	232	139	129	129
query41	65	67	63	63
query42	130	116	128	116
query43	521	487	486	486
query44	1343	885	860	860
query45	185	174	172	172
query46	865	1022	677	677
query47	1786	1790	1745	1745
query48	403	468	324	324
query49	758	517	409	409
query50	653	714	405	405
query51	3870	3992	3914	3914
query52	120	118	111	111
query53	255	277	202	202
query54	625	638	541	541
query55	102	95	93	93
query56	350	341	310	310
query57	1194	1173	1115	1115
query58	294	290	287	287
query59	2651	2729	2599	2599
query60	389	365	353	353
query61	166	158	161	158
query62	801	737	658	658
query63	240	199	199	199
query64	4449	1180	867	867
query65	4053	3997	4006	3997
query66	1122	456	353	353
query67	15321	15439	15166	15166
query68	8349	965	604	604
query69	518	453	292	292
query70	1343	1286	1277	1277
query71	587	361	326	326
query72	5889	5122	5259	5122
query73	793	722	368	368
query74	9019	8957	8788	8788
query75	4061	3255	2824	2824
query76	3711	1179	840	840
query77	804	425	343	343
query78	9697	9837	8908	8908
query79	2151	858	604	604
query80	661	582	539	539
query81	487	266	231	231
query82	469	167	142	142
query83	269	280	250	250
query84	265	113	92	92
query85	886	529	430	430
query86	349	332	323	323
query87	3779	3728	3605	3605
query88	3442	2227	2240	2227
query89	387	331	299	299
query90	1924	225	225	225
query91	165	165	138	138
query92	93	76	71	71
query93	1608	1029	660	660
query94	707	419	325	325
query95	419	346	339	339
query96	492	586	279	279
query97	2960	3022	2855	2855
query98	247	223	221	221
query99	1691	1407	1303	1303
Total cold run time: 277160 ms
Total hot run time: 189429 ms

@doris-robot
Copy link

doris-robot commented Sep 8, 2025

ClickBench: Total hot run time: 30.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 20fd4f424614ca9697de1a7faf60cb59c84fc8f3, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.06
query3	0.25	0.08	0.08
query4	1.61	0.12	0.11
query5	0.28	0.27	0.25
query6	1.17	0.64	0.64
query7	0.03	0.02	0.02
query8	0.06	0.04	0.05
query9	0.62	0.52	0.52
query10	0.58	0.58	0.59
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.62	0.65
query14	1.04	1.03	1.08
query15	0.87	0.86	0.87
query16	0.40	0.40	0.40
query17	1.08	1.09	1.06
query18	0.22	0.20	0.19
query19	1.90	1.82	1.78
query20	0.02	0.02	0.01
query21	15.42	0.95	0.58
query22	0.75	1.16	0.89
query23	14.79	1.39	0.62
query24	7.35	0.88	1.48
query25	0.51	0.30	0.07
query26	0.43	0.16	0.14
query27	0.07	0.06	0.05
query28	10.25	0.92	0.42
query29	12.61	3.96	3.26
query30	0.29	0.14	0.12
query31	2.84	0.60	0.38
query32	3.26	0.56	0.47
query33	3.04	3.06	3.13
query34	16.17	5.53	4.81
query35	4.99	4.99	4.90
query36	0.69	0.52	0.50
query37	0.10	0.08	0.07
query38	0.06	0.05	0.04
query39	0.04	0.02	0.03
query40	0.19	0.16	0.14
query41	0.08	0.04	0.03
query42	0.04	0.04	0.02
query43	0.05	0.05	0.04
Total cold run time: 105.24 s
Total hot run time: 30.03 s

@w41ter w41ter merged commit e9e1caf into apache:master Sep 8, 2025
29 of 31 checks passed
@w41ter w41ter deleted the restore/fix/cancel_clear_restored_resources branch September 8, 2025 11:12
github-actions bot pushed a commit that referenced this pull request Sep 8, 2025
@w41ter
[fix](restore) Clear restored table/partition/resource to reduce over…
a5f1bca 
…head (#55757)
@github-actions github-actions bot mentioned this pull request Sep 8, 2025
Merged
morrySnow pushed a commit that referenced this pull request Sep 8, 2025
@github-actions @w41ter
branch-3.1: [fix](restore) Clear restored table/partition/resource to…
d481a48 
… reduce overhead #55757 (#55784)

Cherry-picked from #55757

Co-authored-by: walter <maochuan@selectdb.com>
@w41ter w41ter mentioned this pull request Sep 9, 2025
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Sep 9, 2025
@w41ter
[fix](restore) Clear restored table/partition/resource to reduce over…
30f5922 
…head (apache#55757)
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Sep 9, 2025
@w41ter
[fix](restore) Clear restored table/partition/resource to reduce over…
9c8f35d 
…head (apache#55757)
@w41ter w41ter mentioned this pull request Sep 9, 2025
Merged
dataroaring pushed a commit that referenced this pull request Sep 9, 2025
@w41ter
branch-3.0: [fix](restore) Clear restored table/partition/resource to…
0e033ff 
… reduce overhead #55757 (#55792)

cherry pick from #55757
yiguolei pushed a commit that referenced this pull request Sep 15, 2025
@w41ter
branch-2.1: [fix](restore) Clear restored table/partition/resource to…
ea716ba 
… reduce overhead #55757 (#55793)

cherry pick from #55757
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. cloud dev/2.1.12-merged dev/3.0.9-merged dev/3.1.1-merged reviewed snapshot

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@w41ter @Thearas @doris-robot @hello-stephen @gavinchou @dataroaring @yiguolei @morrySnow