Skip to content

branch-3.1: [enhancement](filecache) parallel clear cache between multiple cache instance #55259 #55437

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Sep 4, 2025
Merged

branch-3.1: [enhancement](filecache) parallel clear cache between multiple cache instance #55259 #55437

morrySnow merged 1 commit into from
Sep 4, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 28, 2025

Cherry-picked from #55259

@freemandealer
[enhancement](filecache) parallel clear cache between multiple cache …
3af2d22 
…instance (#55259)

…instance

cache instance are indepandent. clear cache can be execute concurrently
to speed up the clear process.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@github-actions github-actions bot requested a review from morrySnow as a code owner August 28, 2025 09:57
@Thearas
Copy link
Contributor

Thearas commented Aug 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Aug 28, 2025
@Thearas
Copy link
Contributor

Thearas commented Aug 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 28, 2025

TPC-H: Total hot run time: 32795 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3af2d22e29a4abbc9539fd031fcda71dfb39c39d, data reload: false

------ Round 1 ----------------------------------
q1	17588	5483	5447	5447
q2	2051	280	154	154
q3	10441	1328	741	741
q4	10236	871	447	447
q5	7988	2364	2163	2163
q6	180	163	132	132
q7	898	749	627	627
q8	9327	1458	1176	1176
q9	5544	4967	4943	4943
q10	6849	2273	1841	1841
q11	484	289	273	273
q12	342	352	212	212
q13	17773	3613	3023	3023
q14	227	234	216	216
q15	531	479	456	456
q16	413	431	375	375
q17	612	868	361	361
q18	7061	6431	6387	6387
q19	1210	949	568	568
q20	346	353	208	208
q21	2987	2211	2061	2061
q22	1062	1029	984	984
Total cold run time: 104150 ms
Total hot run time: 32795 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5560	5538	5581	5538
q2	243	333	229	229
q3	2221	2672	2338	2338
q4	1376	1765	1354	1354
q5	4392	4896	5162	4896
q6	169	159	126	126
q7	2122	1953	1797	1797
q8	2693	2825	2721	2721
q9	7299	7139	7182	7139
q10	3001	3288	2818	2818
q11	565	520	489	489
q12	672	737	619	619
q13	3461	3818	3212	3212
q14	281	296	273	273
q15	519	481	479	479
q16	443	486	459	459
q17	1249	1744	1262	1262
q18	7660	7547	7306	7306
q19	844	1164	1275	1164
q20	2026	2044	1910	1910
q21	5287	4876	4622	4622
q22	1112	1100	1050	1050
Total cold run time: 53195 ms
Total hot run time: 51801 ms

@doris-robot
Copy link

doris-robot commented Aug 28, 2025

TPC-DS: Total hot run time: 191628 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3af2d22e29a4abbc9539fd031fcda71dfb39c39d, data reload: false

query1	974	388	387	387
query2	6534	2007	1955	1955
query3	6710	215	219	215
query4	34108	23730	23391	23391
query5	4389	615	476	476
query6	267	191	183	183
query7	4623	504	315	315
query8	294	231	224	224
query9	9626	2608	2638	2608
query10	478	330	271	271
query11	18517	15822	15311	15311
query12	177	110	103	103
query13	1652	565	427	427
query14	10891	7101	7589	7101
query15	241	193	185	185
query16	8084	617	440	440
query17	1603	728	547	547
query18	2124	392	314	314
query19	224	180	150	150
query20	117	112	111	111
query21	206	126	103	103
query22	4165	4237	4112	4112
query23	34826	33672	33710	33672
query24	7658	2622	2663	2622
query25	551	480	419	419
query26	1198	288	180	180
query27	2023	480	362	362
query28	5007	2211	2164	2164
query29	771	583	448	448
query30	263	193	156	156
query31	959	868	831	831
query32	92	64	65	64
query33	528	363	316	316
query34	736	860	527	527
query35	770	821	717	717
query36	984	1030	936	936
query37	107	95	73	73
query38	3888	3905	3954	3905
query39	1546	1431	1458	1431
query40	214	121	110	110
query41	53	58	52	52
query42	120	110	108	108
query43	520	521	502	502
query44	1328	808	806	806
query45	185	181	180	180
query46	904	1047	666	666
query47	1895	1923	1849	1849
query48	430	445	343	343
query49	810	532	429	429
query50	701	698	429	429
query51	7320	7163	7110	7110
query52	107	111	99	99
query53	236	272	194	194
query54	568	565	488	488
query55	83	84	87	84
query56	288	286	263	263
query57	1217	1210	1171	1171
query58	255	224	220	220
query59	3007	3173	2992	2992
query60	313	295	278	278
query61	140	157	136	136
query62	798	735	675	675
query63	235	199	202	199
query64	4690	1068	655	655
query65	3307	3215	3194	3194
query66	1059	413	314	314
query67	16096	15713	15508	15508
query68	7775	864	558	558
query69	519	316	271	271
query70	1189	1160	1148	1148
query71	503	294	268	268
query72	5639	3770	3778	3770
query73	642	745	364	364
query74	10666	9490	9339	9339
query75	3181	3140	2661	2661
query76	3154	1207	778	778
query77	461	376	286	286
query78	10344	10539	9670	9670
query79	3339	914	601	601
query80	708	536	446	446
query81	503	268	224	224
query82	682	115	90	90
query83	177	177	177	177
query84	240	98	85	85
query85	785	350	302	302
query86	410	316	305	305
query87	4361	4440	4324	4324
query88	5188	2447	2408	2408
query89	411	347	291	291
query90	1859	195	186	186
query91	143	141	111	111
query92	63	55	56	55
query93	2485	946	548	548
query94	705	414	309	309
query95	346	279	263	263
query96	503	612	294	294
query97	3184	3324	3192	3192
query98	236	211	203	203
query99	1621	1438	1295	1295
Total cold run time: 296812 ms
Total hot run time: 191628 ms

@doris-robot
Copy link

doris-robot commented Aug 28, 2025

ClickBench: Total hot run time: 29.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3af2d22e29a4abbc9539fd031fcda71dfb39c39d, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.53	0.52	0.53
query6	1.17	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.03
query9	0.58	0.51	0.49
query10	0.56	0.54	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.61	0.59
query14	0.78	0.84	0.78
query15	0.84	0.81	0.84
query16	0.39	0.38	0.40
query17	0.99	1.03	1.04
query18	0.24	0.22	0.22
query19	1.96	1.88	1.84
query20	0.02	0.01	0.01
query21	15.38	0.94	0.60
query22	0.73	0.77	0.65
query23	15.17	1.46	0.54
query24	2.98	1.25	1.66
query25	0.24	0.18	0.07
query26	0.23	0.15	0.14
query27	0.05	0.05	0.04
query28	14.10	1.00	0.44
query29	12.59	3.96	3.24
query30	0.24	0.08	0.08
query31	2.82	0.62	0.38
query32	3.22	0.55	0.45
query33	3.04	3.06	3.10
query34	16.75	5.18	4.48
query35	4.61	4.57	4.53
query36	0.64	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.03
query40	0.17	0.13	0.14
query41	0.08	0.03	0.03
query42	0.03	0.02	0.03
query43	0.03	0.04	0.03
Total cold run time: 104.25 s
Total hot run time: 29.18 s

@doris-robot
Copy link

doris-robot commented Aug 28, 2025

BE UT Coverage Report

Increment line coverage 100.00% (8/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.49% (12726/27974)
Line Coverage 36.37% (113453/311942)
Region Coverage 34.00% (64945/191011)
Branch Coverage 31.05% (34096/109814)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 28, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (8/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.36% (21003/27504)
Line Coverage 69.69% (216662/310885)
Region Coverage 67.66% (129736/191745)
Branch Coverage 61.22% (67525/110306)

@morrySnow morrySnow merged commit 1752204 into branch-3.1 Sep 4, 2025
22 of 23 checks passed
@morrySnow morrySnow deleted the auto-pick-55259-branch-3.1 branch September 4, 2025 01:55
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Dec 17, 2025
@cambyzju
Revert "branch-3.1: [enhancement](filecache) parallel clear cache bet…
6c37b11 
…ween multiple cache instance apache#55259 (apache#55437)"

This reverts commit 1752204.
cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Dec 30, 2025
@cambyzju
Revert "branch-3.1: [enhancement](filecache) parallel clear cache bet…
90e3a12 
…ween multiple cache instance apache#55259 (apache#55437)"

This reverts commit 1752204.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Thearas @doris-robot @hello-stephen @morrySnow @dataroaring @freemandealer