Skip to content

[enhancement](filecache) parallel clear cache between multiple cache instance #55259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Aug 28, 2025
Merged

[enhancement](filecache) parallel clear cache between multiple cache instance #55259

dataroaring merged 1 commit into from
Aug 28, 2025

Conversation

@freemandealer
Copy link
Contributor

@freemandealer freemandealer commented Aug 25, 2025 •
edited
Loading

…instance

cache instance are indepandent. clear cache can be execute concurrently to speed up the clear process.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@freemandealer
[enhancement](filecache) parallel clear cache between multiple cache …
f165980 
…instance

cache instance are indepandent. clear cache can be execute concurrently
to speed up the clear process.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@freemandealer
Copy link
Contributor Author

freemandealer commented Aug 25, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 25, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@freemandealer
Copy link
Contributor Author

freemandealer commented Aug 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 25, 2025

TPC-H: Total hot run time: 34185 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f16598016b5c592995b9c7a69f6b0216e8f655dc, data reload: false

------ Round 1 ----------------------------------
q1	17585	5400	5121	5121
q2	1922	273	184	184
q3	10332	1242	698	698
q4	10220	1035	509	509
q5	7492	2380	2395	2380
q6	187	162	135	135
q7	905	759	598	598
q8	9317	1336	1136	1136
q9	6962	5136	5186	5136
q10	6940	2368	1985	1985
q11	478	303	261	261
q12	385	347	224	224
q13	17774	3638	3048	3048
q14	223	247	227	227
q15	571	487	489	487
q16	452	437	382	382
q17	596	863	365	365
q18	7797	7174	7148	7148
q19	1087	942	559	559
q20	362	337	223	223
q21	4132	3303	2391	2391
q22	1093	1036	988	988
Total cold run time: 106812 ms
Total hot run time: 34185 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5157	5136	5151	5136
q2	241	323	226	226
q3	2224	2719	2263	2263
q4	1370	1757	1307	1307
q5	4263	4654	4592	4592
q6	223	192	129	129
q7	2014	1961	1819	1819
q8	2807	2619	2592	2592
q9	7409	7252	7240	7240
q10	3140	3290	2842	2842
q11	589	630	524	524
q12	706	789	614	614
q13	3422	3921	3316	3316
q14	290	302	323	302
q15	532	513	498	498
q16	454	531	438	438
q17	1210	1536	1391	1391
q18	7988	7761	7616	7616
q19	813	850	879	850
q20	2064	2118	1951	1951
q21	4855	4492	4330	4330
q22	1065	1026	990	990
Total cold run time: 52836 ms
Total hot run time: 50966 ms

@doris-robot
Copy link

doris-robot commented Aug 25, 2025

TPC-DS: Total hot run time: 185012 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f16598016b5c592995b9c7a69f6b0216e8f655dc, data reload: false

query1	1004	390	398	390
query2	6514	1799	1746	1746
query3	6747	224	224	224
query4	26121	23329	23228	23228
query5	4332	613	474	474
query6	300	205	218	205
query7	4643	494	278	278
query8	270	219	211	211
query9	8566	2857	2792	2792
query10	469	314	299	299
query11	15887	15119	15045	15045
query12	160	114	112	112
query13	1675	565	427	427
query14	9105	5826	5717	5717
query15	205	185	166	166
query16	7333	673	485	485
query17	1176	697	570	570
query18	1988	408	307	307
query19	191	177	161	161
query20	123	118	114	114
query21	214	126	110	110
query22	4119	4224	3994	3994
query23	33675	33037	32876	32876
query24	8189	2362	2360	2360
query25	537	473	417	417
query26	1230	267	156	156
query27	2755	512	349	349
query28	4315	2247	2243	2243
query29	759	561	450	450
query30	282	231	194	194
query31	891	752	710	710
query32	83	74	70	70
query33	549	363	346	346
query34	788	847	507	507
query35	755	847	745	745
query36	990	1013	919	919
query37	131	108	87	87
query38	4106	3985	3914	3914
query39	1486	1416	1418	1416
query40	218	125	114	114
query41	59	55	56	55
query42	120	115	117	115
query43	507	491	489	489
query44	1339	849	868	849
query45	184	174	166	166
query46	858	1020	652	652
query47	1784	1801	1703	1703
query48	441	413	324	324
query49	709	475	390	390
query50	653	686	414	414
query51	4144	4058	4050	4050
query52	115	115	96	96
query53	244	259	200	200
query54	580	582	524	524
query55	130	95	85	85
query56	313	302	303	302
query57	1189	1161	1120	1120
query58	280	262	268	262
query59	2606	2746	2650	2650
query60	346	325	312	312
query61	129	123	129	123
query62	798	710	681	681
query63	227	190	184	184
query64	4375	1044	707	707
query65	4285	4208	4193	4193
query66	1140	412	331	331
query67	15707	15330	15130	15130
query68	7845	910	578	578
query69	481	320	284	284
query70	1237	1171	1146	1146
query71	442	337	340	337
query72	5869	4980	5049	4980
query73	734	711	360	360
query74	8895	8908	8886	8886
query75	3353	3058	2607	2607
query76	3323	1124	718	718
query77	681	391	326	326
query78	9797	9773	8773	8773
query79	2471	791	593	593
query80	596	535	474	474
query81	501	251	223	223
query82	459	133	99	99
query83	253	256	239	239
query84	250	100	98	98
query85	806	376	339	339
query86	386	351	313	313
query87	4250	4300	4172	4172
query88	3710	2218	2167	2167
query89	397	318	289	289
query90	1906	223	229	223
query91	139	146	109	109
query92	83	70	68	68
query93	1924	988	636	636
query94	696	416	306	306
query95	391	310	309	309
query96	484	575	276	276
query97	2635	2719	2559	2559
query98	269	230	219	219
query99	1486	1384	1271	1271
Total cold run time: 272892 ms
Total hot run time: 185012 ms

@doris-robot
Copy link

doris-robot commented Aug 25, 2025

ClickBench: Total hot run time: 32.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f16598016b5c592995b9c7a69f6b0216e8f655dc, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.05	0.04
query3	0.23	0.08	0.07
query4	1.63	0.11	0.10
query5	0.42	0.42	0.39
query6	1.18	0.64	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.60	0.53	0.50
query10	0.57	0.56	0.58
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.63	0.61	0.61
query14	0.79	0.84	0.83
query15	0.88	0.85	0.87
query16	0.39	0.39	0.39
query17	1.04	1.05	1.01
query18	0.21	0.20	0.19
query19	1.91	1.83	1.83
query20	0.01	0.01	0.01
query21	15.43	0.96	0.59
query22	0.78	1.14	0.77
query23	14.83	1.42	0.62
query24	6.45	1.62	0.97
query25	0.51	0.21	0.09
query26	0.51	0.17	0.12
query27	0.05	0.05	0.05
query28	10.31	0.96	0.43
query29	12.59	4.00	3.21
query30	3.08	3.01	2.98
query31	2.82	0.59	0.37
query32	3.23	0.54	0.47
query33	3.11	3.08	3.03
query34	16.07	5.42	4.84
query35	4.90	4.87	4.94
query36	0.68	0.52	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.84 s
Total hot run time: 32.55 s

@doris-robot
Copy link

doris-robot commented Aug 25, 2025

BE UT Coverage Report

Increment line coverage 100.00% (8/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 51.75% (17132/33103)
Line Coverage 37.23% (156064/419210)
Region Coverage 31.93% (119049/372792)
Branch Coverage 33.23% (52303/157416)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 25, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (8/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.68% (22980/32511)
Line Coverage 56.90% (238409/419025)
Region Coverage 52.45% (198387/378258)
Branch Coverage 53.99% (85518/158393)

dataroaring
dataroaring approved these changes Aug 27, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Aug 27, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 27, 2025

PR approved by anyone and no changes requested.

@gavinchou gavinchou changed the title [enhancement](filecache) parallel clear cache between multiple cache … [enhancement](filecache) parallel clear cache between multiple cache instance Aug 28, 2025
@dataroaring dataroaring merged commit 20f12f4 into apache:master Aug 28, 2025
31 of 34 checks passed
github-actions bot pushed a commit that referenced this pull request Aug 28, 2025
@freemandealer
[enhancement](filecache) parallel clear cache between multiple cache …
3af2d22 
…instance (#55259)

…instance

cache instance are indepandent. clear cache can be execute concurrently
to speed up the clear process.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@github-actions github-actions bot mentioned this pull request Aug 28, 2025
Merged
zhiqiang-hhhh pushed a commit to zhiqiang-hhhh/doris that referenced this pull request Aug 29, 2025
@freemandealer @zhiqiang-hhhh
[enhancement](filecache) parallel clear cache between multiple cache …
0e7b56a 
…instance (apache#55259)

…instance

cache instance are indepandent. clear cache can be execute concurrently
to speed up the clear process.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
morrySnow pushed a commit that referenced this pull request Sep 4, 2025
@github-actions @freemandealer
branch-3.1: [enhancement](filecache) parallel clear cache between mul…
1752204 
…tiple cache instance #55259 (#55437)

Cherry-picked from #55259

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
Co-authored-by: zhengyu <zhangzhengyu@selectdb.com>
@zy-kkk zy-kkk mentioned this pull request Sep 11, 2025
Closed
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Dec 17, 2025
@cambyzju
Revert "branch-3.1: [enhancement](filecache) parallel clear cache bet…
6c37b11 
…ween multiple cache instance apache#55259 (apache#55437)"

This reverts commit 1752204.
cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Dec 30, 2025
@cambyzju
Revert "branch-3.1: [enhancement](filecache) parallel clear cache bet…
90e3a12 
…ween multiple cache instance apache#55259 (apache#55437)"

This reverts commit 1752204.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@freemandealer @hello-stephen @doris-robot @gavinchou @dataroaring @morrySnow