Skip to content

[fix](filecache) no fd num limit for segment cache when cloud mode #55610

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
gavinchou merged 1 commit into from
Sep 3, 2025
Merged

[fix](filecache) no fd num limit for segment cache when cloud mode #55610

gavinchou merged 1 commit into from
Sep 3, 2025

Conversation

@freemandealer
Copy link
Contributor

@freemandealer freemandealer commented Sep 3, 2025

when in cloud mode, segment cache hold no system FD, thus the FD num limit makes no sense. cloud mode use FDCache to control FD.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@freemandealer
[fix](filecache) no fd num limit for segment cache when cloud mode
a910f08 
when in cloud mode, segment cache hold no system FD, thus the FD
num limit makes no sense. cloud mode use FDCache to control FD.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@freemandealer
Copy link
Contributor Author

freemandealer commented Sep 3, 2025

run buildall

dataroaring
dataroaring approved these changes Sep 3, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 3, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 3, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Sep 3, 2025

TPC-H: Total hot run time: 34844 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a910f084faa5ba19efb8332c7e3f688dbf69fa63, data reload: false

------ Round 1 ----------------------------------
q1	17625	5510	5184	5184
q2	2002	332	215	215
q3	10235	1416	739	739
q4	10237	1079	526	526
q5	7539	2528	2481	2481
q6	210	172	141	141
q7	998	768	640	640
q8	9380	1556	1344	1344
q9	7141	5427	5324	5324
q10	6991	2399	1991	1991
q11	517	303	293	293
q12	387	384	223	223
q13	17776	3788	2999	2999
q14	240	243	222	222
q15	582	519	483	483
q16	437	450	383	383
q17	625	919	374	374
q18	7452	7253	7067	7067
q19	1163	1086	610	610
q20	363	361	234	234
q21	4102	3354	2383	2383
q22	1099	1038	988	988
Total cold run time: 107101 ms
Total hot run time: 34844 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5506	5513	5422	5422
q2	273	357	233	233
q3	2238	2744	2321	2321
q4	1333	1818	1298	1298
q5	4556	4600	4585	4585
q6	259	183	130	130
q7	2090	2036	1760	1760
q8	2885	2832	2750	2750
q9	7554	7378	7453	7378
q10	3166	3324	2870	2870
q11	645	525	511	511
q12	732	797	630	630
q13	3631	4024	3291	3291
q14	306	304	277	277
q15	546	485	478	478
q16	468	513	458	458
q17	1260	1776	1510	1510
q18	8090	7714	7582	7582
q19	854	891	1030	891
q20	1975	1976	1815	1815
q21	4948	4444	4327	4327
q22	1100	1067	1020	1020
Total cold run time: 54415 ms
Total hot run time: 51537 ms

@doris-robot
Copy link

doris-robot commented Sep 3, 2025

TPC-DS: Total hot run time: 187620 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a910f084faa5ba19efb8332c7e3f688dbf69fa63, data reload: false

query1	1048	441	419	419
query2	6560	1710	1727	1710
query3	6762	223	220	220
query4	26604	23393	23787	23393
query5	4394	633	504	504
query6	328	233	236	233
query7	4643	543	299	299
query8	294	251	248	248
query9	8639	2937	2914	2914
query10	477	344	302	302
query11	15432	15078	14833	14833
query12	174	121	117	117
query13	1674	551	431	431
query14	8757	5806	5894	5806
query15	215	189	176	176
query16	7177	686	468	468
query17	1002	769	649	649
query18	2017	452	352	352
query19	216	211	182	182
query20	139	133	126	126
query21	212	130	122	122
query22	4162	4168	4021	4021
query23	33954	33066	33049	33049
query24	8243	2458	2462	2458
query25	613	551	476	476
query26	1258	290	173	173
query27	2722	525	357	357
query28	4399	2316	2354	2316
query29	814	602	482	482
query30	287	220	194	194
query31	891	826	719	719
query32	97	84	80	80
query33	580	410	361	361
query34	859	863	554	554
query35	828	829	748	748
query36	978	1030	909	909
query37	127	112	86	86
query38	4099	4124	4060	4060
query39	1479	1425	1424	1424
query40	229	136	128	128
query41	69	61	60	60
query42	129	120	119	119
query43	535	514	479	479
query44	1354	889	866	866
query45	186	182	178	178
query46	866	1036	677	677
query47	1763	1786	1724	1724
query48	410	432	329	329
query49	763	510	409	409
query50	646	687	397	397
query51	4191	4207	4101	4101
query52	115	114	110	110
query53	254	264	207	207
query54	623	604	552	552
query55	94	91	91	91
query56	362	336	323	323
query57	1183	1190	1118	1118
query58	292	300	285	285
query59	2615	2777	2587	2587
query60	373	354	340	340
query61	159	155	157	155
query62	816	739	680	680
query63	234	205	197	197
query64	4466	1194	854	854
query65	4307	4195	4255	4195
query66	1167	435	344	344
query67	15375	15519	15111	15111
query68	8948	940	588	588
query69	508	336	306	306
query70	1247	1184	1183	1183
query71	584	351	331	331
query72	6067	5180	5098	5098
query73	780	668	365	365
query74	8955	9306	9074	9074
query75	4279	3087	2643	2643
query76	3657	1183	744	744
query77	837	399	341	341
query78	9546	9711	8855	8855
query79	4489	839	587	587
query80	698	568	531	531
query81	479	262	233	233
query82	632	140	119	119
query83	294	273	264	264
query84	307	157	95	95
query85	870	450	424	424
query86	348	315	301	301
query87	4379	4300	4359	4300
query88	2983	2237	2227	2227
query89	462	340	306	306
query90	1935	233	229	229
query91	161	164	131	131
query92	89	81	76	76
query93	2385	1025	641	641
query94	694	410	327	327
query95	408	338	324	324
query96	494	601	282	282
query97	2675	2702	2605	2605
query98	247	218	224	218
query99	1447	1404	1285	1285
Total cold run time: 278614 ms
Total hot run time: 187620 ms

@doris-robot
Copy link

doris-robot commented Sep 3, 2025

ClickBench: Total hot run time: 33.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a910f084faa5ba19efb8332c7e3f688dbf69fa63, data reload: false

query1	0.06	0.05	0.06
query2	0.10	0.05	0.06
query3	0.26	0.09	0.08
query4	1.61	0.12	0.12
query5	0.46	0.43	0.42
query6	1.17	0.65	0.68
query7	0.03	0.03	0.02
query8	0.06	0.05	0.04
query9	0.60	0.54	0.54
query10	0.59	0.58	0.59
query11	0.16	0.12	0.11
query12	0.16	0.13	0.12
query13	0.63	0.62	0.62
query14	0.80	0.86	0.84
query15	0.87	0.86	0.85
query16	0.40	0.41	0.40
query17	1.05	1.07	1.04
query18	0.22	0.20	0.21
query19	2.03	1.85	1.85
query20	0.02	0.02	0.01
query21	15.40	1.01	0.58
query22	0.76	1.12	0.75
query23	14.93	1.40	0.62
query24	6.68	1.16	1.16
query25	0.51	0.28	0.09
query26	0.62	0.16	0.13
query27	0.06	0.05	0.05
query28	9.89	0.88	0.43
query29	12.68	3.89	3.22
query30	3.13	3.05	3.01
query31	2.83	0.59	0.39
query32	3.22	0.58	0.47
query33	3.08	3.06	3.13
query34	16.05	5.40	4.86
query35	4.93	4.91	4.94
query36	0.70	0.52	0.50
query37	0.12	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 107.33 s
Total hot run time: 33.2 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 3, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/9) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 51.83% (17205/33192)
Line Coverage 37.27% (157099/421541)
Region Coverage 31.88% (119862/375955)
Branch Coverage 33.29% (52655/158192)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 3, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (9/9) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.62% (23019/32594)
Line Coverage 56.99% (240068/421278)
Region Coverage 52.37% (199701/381333)
Branch Coverage 54.09% (86056/159085)

@gavinchou gavinchou merged commit c7ddfbb into apache:master Sep 3, 2025
27 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Sep 3, 2025
@freemandealer
[fix](filecache) no fd num limit for segment cache when cloud mode (#…
e1f0471 
…55610)

when in cloud mode, segment cache hold no system FD, thus the FD num
limit makes no sense. cloud mode use FDCache to control FD.
@github-actions github-actions bot mentioned this pull request Sep 3, 2025
Merged
morrySnow pushed a commit that referenced this pull request Sep 4, 2025
@github-actions @freemandealer
branch-3.1: [fix](filecache) no fd num limit for segment cache when c…
74ec1cf 
…loud mode #55610 (#55638)

Cherry-picked from #55610

Co-authored-by: zhengyu <freeman.zhang1992@gmail.com>
wenzhenghu pushed a commit to wenzhenghu/doris that referenced this pull request Sep 8, 2025
@freemandealer @wenzhenghu
[fix](filecache) no fd num limit for segment cache when cloud mode (a…
3483feb 
…pache#55610)

when in cloud mode, segment cache hold no system FD, thus the FD num
limit makes no sense. cloud mode use FDCache to control FD.
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@freemandealer @hello-stephen @doris-robot @gavinchou @dataroaring @morrySnow