Skip to content

[fix](file cache) Fix inverted index cache not cleaned on compaction failure #58141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
hello-stephen merged 1 commit into from
Nov 20, 2025
Merged

[fix](file cache) Fix inverted index cache not cleaned on compaction failure #58141

hello-stephen merged 1 commit into from
Nov 20, 2025

Conversation

@zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Nov 19, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Previously, generated inverted index files were not removed from the block file cache when a compaction task failed or was aborted. This PR adds logic to correctly iterate and clean up these files.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Nov 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Nov 19, 2025

run buildall

freemandealer
freemandealer approved these changes Nov 19, 2025
View reviewed changes
Copy link
Contributor

@freemandealer freemandealer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Nov 19, 2025

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 19, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 19, 2025

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

doris-robot commented Nov 19, 2025

TPC-H: Total hot run time: 33992 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 350c43a10438c4d3c27671a0165c7bce4ed062ac, data reload: false

------ Round 1 ----------------------------------
q1	17609	5104	4937	4937
q2	2050	314	199	199
q3	10285	1305	735	735
q4	10227	907	368	368
q5	7486	2452	2306	2306
q6	181	169	137	137
q7	928	767	629	629
q8	9337	1410	1084	1084
q9	6952	5132	5103	5103
q10	6900	2250	1794	1794
q11	484	290	284	284
q12	365	363	236	236
q13	17797	3637	3024	3024
q14	227	235	209	209
q15	587	511	499	499
q16	1003	1004	940	940
q17	586	903	364	364
q18	7467	7142	7132	7132
q19	1099	959	555	555
q20	362	347	233	233
q21	3749	2644	2267	2267
q22	1041	1044	957	957
Total cold run time: 106722 ms
Total hot run time: 33992 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4956	4988	4919	4919
q2	253	323	224	224
q3	2198	2659	2349	2349
q4	1348	1800	1349	1349
q5	4212	4186	4476	4186
q6	223	172	147	147
q7	2105	1940	1814	1814
q8	2652	2694	2517	2517
q9	7355	7180	7257	7180
q10	3081	3235	2851	2851
q11	569	531	508	508
q12	671	791	618	618
q13	3827	3839	3433	3433
q14	291	315	272	272
q15	550	522	518	518
q16	1090	1120	1033	1033
q17	1175	1791	1371	1371
q18	7681	7657	7536	7536
q19	822	845	884	845
q20	2016	2257	1919	1919
q21	5041	4597	4278	4278
q22	1092	1045	1010	1010
Total cold run time: 53208 ms
Total hot run time: 50877 ms

@doris-robot
Copy link

doris-robot commented Nov 19, 2025

TPC-DS: Total hot run time: 188861 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 350c43a10438c4d3c27671a0165c7bce4ed062ac, data reload: false

query1	1030	420	396	396
query2	6601	1656	1718	1656
query3	6755	229	222	222
query4	26382	23342	23042	23042
query5	4453	644	527	527
query6	329	255	235	235
query7	4651	500	302	302
query8	296	274	263	263
query9	8727	2946	2982	2946
query10	522	359	301	301
query11	15881	15511	14791	14791
query12	191	125	118	118
query13	1709	594	470	470
query14	10734	9537	9352	9352
query15	204	192	189	189
query16	7244	681	491	491
query17	1247	789	651	651
query18	2000	435	347	347
query19	247	202	179	179
query20	136	127	122	122
query21	215	133	113	113
query22	4217	4009	3967	3967
query23	33821	32908	33329	32908
query24	8154	2401	2436	2401
query25	612	540	472	472
query26	1231	278	172	172
query27	2733	492	358	358
query28	4409	2277	2221	2221
query29	832	648	500	500
query30	296	227	195	195
query31	879	776	724	724
query32	94	90	92	90
query33	580	404	352	352
query34	794	849	533	533
query35	843	837	781	781
query36	954	1006	917	917
query37	134	114	103	103
query38	3531	3496	3497	3496
query39	1473	1424	1389	1389
query40	233	139	127	127
query41	64	62	61	61
query42	129	120	118	118
query43	508	509	458	458
query44	1281	810	802	802
query45	187	188	173	173
query46	888	1001	653	653
query47	1759	1807	1744	1744
query48	403	456	329	329
query49	772	496	416	416
query50	655	685	407	407
query51	3964	4000	3846	3846
query52	119	117	106	106
query53	256	281	209	209
query54	337	346	320	320
query55	95	97	96	96
query56	364	350	363	350
query57	1160	1180	1098	1098
query58	308	293	287	287
query59	2559	2602	2482	2482
query60	378	368	355	355
query61	164	160	159	159
query62	795	736	649	649
query63	238	204	201	201
query64	4497	1265	877	877
query65	4046	3951	3957	3951
query66	1193	435	360	360
query67	15128	14980	15039	14980
query68	6206	952	646	646
query69	520	344	312	312
query70	1328	1239	1364	1239
query71	435	350	333	333
query72	5635	4899	4853	4853
query73	644	588	365	365
query74	8932	9087	8818	8818
query75	3317	3245	2761	2761
query76	3257	1127	776	776
query77	535	415	349	349
query78	9453	9614	8945	8945
query79	1254	833	623	623
query80	702	606	536	536
query81	493	257	230	230
query82	269	167	145	145
query83	274	269	250	250
query84	251	115	99	99
query85	879	591	562	562
query86	334	338	292	292
query87	3804	3682	3632	3632
query88	2876	2288	2295	2288
query89	404	343	306	306
query90	1786	249	248	248
query91	196	190	159	159
query92	87	82	81	81
query93	1113	974	692	692
query94	694	442	354	354
query95	456	368	363	363
query96	489	610	289	289
query97	2896	2966	2883	2883
query98	244	218	216	216
query99	1265	1401	1310	1310
Total cold run time: 268649 ms
Total hot run time: 188861 ms

@doris-robot
Copy link

doris-robot commented Nov 19, 2025

ClickBench: Total hot run time: 27.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 350c43a10438c4d3c27671a0165c7bce4ed062ac, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.04
query3	0.26	0.08	0.08
query4	1.60	0.11	0.11
query5	0.26	0.25	0.24
query6	1.19	0.66	0.65
query7	0.03	0.02	0.02
query8	0.06	0.04	0.04
query9	0.58	0.54	0.53
query10	0.58	0.57	0.57
query11	0.16	0.12	0.11
query12	0.16	0.12	0.12
query13	0.61	0.61	0.61
query14	1.02	1.00	1.02
query15	0.84	0.83	0.85
query16	0.39	0.40	0.40
query17	1.04	1.00	1.01
query18	0.22	0.20	0.20
query19	1.92	1.80	1.80
query20	0.02	0.02	0.01
query21	15.45	0.19	0.12
query22	5.10	0.08	0.05
query23	15.66	0.25	0.10
query24	3.29	0.66	0.27
query25	0.07	0.07	0.06
query26	0.15	0.13	0.13
query27	0.06	0.05	0.06
query28	4.72	1.18	0.96
query29	12.58	4.00	3.24
query30	0.29	0.13	0.12
query31	2.81	0.60	0.40
query32	3.22	0.54	0.46
query33	3.01	3.11	3.06
query34	15.57	5.19	4.49
query35	4.59	4.60	4.57
query36	0.68	0.50	0.49
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 98.86 s
Total hot run time: 27.34 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 19, 2025

BE UT Coverage Report

Increment line coverage 65.00% (13/20) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.74% (18274/34651)
Line Coverage 38.10% (166029/435743)
Region Coverage 33.02% (129017/390749)
Branch Coverage 33.82% (55401/163821)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 19, 2025

BE Regression && UT Coverage Report

Increment line coverage 70.00% (14/20) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.48% (24338/34051)
Line Coverage 57.93% (252834/436410)
Region Coverage 53.07% (210346/396322)
Branch Coverage 54.52% (89903/164898)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 19, 2025

BE Regression && UT Coverage Report

Increment line coverage 70.00% (14/20) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.48% (24338/34051)
Line Coverage 57.93% (252820/436410)
Region Coverage 53.07% (210317/396322)
Branch Coverage 54.52% (89903/164898)

@hello-stephen hello-stephen merged commit e7c43f4 into apache:master Nov 20, 2025
27 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 20, 2025
@zzzxl1993
[fix](file cache) Fix inverted index cache not cleaned on compaction …
007a497 
…failure (#58141)
@github-actions github-actions bot mentioned this pull request Nov 20, 2025
Merged
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Nov 20, 2025
@zzzxl1993
[fix](file cache) Fix inverted index cache not cleaned on compaction …
68091ad 
…failure (apache#58141)
@zzzxl1993 zzzxl1993 mentioned this pull request Nov 20, 2025
Merged
yiguolei pushed a commit that referenced this pull request Nov 20, 2025
@github-actions @zzzxl1993
branch-4.0: [fix](file cache) Fix inverted index cache not cleaned on…
d51a156 
… compaction failure #58141 (#58165)

Cherry-picked from #58141

Co-authored-by: zzzxl <yangsiyu@selectdb.com>
morrySnow pushed a commit that referenced this pull request Nov 25, 2025
@zzzxl1993
branch-3.1: [fix](file cache) Fix inverted index cache not cleaned on…
66ff7c0 
… compaction failure #58141 (#58196)

picked from #58141
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@zzzxl1993 @nagisa-kunhah
[fix](file cache) Fix inverted index cache not cleaned on compaction …
adfd5d4 
…failure (apache#58141)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

+1 more reviewer

@freemandealer freemandealer freemandealer approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@zzzxl1993 @Thearas @doris-robot @hello-stephen @freemandealer @gavinchou @yiguolei @morrySnow