Skip to content

[opt](warm up) support delaying compaction commit until rowset warm-up finishes #54416

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zhannngchen merged 3 commits into from
Aug 7, 2025
Merged

[opt](warm up) support delaying compaction commit until rowset warm-up finishes #54416

zhannngchen merged 3 commits into from
Aug 7, 2025

Conversation

@zhannngchen
Copy link
Contributor

@zhannngchen zhannngchen commented Aug 6, 2025 •
edited
Loading

What problem does this PR solve?

Problem Summary:

This PR introduces an optimization to the event driven warm-up mechanism by implementing delayed compaction commit functionality. The change ensures that compaction operations wait for rowset warm-up processes to complete before finalizing the commit, improving file-cache hit ratio on read cluster.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 6, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhannngchen zhannngchen mentioned this pull request Aug 7, 2025
Merged
@zhannngchen
Copy link
Contributor Author

zhannngchen commented Aug 7, 2025

run buildall

1 similar comment
@zhannngchen
Copy link
Contributor Author

zhannngchen commented Aug 7, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 7, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.79% (1406/1719)
Line Coverage 66.07% (23980/36297)
Region Coverage 67.32% (11911/17693)
Branch Coverage 56.88% (6230/10952)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 7, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@zhannngchen
Copy link
Contributor Author

zhannngchen commented Aug 7, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 7, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.79% (1406/1719)
Line Coverage 66.09% (23990/36297)
Region Coverage 67.31% (11909/17693)
Branch Coverage 56.96% (6238/10952)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 7, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@zhannngchen
Copy link
Contributor Author

zhannngchen commented Aug 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 7, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.79% (1406/1719)
Line Coverage 66.07% (23982/36297)
Region Coverage 67.33% (11913/17693)
Branch Coverage 56.92% (6234/10952)

@doris-robot
Copy link

doris-robot commented Aug 7, 2025

TPC-H: Total hot run time: 33906 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5a3d0210c93c6c4e12b57400eef72cc8c117c000, data reload: false

------ Round 1 ----------------------------------
q1	17327	5498	5235	5235
q2	1934	289	201	201
q3	10137	1300	682	682
q4	10208	1114	537	537
q5	7639	2416	2315	2315
q6	180	166	133	133
q7	902	749	633	633
q8	9294	1287	1111	1111
q9	7341	5222	5244	5222
q10	6928	2360	1995	1995
q11	467	285	269	269
q12	348	366	224	224
q13	17764	3460	2956	2956
q14	238	243	226	226
q15	529	453	465	453
q16	435	426	372	372
q17	578	837	344	344
q18	7606	6984	7026	6984
q19	1101	1004	561	561
q20	325	303	209	209
q21	3495	2443	2228	2228
q22	1065	1086	1016	1016
Total cold run time: 105841 ms
Total hot run time: 33906 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5361	5443	5354	5354
q2	239	313	215	215
q3	2072	2550	2194	2194
q4	1331	1724	1327	1327
q5	4119	4304	4195	4195
q6	225	193	142	142
q7	2261	1921	1753	1753
q8	2543	2565	2512	2512
q9	7253	7265	7351	7265
q10	3218	3279	2978	2978
q11	554	510	487	487
q12	705	793	642	642
q13	3447	3692	3378	3378
q14	289	304	274	274
q15	500	439	466	439
q16	439	497	442	442
q17	1185	1358	1265	1265
q18	14754	7871	7681	7681
q19	7692	1126	1133	1126
q20	2061	2101	1985	1985
q21	15353	4319	4286	4286
q22	1110	1055	992	992
Total cold run time: 76711 ms
Total hot run time: 50932 ms

@doris-robot
Copy link

doris-robot commented Aug 7, 2025

TPC-DS: Total hot run time: 170958 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5a3d0210c93c6c4e12b57400eef72cc8c117c000, data reload: false

============================================
query1	994	395	469	395
query2	6541	1963	1717	1717
query3	6748	227	219	219
query4	27307	23317	22988	22988
query5	4346	623	561	561
query6	333	238	217	217
query7	4653	538	301	301
query8	293	257	244	244
query9	8561	2973	3001	2973
query10	457	367	299	299
query11	16019	14952	14777	14777
query12	185	139	143	139
query13	1686	560	418	418
query14	8647	5822	5976	5822
query15	213	199	170	170
query16	7129	651	444	444
query17	980	820	663	663
query18	1999	450	331	331
query19	244	220	193	193
query20	152	151	137	137
query21	221	134	114	114
query22	4023	4095	3920	3920
query23	34591	34471	34520	34471
query24	5251	2410	2454	2410
query25	484	521	438	438
query26	726	293	168	168
query27	2268	507	364	364
query28	3046	2356	2335	2335
query29	669	663	501	501
query30	293	235	198	198
query31	867	788	712	712
query32	91	77	81	77
query33	521	428	376	376
query34	832	843	540	540
query35	846	862	778	778
query36	1030	1027	932	932
query37	134	115	93	93
query38	3934	3977	3967	3967
query39	1464	1385	1364	1364
query40	248	149	135	135
query41	65	59	57	57
query42	142	132	135	132
query43	516	517	490	490
query44	1478	890	889	889
query45	205	206	188	188
query46	952	1073	698	698
query47	1803	1791	1744	1744
query48	406	449	342	342
query49	689	516	438	438
query50	698	729	420	420
query51	4185	4247	4188	4188
query52	134	142	123	123
query53	272	299	223	223
query54	661	652	573	573
query55	92	92	87	87
query56	365	364	363	363
query57	1207	1212	1126	1126
query58	345	349	331	331
query59	2615	2644	2612	2612
query60	411	401	397	397
query61	130	154	126	126
query62	757	724	666	666
query63	262	227	227	227
query64	2343	1102	823	823
query65	4236	4120	4119	4119
query66	1033	459	340	340
query67	query68	17683	1035	1073	1035
query69	1031	286	293	286
query70	1403	1204	1150	1150
query71	718	355	324	324
query72	9245	2345	4774	2345
query73	3391	785	363	363
query74	8954	8919	8560	8560
query75	7558	3131	2662	2662
query76	8846	1232	813	813
query77	1139	434	336	336
query78	9594	10248	9210	9210
query79	15127	659	583	583
query80	1889	578	505	505
query81	550	271	232	232
query82	488	151	118	118
query83	359	292	283	283
query84	296	101	79	79
query85	1055	376	345	345
query86	369	335	291	291
query87	4274	4219	4139	4139
query88	5467	2224	2190	2190
query89	490	364	328	328
query90	2576	231	247	231
query91	151	147	112	112
query92	87	73	69	69
query93	6362	978	658	658
query94	1125	398	281	281
query95	435	345	328	328
query96	504	594	284	284
query97	2698	2731	2599	2599
query98	250	248	226	226
query99	1460	1421	1370	1370
Total cold run time: 300602 ms
Total hot run time: 170958 ms

@doris-robot
Copy link

doris-robot commented Aug 7, 2025

ClickBench: Total hot run time: 34.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5a3d0210c93c6c4e12b57400eef72cc8c117c000, data reload: false

query1	0.04	0.03	0.04
query2	0.11	0.06	0.08
query3	0.30	0.07	0.06
query4	1.60	0.09	0.09
query5	0.42	0.42	0.42
query6	1.18	0.67	0.67
query7	0.02	0.01	0.02
query8	0.07	0.06	0.06
query9	0.59	0.49	0.50
query10	0.54	0.55	0.55
query11	0.25	0.13	0.12
query12	0.25	0.12	0.15
query13	0.68	0.69	0.68
query14	0.99	1.17	1.06
query15	1.02	0.90	0.93
query16	0.41	0.40	0.39
query17	1.06	1.08	1.12
query18	0.25	0.23	0.24
query19	2.08	1.87	1.98
query20	0.02	0.01	0.01
query21	15.42	0.96	0.71
query22	0.97	1.10	0.91
query23	14.70	1.52	0.92
query24	4.85	0.63	0.38
query25	0.19	0.11	0.10
query26	0.56	0.22	0.20
query27	0.11	0.11	0.11
query28	11.09	1.14	0.64
query29	12.59	4.03	3.51
query30	3.11	2.99	2.99
query31	2.85	0.63	0.48
query32	3.24	0.66	0.57
query33	3.08	3.22	3.29
query34	16.74	5.43	4.85
query35	4.88	5.05	5.07
query36	0.74	0.55	0.53
query37	0.25	0.22	0.23
query38	0.24	0.23	0.23
query39	0.07	0.06	0.06
query40	0.21	0.17	0.17
query41	0.12	0.08	0.07
query42	0.09	0.07	0.08
query43	0.07	0.06	0.07
Total cold run time: 108.05 s
Total hot run time: 34.95 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 7, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/44) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 58.83% (16539/28111)
Line Coverage 47.77% (150104/314204)
Region Coverage 36.72% (112719/306948)
Branch Coverage 39.66% (50053/126215)

@github-actions
Copy link
Contributor

github-actions bot commented Aug 7, 2025

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 7, 2025

BE Regression && UT Coverage Report

Increment line coverage 19.44% (14/72) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.35% (22445/27590)
Line Coverage 74.06% (232690/314187)
Region Coverage 61.32% (193290/315206)
Branch Coverage 65.47% (83641/127763)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 7, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Aug 7, 2025

PR approved by at least one committer and no changes requested.

@zhannngchen zhannngchen merged commit 290dc2a into apache:master Aug 7, 2025
24 of 26 checks passed
morrySnow pushed a commit that referenced this pull request Aug 7, 2025
branch-3.1: [opt](warm up) support delaying compaction commit until r…
3ddfa55 
…owset warm-up finishes #54416 (#54284)

cherry-pick #54416
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

+1 more reviewer

@bobhan1 bobhan1 bobhan1 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zhannngchen @hello-stephen @doris-robot @gavinchou @bobhan1 @morrySnow