Skip to content

[opt](warm up) support delaying compaction commit until rowset warm-up finishes #54275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zhannngchen wants to merge 2 commits into from
Closed

[opt](warm up) support delaying compaction commit until rowset warm-up finishes #54275

zhannngchen wants to merge 2 commits into from

Conversation

@zhannngchen
Copy link
Contributor

@zhannngchen zhannngchen commented Aug 4, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Aug 4, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhannngchen
Copy link
Contributor Author

zhannngchen commented Aug 4, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.93% (1171/1447)
Line Coverage 65.01% (19963/30709)
Region Coverage 66.53% (10057/15117)
Branch Coverage 56.26% (5317/9450)

@zhannngchen
Copy link
Contributor Author

zhannngchen commented Aug 4, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.93% (1171/1447)
Line Coverage 65.00% (19960/30709)
Region Coverage 66.51% (10055/15117)
Branch Coverage 56.18% (5309/9450)

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

TPC-H: Total hot run time: 40283 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb79a32c76a31a824c9c1a4911fbd766092d8abf, data reload: false

------ Round 1 ----------------------------------
q1	19242	7388	6806	6806
q2	2085	162	162	162
q3	10677	1350	1222	1222
q4	10437	776	792	776
q5	7750	2896	2816	2816
q6	213	128	129	128
q7	985	603	598	598
q8	9368	1987	2089	1987
q9	6778	6406	6457	6406
q10	6974	2337	2308	2308
q11	451	255	251	251
q12	408	210	220	210
q13	17818	2991	3060	2991
q14	249	210	214	210
q15	522	472	455	455
q16	489	384	374	374
q17	987	578	542	542
q18	7349	6753	6653	6653
q19	1457	1136	1118	1118
q20	473	203	195	195
q21	3911	3203	3089	3089
q22	1115	986	1014	986
Total cold run time: 109738 ms
Total hot run time: 40283 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6665	6606	6603	6603
q2	334	252	248	248
q3	3163	2974	2973	2973
q4	2036	1852	1828	1828
q5	5698	5704	5699	5699
q6	206	124	122	122
q7	2194	1834	1742	1742
q8	3388	3507	3522	3507
q9	8900	8847	8928	8847
q10	3572	3525	3533	3525
q11	603	487	497	487
q12	795	611	609	609
q13	8246	3143	3126	3126
q14	297	279	265	265
q15	502	451	460	451
q16	508	449	432	432
q17	1844	1607	1597	1597
q18	8301	7617	7817	7617
q19	1726	1490	1562	1490
q20	2100	1896	1863	1863
q21	5273	4950	4825	4825
q22	1082	1006	989	989
Total cold run time: 67433 ms
Total hot run time: 58845 ms

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/32) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.89% (11135/26582)
Line Coverage 32.40% (95346/294278)
Region Coverage 31.54% (49258/156152)
Branch Coverage 27.99% (25247/90208)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@zhannngchen
Copy link
Contributor Author

zhannngchen commented Aug 4, 2025

not merge to 3.0, will merge to 3.1, see #54284

@zhannngchen zhannngchen closed this Aug 4, 2025
@doris-robot
Copy link

doris-robot commented Aug 4, 2025

TPC-DS: Total hot run time: 189748 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bb79a32c76a31a824c9c1a4911fbd766092d8abf, data reload: false

query1	973	372	359	359
query2	6540	1928	1859	1859
query3	6697	222	223	222
query4	34437	23601	23912	23601
query5	4298	457	439	439
query6	253	166	174	166
query7	4634	314	313	313
query8	286	224	221	221
query9	9494	2565	2576	2565
query10	495	261	263	261
query11	18213	15684	15260	15260
query12	158	103	105	103
query13	1679	442	429	429
query14	10019	7504	6660	6660
query15	264	175	181	175
query16	8121	459	511	459
query17	1627	574	564	564
query18	2154	318	332	318
query19	391	174	161	161
query20	114	103	106	103
query21	205	106	104	104
query22	4311	4176	4294	4176
query23	34539	33318	33892	33318
query24	11290	2844	2930	2844
query25	693	409	409	409
query26	1463	175	173	173
query27	2881	356	337	337
query28	7765	2107	2121	2107
query29	930	466	445	445
query30	330	157	162	157
query31	1054	796	801	796
query32	103	62	58	58
query33	799	311	304	304
query34	934	521	522	521
query35	858	729	717	717
query36	1090	933	939	933
query37	128	70	66	66
query38	3905	3820	3832	3820
query39	1480	1427	1417	1417
query40	290	103	101	101
query41	57	56	51	51
query42	113	106	102	102
query43	523	482	497	482
query44	1283	791	796	791
query45	190	169	170	169
query46	1147	734	728	728
query47	1898	1785	1817	1785
query48	480	409	392	392
query49	1208	418	414	414
query50	807	425	424	424
query51	7277	7103	7211	7103
query52	107	95	95	95
query53	268	190	199	190
query54	1296	469	464	464
query55	78	77	79	77
query56	270	240	243	240
query57	1259	1131	1159	1131
query58	235	207	218	207
query59	3057	2897	2734	2734
query60	282	256	260	256
query61	111	106	109	106
query62	891	677	697	677
query63	219	196	196	196
query64	5477	660	640	640
query65	3262	3212	3216	3212
query66	1446	309	297	297
query67	16171	15360	15425	15360
query68	4722	585	574	574
query69	422	270	266	266
query70	1196	1095	1098	1095
query71	346	253	254	253
query72	6423	4018	4128	4018
query73	750	355	372	355
query74	10315	8847	9168	8847
query75	3332	2622	2651	2622
query76	2916	1076	1093	1076
query77	434	282	269	269
query78	10496	9579	9531	9531
query79	1540	590	609	590
query80	1174	441	424	424
query81	526	222	222	222
query82	948	92	85	85
query83	218	148	148	148
query84	239	83	77	77
query85	1296	303	282	282
query86	367	312	303	303
query87	4487	4241	4277	4241
query88	3541	2430	2403	2403
query89	413	299	297	297
query90	1973	190	189	189
query91	179	146	153	146
query92	60	48	55	48
query93	1218	571	553	553
query94	951	304	303	303
query95	359	259	251	251
query96	606	296	287	287
query97	3291	3142	3139	3139
query98	212	202	195	195
query99	1503	1318	1295	1295
Total cold run time: 301760 ms
Total hot run time: 189748 ms

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

ClickBench: Total hot run time: 29.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bb79a32c76a31a824c9c1a4911fbd766092d8abf, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.11	0.11
query5	0.52	0.50	0.51
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.55	0.54	0.55
query11	0.15	0.10	0.11
query12	0.15	0.12	0.11
query13	0.61	0.60	0.61
query14	0.79	0.78	0.81
query15	0.86	0.82	0.82
query16	0.39	0.36	0.38
query17	1.06	1.07	1.03
query18	0.22	0.21	0.21
query19	1.92	1.86	1.86
query20	0.02	0.01	0.01
query21	15.37	0.58	0.57
query22	2.31	3.01	1.89
query23	16.88	0.87	0.81
query24	2.92	1.36	0.68
query25	0.18	0.27	0.08
query26	0.42	0.14	0.14
query27	0.04	0.04	0.04
query28	10.67	0.48	0.51
query29	12.59	3.19	3.17
query30	0.25	0.06	0.07
query31	2.82	0.38	0.38
query32	3.26	0.45	0.45
query33	2.99	2.98	3.00
query34	16.92	4.54	4.50
query35	4.62	4.60	4.55
query36	0.69	0.48	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.03	0.02
query40	0.15	0.14	0.13
query41	0.09	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 104.34 s
Total hot run time: 29.96 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring Awaiting requested review from dataroaring

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zhannngchen @Thearas @hello-stephen @doris-robot