Skip to content

[Cherry-pick](branch-3.0) Pick "[Feature](Compaction)Improve Compaction Profiling and Logging #50950" #51125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
May 22, 2025
Merged

[Cherry-pick](branch-3.0) Pick "[Feature](Compaction)Improve Compaction Profiling and Logging #50950" #51125

dataroaring merged 2 commits into from
May 22, 2025

Conversation

@Yukang-Lian
Copy link
Collaborator

@Yukang-Lian Yukang-Lian commented May 21, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Pick #50950

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Yukang-Lian and others added 2 commits May 21, 2025 16:54
@Yukang-Lian @Yoruet
[Feature](Compaction)Improve Compaction Profiling and Logging (apache…
2888fe1 
…#50950)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache#42227

Problem Summary:

Issue Number: close apache#41753
Add detailed parameters to compaction status:

add

last cumulative failure time
last base failure time
last full failure time
last cumulative success time
last base success time
last full success time
last cumulative schedule time
last base schedule time
last full schedule time
last cumulative status
last base status
last full status
in cloud_tablet.cpp

and add

last cumulative schedule time
last full schedule time
last cumulative status
last full status
in tablet.cpp

Co-authored-by: yoruet <1559650411@qq.com>
@Yukang-Lian
1
2bae725
@Yukang-Lian Yukang-Lian requested a review from dataroaring as a code owner May 21, 2025 10:52
@Thearas
Copy link
Contributor

Thearas commented May 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Yukang-Lian
Copy link
Collaborator Author

Yukang-Lian commented May 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 21, 2025

TPC-H: Total hot run time: 39944 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2bae7250d66a02286b7ce9140b822d411dbd663e, data reload: false

------ Round 1 ----------------------------------
q1	17605	6971	6661	6661
q2	2058	170	176	170
q3	10642	1084	1163	1084
q4	10546	761	776	761
q5	7779	2907	2873	2873
q6	215	139	134	134
q7	998	631	612	612
q8	9374	1966	2081	1966
q9	6628	6424	6386	6386
q10	7043	2290	2306	2290
q11	478	268	257	257
q12	394	213	217	213
q13	17768	2969	2986	2969
q14	229	213	215	213
q15	525	466	460	460
q16	491	393	381	381
q17	984	601	534	534
q18	7275	6641	6685	6641
q19	1393	1067	977	977
q20	488	203	208	203
q21	4000	3195	3183	3183
q22	1108	994	976	976
Total cold run time: 108021 ms
Total hot run time: 39944 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6614	6577	6572	6572
q2	335	231	224	224
q3	2880	2714	2910	2714
q4	2037	1814	1830	1814
q5	5753	5773	5751	5751
q6	210	129	137	129
q7	2199	1797	1777	1777
q8	3346	3568	3527	3527
q9	8999	8776	8866	8776
q10	3613	3505	3511	3505
q11	595	503	484	484
q12	799	577	601	577
q13	8895	3128	3130	3128
q14	294	276	297	276
q15	527	478	459	459
q16	476	441	431	431
q17	1847	1651	1599	1599
q18	8307	7707	7808	7707
q19	1699	1582	1649	1582
q20	2109	1869	1840	1840
q21	5462	5212	5223	5212
q22	1121	1086	1048	1048
Total cold run time: 68117 ms
Total hot run time: 59132 ms

@doris-robot
Copy link

doris-robot commented May 21, 2025

BE UT Coverage Report

Increment line coverage 2.68% (8/298) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.07% (10861/26443)
Line Coverage 31.88% (92728/290863)
Region Coverage 30.97% (47830/154453)
Branch Coverage 27.43% (24486/89252)

@doris-robot
Copy link

doris-robot commented May 21, 2025

TPC-DS: Total hot run time: 197600 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2bae7250d66a02286b7ce9140b822d411dbd663e, data reload: false

query1	1273	915	879	879
query2	6390	1902	1902	1902
query3	10938	4554	4493	4493
query4	61562	28800	23792	23792
query5	5245	458	439	439
query6	385	177	166	166
query7	5460	325	306	306
query8	314	239	223	223
query9	8390	2597	2595	2595
query10	457	272	256	256
query11	17617	15623	16105	15623
query12	167	106	104	104
query13	1440	436	416	416
query14	9791	7752	7397	7397
query15	202	189	179	179
query16	7493	496	512	496
query17	1153	590	575	575
query18	1837	331	306	306
query19	217	158	153	153
query20	119	113	109	109
query21	205	100	105	100
query22	4620	4349	4663	4349
query23	34585	34048	34133	34048
query24	6136	2930	2905	2905
query25	545	425	417	417
query26	652	168	173	168
query27	1767	355	381	355
query28	3883	2162	2109	2109
query29	707	480	428	428
query30	227	157	153	153
query31	985	819	826	819
query32	69	53	56	53
query33	419	292	308	292
query34	909	506	510	506
query35	843	719	741	719
query36	1082	939	960	939
query37	114	70	72	70
query38	4041	3926	3998	3926
query39	1502	1456	1476	1456
query40	201	96	100	96
query41	52	47	48	47
query42	115	110	101	101
query43	526	481	495	481
query44	1151	815	808	808
query45	187	165	168	165
query46	1171	727	712	712
query47	1979	1875	1916	1875
query48	470	400	379	379
query49	704	413	399	399
query50	841	427	412	412
query51	7465	7172	7404	7172
query52	106	93	94	93
query53	257	187	181	181
query54	550	479	475	475
query55	79	79	76	76
query56	274	243	265	243
query57	1267	1194	1144	1144
query58	235	207	212	207
query59	3160	3116	3081	3081
query60	300	251	256	251
query61	114	112	105	105
query62	790	717	680	680
query63	224	187	187	187
query64	1352	646	619	619
query65	3263	3153	3212	3153
query66	711	327	296	296
query67	15893	15504	15395	15395
query68	4366	571	563	563
query69	423	261	264	261
query70	1160	1089	1089	1089
query71	323	262	247	247
query72	6357	3989	3928	3928
query73	744	342	355	342
query74	10412	9249	9345	9249
query75	3353	2644	2660	2644
query76	2115	1033	1156	1033
query77	477	270	260	260
query78	10571	9552	9596	9552
query79	2202	591	596	591
query80	1408	444	431	431
query81	507	220	222	220
query82	1246	87	87	87
query83	271	138	140	138
query84	279	82	79	79
query85	1035	317	292	292
query86	411	283	300	283
query87	4385	4214	4270	4214
query88	3866	2394	2380	2380
query89	416	293	289	289
query90	1959	186	188	186
query91	179	147	146	146
query92	61	46	49	46
query93	2966	560	541	541
query94	802	302	284	284
query95	364	251	251	251
query96	617	282	283	282
query97	3284	3134	3143	3134
query98	213	206	196	196
query99	1582	1308	1338	1308
Total cold run time: 315532 ms
Total hot run time: 197600 ms

@doris-robot
Copy link

doris-robot commented May 21, 2025

ClickBench: Total hot run time: 29.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2bae7250d66a02286b7ce9140b822d411dbd663e, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.52	0.51	0.50
query6	1.14	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.50	0.50
query10	0.55	0.54	0.54
query11	0.14	0.10	0.11
query12	0.14	0.11	0.10
query13	0.61	0.59	0.60
query14	0.77	0.78	0.82
query15	0.84	0.82	0.82
query16	0.37	0.36	0.38
query17	0.97	1.00	1.02
query18	0.23	0.22	0.23
query19	1.95	1.77	1.84
query20	0.01	0.01	0.01
query21	15.39	0.62	0.59
query22	2.76	2.27	1.45
query23	17.09	0.94	0.86
query24	3.10	1.23	0.46
query25	0.25	0.24	0.08
query26	0.28	0.14	0.13
query27	0.05	0.04	0.04
query28	10.88	0.47	0.46
query29	12.57	3.29	3.30
query30	0.24	0.06	0.07
query31	2.86	0.38	0.39
query32	3.25	0.46	0.46
query33	2.93	3.01	2.98
query34	17.08	4.50	4.50
query35	4.54	4.58	4.58
query36	0.66	0.48	0.47
query37	0.08	0.05	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 105.21 s
Total hot run time: 29.35 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 21, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 21, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 21, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes May 22, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 1992e53 into apache:branch-3.0 May 22, 2025
24 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Yukang-Lian @Thearas @doris-robot @gavinchou @dataroaring