[Enhancement] Improve Compaction Profiling and Logging-task2

[Yoruet]

Proposed changes

Issue Number: close #41753
Add detailed parameters to compaction status:

add

• last cumulative failure time
• last base failure time
• last full failure time
• last cumulative success time
• last base success time
• last full success time
• last cumulative schedule time
• last base schedule time
• last full schedule time
• last cumulative status
• last base status
• last full status

in cloud_tablet.cpp

and add

• last cumulative schedule time
• last full schedule time
• last cumulative status
• last full status

in tablet.cpp

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[Yukang-Lian]

plz add some regression tests

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[Yoruet]

run buildall

[Yukang-Lian]

run buildall

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.43% (9715/25952)
Line Coverage: 28.72% (80595/280660)
Region Coverage: 28.15% (41655/147964)
Branch Coverage: 24.72% (21162/85598)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e778805e3b662359295431eac7ec4f5a2c3a1a22_e778805e3b662359295431eac7ec4f5a2c3a1a22/report/index.html

[doris-robot]

TPC-H: Total hot run time: 41424 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e778805e3b662359295431eac7ec4f5a2c3a1a22, data reload: false

------ Round 1 ----------------------------------
q1	17693	7542	7319	7319
q2	2023	290	279	279
q3	12170	1074	1155	1074
q4	10585	875	846	846
q5	7730	3123	3034	3034
q6	235	145	144	144
q7	1031	615	598	598
q8	9344	1968	1988	1968
q9	6677	6481	6422	6422
q10	7064	2447	2450	2447
q11	446	249	253	249
q12	405	213	208	208
q13	17769	3013	2975	2975
q14	238	209	206	206
q15	567	515	511	511
q16	649	588	583	583
q17	973	588	536	536
q18	7337	6798	6813	6798
q19	1336	1054	992	992
q20	482	183	183	183
q21	3993	3096	3036	3036
q22	1132	1016	1040	1016
Total cold run time: 109879 ms
Total hot run time: 41424 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7282	7215	7274	7215
q2	334	239	228	228
q3	3059	2959	2939	2939
q4	2074	1817	1805	1805
q5	5731	5752	5758	5752
q6	229	141	142	141
q7	2239	1827	1772	1772
q8	3317	3560	3425	3425
q9	8970	8925	8897	8897
q10	3579	3557	3549	3549
q11	579	482	482	482
q12	823	641	586	586
q13	9567	3166	3178	3166
q14	298	271	273	271
q15	575	517	519	517
q16	689	661	662	661
q17	1848	1627	1608	1608
q18	8370	7760	7553	7553
q19	1722	1635	1534	1534
q20	2171	1878	1842	1842
q21	5607	5361	5320	5320
q22	1181	1094	1043	1043
Total cold run time: 70244 ms
Total hot run time: 60306 ms

[doris-robot]

TPC-DS: Total hot run time: 191918 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e778805e3b662359295431eac7ec4f5a2c3a1a22, data reload: false

query1	850	403	393	393
query2	6282	2091	2032	2032
query3	8678	206	212	206
query4	35457	23683	23648	23648
query5	4872	443	446	443
query6	274	168	167	167
query7	4200	301	293	293
query8	291	223	244	223
query9	9485	2654	2648	2648
query10	493	263	253	253
query11	18189	15160	15271	15160
query12	160	107	103	103
query13	1571	435	414	414
query14	10649	7009	7012	7009
query15	246	175	182	175
query16	7993	480	478	478
query17	1471	587	597	587
query18	2195	308	318	308
query19	255	157	167	157
query20	129	112	120	112
query21	203	105	109	105
query22	4685	4422	4688	4422
query23	35012	34457	34144	34144
query24	10992	2781	2748	2748
query25	571	406	401	401
query26	803	160	163	160
query27	2213	291	290	290
query28	7068	2430	2437	2430
query29	691	439	426	426
query30	268	167	161	161
query31	1023	803	787	787
query32	97	56	59	56
query33	774	283	270	270
query34	906	520	524	520
query35	1035	897	874	874
query36	1094	937	940	937
query37	120	71	75	71
query38	4425	4244	4184	4184
query39	1476	1414	1429	1414
query40	201	102	100	100
query41	49	46	47	46
query42	110	99	99	99
query43	529	487	488	487
query44	1277	811	808	808
query45	186	165	166	165
query46	1144	703	710	703
query47	1930	1825	1823	1823
query48	427	323	324	323
query49	915	409	414	409
query50	808	403	409	403
query51	7041	6972	6918	6918
query52	99	92	86	86
query53	260	182	181	181
query54	1183	401	404	401
query55	80	76	77	76
query56	259	243	249	243
query57	1331	1133	1157	1133
query58	210	190	203	190
query59	3161	2957	3046	2957
query60	276	246	251	246
query61	106	100	104	100
query62	849	678	660	660
query63	221	188	186	186
query64	4018	652	600	600
query65	3322	3203	3228	3203
query66	803	328	303	303
query67	15969	15676	15639	15639
query68	4483	576	565	565
query69	439	253	256	253
query70	1231	1156	1171	1156
query71	399	267	263	263
query72	6133	4023	4041	4023
query73	787	365	361	361
query74	10230	9020	9019	9019
query75	3398	2658	2694	2658
query76	2609	1009	1071	1009
query77	405	279	272	272
query78	10525	9622	9548	9548
query79	1724	587	612	587
query80	1027	425	430	425
query81	564	243	248	243
query82	632	115	112	112
query83	238	139	142	139
query84	242	79	69	69
query85	1297	289	285	285
query86	435	298	298	298
query87	4769	4638	4610	4610
query88	3391	2221	2175	2175
query89	403	282	288	282
query90	2029	186	183	183
query91	136	102	103	102
query92	61	48	46	46
query93	2461	547	542	542
query94	922	305	261	261
query95	348	247	264	247
query96	644	288	291	288
query97	2903	2695	2738	2695
query98	214	194	200	194
query99	1560	1311	1329	1311
Total cold run time: 300599 ms
Total hot run time: 191918 ms

[doris-robot]

ClickBench: Total hot run time: 33.27 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e778805e3b662359295431eac7ec4f5a2c3a1a22, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.43	0.40	0.38
query6	1.18	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.50	0.50
query10	0.53	0.54	0.55
query11	0.15	0.12	0.11
query12	0.14	0.10	0.11
query13	0.61	0.60	0.58
query14	2.82	2.72	2.75
query15	0.92	0.82	0.83
query16	0.38	0.40	0.37
query17	0.97	1.03	1.00
query18	0.20	0.20	0.20
query19	1.96	1.90	1.83
query20	0.01	0.01	0.01
query21	15.36	0.60	0.58
query22	2.71	2.40	2.45
query23	16.97	0.94	0.82
query24	3.69	1.06	1.25
query25	0.25	0.05	0.04
query26	0.66	0.13	0.14
query27	0.04	0.04	0.04
query28	10.02	1.09	1.06
query29	12.62	3.41	3.39
query30	0.25	0.06	0.06
query31	2.85	0.37	0.38
query32	3.26	0.46	0.46
query33	3.10	2.99	3.05
query34	17.01	4.50	4.51
query35	4.55	4.53	4.50
query36	0.66	0.49	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.39 s
Total hot run time: 33.27 s

[Yoruet]

run buildall

[doris-robot]

TPC-DS: Total hot run time: 192472 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 83623c41b480c208f3ec29151d1f8e0498c16a8a, data reload: false

query1	814	397	403	397
query2	6258	2157	2111	2111
query3	8686	199	210	199
query4	34871	23920	23541	23541
query5	4696	457	431	431
query6	277	170	167	167
query7	4198	299	284	284
query8	299	245	243	243
query9	9315	2748	2726	2726
query10	501	253	247	247
query11	18227	15306	15192	15192
query12	153	104	97	97
query13	1567	417	426	417
query14	10443	7304	7331	7304
query15	254	171	189	171
query16	7964	476	493	476
query17	1438	586	573	573
query18	2173	314	317	314
query19	370	160	160	160
query20	124	121	118	118
query21	206	120	113	113
query22	4814	4770	4591	4591
query23	35095	34322	34101	34101
query24	10996	2761	2755	2755
query25	617	414	397	397
query26	1161	161	163	161
query27	2072	280	289	280
query28	6749	2484	2472	2472
query29	832	424	420	420
query30	264	159	161	159
query31	1054	801	779	779
query32	94	55	57	55
query33	780	273	290	273
query34	919	518	519	518
query35	1015	890	905	890
query36	1116	960	952	952
query37	121	76	72	72
query38	4299	4207	4186	4186
query39	1502	1433	1384	1384
query40	212	101	101	101
query41	51	47	46	46
query42	112	98	96	96
query43	512	497	507	497
query44	1292	819	822	819
query45	181	169	167	167
query46	1247	697	683	683
query47	1922	1874	1826	1826
query48	411	328	328	328
query49	878	416	403	403
query50	799	409	393	393
query51	7248	6953	7047	6953
query52	99	92	90	90
query53	255	184	179	179
query54	1188	412	416	412
query55	78	84	78	78
query56	294	230	233	230
query57	1306	1151	1152	1151
query58	221	197	197	197
query59	3212	3220	2918	2918
query60	274	245	263	245
query61	115	100	107	100
query62	855	676	670	670
query63	212	188	180	180
query64	3902	624	604	604
query65	3267	3213	3217	3213
query66	818	300	307	300
query67	15987	15836	15950	15836
query68	4326	561	567	561
query69	437	258	260	258
query70	1238	1168	1094	1094
query71	338	276	264	264
query72	6301	4004	3992	3992
query73	766	360	359	359
query74	10173	9060	9091	9060
query75	3424	2612	2660	2612
query76	2785	1021	986	986
query77	400	266	287	266
query78	10596	9576	9464	9464
query79	1143	593	601	593
query80	890	425	444	425
query81	579	235	240	235
query82	364	120	115	115
query83	238	140	138	138
query84	244	74	71	71
query85	1222	293	285	285
query86	425	297	306	297
query87	4776	4706	4584	4584
query88	3275	2228	2271	2228
query89	402	286	288	286
query90	2086	187	189	187
query91	136	103	100	100
query92	60	46	46	46
query93	2186	541	553	541
query94	916	295	274	274
query95	353	246	245	245
query96	626	279	291	279
query97	2882	2706	2703	2703
query98	220	196	194	194
query99	1643	1300	1289	1289
Total cold run time: 298732 ms
Total hot run time: 192472 ms

[doris-robot]

ClickBench: Total hot run time: 33.01 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 83623c41b480c208f3ec29151d1f8e0498c16a8a, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.22	0.06	0.06
query4	1.65	0.11	0.11
query5	0.41	0.39	0.41
query6	1.18	0.64	0.68
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.55	0.50	0.50
query10	0.55	0.55	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	2.69	2.83	2.76
query15	0.89	0.82	0.84
query16	0.38	0.39	0.40
query17	1.07	1.10	1.03
query18	0.19	0.20	0.19
query19	1.96	1.88	2.02
query20	0.01	0.02	0.01
query21	15.35	0.60	0.58
query22	2.84	2.75	2.29
query23	16.83	1.07	0.81
query24	3.21	0.92	1.28
query25	0.34	0.13	0.09
query26	0.43	0.14	0.13
query27	0.04	0.03	0.04
query28	10.57	1.09	1.08
query29	12.58	3.30	3.22
query30	0.25	0.06	0.06
query31	2.87	0.39	0.40
query32	3.27	0.47	0.46
query33	2.99	3.05	3.02
query34	17.11	4.50	4.46
query35	4.44	4.50	4.48
query36	0.67	0.49	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 107.02 s
Total hot run time: 33.01 s

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[Yoruet]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[Yoruet]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.44% (9722/25967)
Line Coverage: 28.70% (80631/280941)
Region Coverage: 28.15% (41688/148116)
Branch Coverage: 24.71% (21177/85704)
Coverage Report: http://coverage.selectdb-in.cc/coverage/869252e04254ab848b86fb6364d44ed6f655d110_869252e04254ab848b86fb6364d44ed6f655d110/report/index.html

[doris-robot]

TPC-H: Total hot run time: 40982 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 869252e04254ab848b86fb6364d44ed6f655d110, data reload: false

------ Round 1 ----------------------------------
q1	17582	7450	7273	7273
q2	2052	157	156	156
q3	10858	1053	1170	1053
q4	10431	870	759	759
q5	7783	3086	3100	3086
q6	235	148	145	145
q7	1044	616	591	591
q8	9364	2001	2041	2001
q9	6570	6392	6434	6392
q10	7115	2419	2436	2419
q11	465	231	239	231
q12	415	224	210	210
q13	17782	3011	3004	3004
q14	241	208	220	208
q15	556	522	514	514
q16	673	577	606	577
q17	979	640	503	503
q18	7300	6687	6662	6662
q19	1331	1049	1008	1008
q20	486	179	183	179
q21	4381	3011	3234	3011
q22	1136	1021	1000	1000
Total cold run time: 108779 ms
Total hot run time: 40982 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7269	7234	7239	7234
q2	341	233	232	232
q3	2996	2966	2918	2918
q4	2061	1882	1781	1781
q5	5728	5715	5815	5715
q6	227	143	142	142
q7	2258	1838	1796	1796
q8	3408	3551	3431	3431
q9	8851	8891	8815	8815
q10	3572	3602	3571	3571
q11	594	491	497	491
q12	849	686	630	630
q13	10933	3154	3177	3154
q14	292	278	292	278
q15	577	528	516	516
q16	697	659	662	659
q17	1844	1625	1641	1625
q18	8311	7834	7524	7524
q19	1696	1491	1627	1491
q20	2111	1858	1873	1858
q21	5552	5455	5406	5406
q22	1106	1045	1018	1018
Total cold run time: 71273 ms
Total hot run time: 60285 ms

[doris-robot]

TPC-DS: Total hot run time: 196980 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 869252e04254ab848b86fb6364d44ed6f655d110, data reload: false

query1	1259	1030	965	965
query2	6245	2074	2020	2020
query3	11498	4768	4830	4768
query4	33882	23629	23679	23629
query5	4918	441	424	424
query6	263	173	174	173
query7	3999	294	285	285
query8	289	223	215	215
query9	9276	2614	2613	2613
query10	498	248	260	248
query11	18169	15205	15372	15205
query12	145	101	98	98
query13	1552	429	393	393
query14	9875	7152	7628	7152
query15	256	173	175	173
query16	8040	496	489	489
query17	1368	561	547	547
query18	2135	291	287	287
query19	282	147	149	147
query20	121	111	108	108
query21	197	113	103	103
query22	5001	4606	4472	4472
query23	35035	34321	34194	34194
query24	11004	2775	2730	2730
query25	633	408	418	408
query26	1158	155	155	155
query27	2434	280	288	280
query28	7504	2383	2384	2383
query29	824	411	418	411
query30	260	160	161	160
query31	1041	817	823	817
query32	91	54	55	54
query33	756	285	273	273
query34	983	518	520	518
query35	1047	868	873	868
query36	1103	940	948	940
query37	135	78	74	74
query38	4452	4372	4240	4240
query39	1488	1450	1396	1396
query40	204	96	98	96
query41	47	46	47	46
query42	104	97	98	97
query43	529	478	483	478
query44	1236	795	799	795
query45	180	164	169	164
query46	1136	706	703	703
query47	1976	1868	1853	1853
query48	441	328	323	323
query49	920	390	407	390
query50	810	373	389	373
query51	7156	7016	6992	6992
query52	105	89	83	83
query53	255	179	177	177
query54	1169	401	407	401
query55	79	79	80	79
query56	265	255	247	247
query57	1330	1186	1171	1171
query58	223	207	220	207
query59	3194	2889	3062	2889
query60	283	242	260	242
query61	125	119	121	119
query62	836	672	676	672
query63	220	189	180	180
query64	5230	682	589	589
query65	3259	3163	3212	3163
query66	1343	311	295	295
query67	16115	15717	15942	15717
query68	4526	556	545	545
query69	434	253	253	253
query70	1198	1069	1125	1069
query71	367	247	254	247
query72	6492	3978	3988	3978
query73	767	350	357	350
query74	10369	9019	9018	9018
query75	3439	2660	2656	2656
query76	2602	1030	1045	1030
query77	407	283	286	283
query78	10546	9613	9632	9613
query79	1258	581	593	581
query80	727	424	420	420
query81	541	239	233	233
query82	1326	115	117	115
query83	209	136	134	134
query84	241	66	68	66
query85	1131	288	272	272
query86	310	306	300	300
query87	4858	4661	4590	4590
query88	3257	2193	2157	2157
query89	416	284	287	284
query90	1894	179	178	178
query91	139	105	102	102
query92	54	49	47	47
query93	1164	525	520	520
query94	756	277	291	277
query95	339	242	246	242
query96	611	277	279	277
query97	2950	2703	2726	2703
query98	208	197	202	197
query99	1725	1320	1287	1287
Total cold run time: 303247 ms
Total hot run time: 196980 ms

[doris-robot]

ClickBench: Total hot run time: 32.61 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 869252e04254ab848b86fb6364d44ed6f655d110, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.06
query4	1.64	0.11	0.10
query5	0.41	0.42	0.38
query6	1.17	0.66	0.64
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.57	0.51	0.51
query10	0.55	0.55	0.54
query11	0.13	0.12	0.11
query12	0.14	0.12	0.11
query13	0.60	0.60	0.60
query14	2.71	2.79	2.74
query15	0.90	0.82	0.82
query16	0.38	0.37	0.38
query17	1.05	1.08	1.05
query18	0.20	0.20	0.20
query19	1.99	1.75	1.94
query20	0.01	0.02	0.01
query21	15.37	0.58	0.57
query22	2.44	2.50	1.84
query23	16.90	1.18	0.98
query24	3.12	2.18	0.80
query25	0.21	0.22	0.08
query26	0.52	0.15	0.14
query27	0.04	0.04	0.05
query28	9.83	1.09	1.07
query29	12.59	3.39	3.39
query30	0.25	0.06	0.06
query31	2.86	0.37	0.38
query32	3.29	0.46	0.45
query33	3.02	3.04	3.05
query34	17.02	4.52	4.44
query35	4.51	4.49	4.49
query36	0.67	0.47	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.01
query43	0.04	0.03	0.03
Total cold run time: 105.98 s
Total hot run time: 32.61 s

[github-actions]

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!