branch-3.1: [opt](warm up) support delaying compaction commit until rowset warm-up finishes #54416 #54284

zhannngchen · 2025-08-04T08:49:49Z

cherry-pick #54416

hello-stephen · 2025-08-04T08:49:54Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zhannngchen · 2025-08-04T08:50:31Z

run buildall

doris-robot · 2025-08-04T09:00:41Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.10% (1210/1492)
Line Coverage	65.33% (21348/32678)
Region Coverage	66.78% (10725/16059)
Branch Coverage	56.52% (5651/9998)

doris-robot · 2025-08-04T10:07:31Z

TPC-H: Total hot run time: 32897 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3e655eb8e587a641c58b722dff90951bf0ab14d, data reload: false

------ Round 1 ----------------------------------
q1	17801	5620	5414	5414
q2	2078	290	172	172
q3	10603	1276	785	785
q4	10260	895	463	463
q5	8576	2416	2176	2176
q6	187	167	135	135
q7	919	754	613	613
q8	9327	1483	1241	1241
q9	5277	4996	4894	4894
q10	6804	2316	1837	1837
q11	470	303	267	267
q12	341	365	214	214
q13	17759	3694	3033	3033
q14	239	229	207	207
q15	535	480	460	460
q16	430	447	384	384
q17	600	890	366	366
q18	7030	6423	6511	6423
q19	1569	980	574	574
q20	341	348	212	212
q21	2839	2267	2041	2041
q22	1090	1056	986	986
Total cold run time: 105075 ms
Total hot run time: 32897 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5673	5612	5622	5612
q2	233	332	227	227
q3	2272	2625	2331	2331
q4	1404	1827	1422	1422
q5	4480	5007	5065	5007
q6	177	178	131	131
q7	2085	2057	1826	1826
q8	2664	2844	2720	2720
q9	7367	7321	7247	7247
q10	3064	3356	2730	2730
q11	568	509	498	498
q12	669	760	638	638
q13	3513	3844	3219	3219
q14	293	308	275	275
q15	528	476	469	469
q16	442	486	459	459
q17	1246	1736	1243	1243
q18	7666	7645	7434	7434
q19	871	1177	1113	1113
q20	2035	2059	1934	1934
q21	5365	4957	4754	4754
q22	1115	1069	1009	1009
Total cold run time: 53730 ms
Total hot run time: 52298 ms

hello-stephen · 2025-08-04T10:15:33Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-08-04T10:19:52Z

TPC-DS: Total hot run time: 197038 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3e655eb8e587a641c58b722dff90951bf0ab14d, data reload: false

query1	1319	938	914	914
query2	6226	1884	1839	1839
query3	10916	4460	4750	4460
query4	33079	23810	23883	23810
query5	4374	627	461	461
query6	301	210	178	178
query7	3982	500	320	320
query8	290	239	225	225
query9	9569	2625	2627	2625
query10	488	332	256	256
query11	18044	15426	15261	15261
query12	157	107	109	107
query13	1559	542	448	448
query14	10098	6768	7527	6768
query15	228	203	183	183
query16	7954	644	513	513
query17	1640	792	598	598
query18	2086	421	329	329
query19	223	190	173	173
query20	126	121	115	115
query21	214	133	109	109
query22	4667	4563	4458	4458
query23	35327	34331	34167	34167
query24	7425	2714	2756	2714
query25	504	515	436	436
query26	828	290	176	176
query27	2127	491	352	352
query28	5572	2276	2238	2238
query29	641	607	489	489
query30	241	196	167	167
query31	997	933	844	844
query32	69	57	59	57
query33	487	367	310	310
query34	732	884	542	542
query35	790	835	745	745
query36	1003	1045	961	961
query37	102	94	70	70
query38	4037	4054	4008	4008
query39	1577	1465	1550	1465
query40	209	126	106	106
query41	52	53	48	48
query42	127	110	105	105
query43	507	525	479	479
query44	1327	830	831	830
query45	193	178	172	172
query46	899	1080	678	678
query47	2048	2050	1971	1971
query48	411	443	372	372
query49	728	492	433	433
query50	700	713	426	426
query51	7445	7314	7232	7232
query52	106	99	93	93
query53	227	258	189	189
query54	552	537	497	497
query55	81	79	83	79
query56	267	274	264	264
query57	1316	1286	1213	1213
query58	249	216	220	216
query59	3052	3118	3065	3065
query60	303	284	275	275
query61	125	120	116	116
query62	808	773	680	680
query63	235	202	194	194
query64	3262	1011	652	652
query65	3378	3290	3297	3290
query66	792	416	319	319
query67	16153	15793	15587	15587
query68	7334	865	559	559
query69	509	316	275	275
query70	1156	1153	1127	1127
query71	406	300	263	263
query72	5826	2687	3946	2687
query73	643	752	353	353
query74	10808	9414	9230	9230
query75	3218	3156	2674	2674
query76	3105	1177	766	766
query77	478	364	286	286
query78	10401	10365	9681	9681
query79	3181	890	593	593
query80	638	521	447	447
query81	514	264	227	227
query82	644	125	90	90
query83	170	162	143	143
query84	247	100	84	84
query85	807	391	302	302
query86	362	321	300	300
query87	4322	4329	4276	4276
query88	4861	2472	2435	2435
query89	411	336	301	301
query90	1807	196	193	193
query91	137	135	111	111
query92	63	57	54	54
query93	1951	933	561	561
query94	699	421	297	297
query95	338	283	273	273
query96	488	609	288	288
query97	3228	3344	3177	3177
query98	218	214	203	203
query99	1536	1422	1310	1310
Total cold run time: 296057 ms
Total hot run time: 197038 ms

hello-stephen · 2025-08-04T10:23:01Z

BE UT Coverage Report

Increment line coverage 0.00% (0/34) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	45.41% (12643/27842)
Line Coverage	36.27% (112709/310787)
Region Coverage	35.30% (58242/164991)
Branch Coverage	32.49% (31666/97466)

doris-robot · 2025-08-04T10:25:16Z

ClickBench: Total hot run time: 29.27 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c3e655eb8e587a641c58b722dff90951bf0ab14d, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.04	0.04
query3	0.24	0.06	0.05
query4	1.64	0.09	0.08
query5	0.52	0.52	0.51
query6	1.13	0.75	0.74
query7	0.02	0.02	0.02
query8	0.06	0.06	0.05
query9	0.56	0.49	0.50
query10	0.55	0.55	0.56
query11	0.18	0.11	0.12
query12	0.15	0.13	0.13
query13	0.62	0.59	0.60
query14	0.78	0.78	0.81
query15	0.86	0.84	0.84
query16	0.37	0.37	0.38
query17	1.02	0.98	1.07
query18	0.19	0.19	0.21
query19	1.94	1.84	1.88
query20	0.02	0.01	0.02
query21	15.38	0.96	0.66
query22	0.76	0.76	0.69
query23	14.84	1.45	0.69
query24	2.31	0.38	0.23
query25	0.15	0.09	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.39	1.25	0.57
query29	12.63	4.06	3.36
query30	0.25	0.10	0.07
query31	2.83	0.62	0.41
query32	3.24	0.58	0.49
query33	2.98	3.06	3.07
query34	16.35	5.23	4.56
query35	4.60	4.60	4.62
query36	0.63	0.52	0.51
query37	0.20	0.17	0.16
query38	0.17	0.16	0.16
query39	0.05	0.05	0.04
query40	0.16	0.13	0.14
query41	0.09	0.06	0.05
query42	0.06	0.05	0.06
query43	0.05	0.05	0.04
Total cold run time: 102.42 s
Total hot run time: 29.27 s

hello-stephen · 2025-08-04T11:59:58Z

BE Regression && UT Coverage Report

Increment line coverage 23.33% (14/60) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.15% (20950/27510)
Line Coverage	69.55% (216043/310618)
Region Coverage	67.61% (129375/191354)
Branch Coverage	61.17% (67297/110010)

hello-stephen · 2025-08-04T13:10:30Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

be/src/cloud/cloud_meta_mgr.cpp

be/src/cloud/cloud_warm_up_manager.h

be/src/cloud/config.h

be/src/cloud/cloud_warm_up_manager.cpp

be/src/cloud/cloud_internal_service.cpp

zhannngchen · 2025-08-06T15:34:16Z

run buildall

hello-stephen · 2025-08-06T16:41:10Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

zhannngchen · 2025-08-07T03:29:58Z

run buildall

zhannngchen · 2025-08-07T03:39:03Z

run buildall

doris-robot · 2025-08-07T03:59:34Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.12% (1212/1494)
Line Coverage	65.40% (21455/32808)
Region Coverage	66.94% (10808/16145)
Branch Coverage	56.53% (5687/10060)

zhannngchen · 2025-08-07T09:14:06Z

run buildall

hello-stephen · 2025-08-07T09:38:06Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	81.12% (1212/1494)
Line Coverage	65.40% (21458/32808)
Region Coverage	66.95% (10809/16145)
Branch Coverage	56.51% (5685/10060)

doris-robot · 2025-08-07T10:07:10Z

TPC-H: Total hot run time: 32648 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 792102371c98beb57feb117bd27e5f252c8cb791, data reload: false

------ Round 1 ----------------------------------
q1	17929	5509	5507	5507
q2	2057	291	162	162
q3	10870	1254	745	745
q4	10226	877	463	463
q5	8402	2363	2122	2122
q6	186	168	132	132
q7	890	763	599	599
q8	9337	1442	1185	1185
q9	5235	4935	4943	4935
q10	6777	2276	1804	1804
q11	467	274	260	260
q12	343	355	213	213
q13	17770	3605	3003	3003
q14	228	230	213	213
q15	539	460	462	460
q16	408	413	371	371
q17	612	862	358	358
q18	6995	6455	6406	6406
q19	1818	948	549	549
q20	313	333	201	201
q21	2858	2142	1975	1975
q22	1037	1022	985	985
Total cold run time: 105297 ms
Total hot run time: 32648 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5642	5521	5502	5502
q2	246	342	228	228
q3	2225	2645	2309	2309
q4	1347	1747	1374	1374
q5	4417	4915	4949	4915
q6	166	163	133	133
q7	2071	1928	1827	1827
q8	2611	2830	2695	2695
q9	7266	7227	7204	7204
q10	3026	3300	2810	2810
q11	565	499	488	488
q12	631	744	587	587
q13	3468	3817	3180	3180
q14	291	301	261	261
q15	517	478	466	466
q16	433	496	438	438
q17	1228	1752	1248	1248
q18	7628	7417	7470	7417
q19	820	1140	1081	1081
q20	1981	2034	1914	1914
q21	5306	5002	4626	4626
q22	1102	1066	991	991
Total cold run time: 52987 ms
Total hot run time: 51694 ms

doris-robot · 2025-08-07T10:19:30Z

TPC-DS: Total hot run time: 197791 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 792102371c98beb57feb117bd27e5f252c8cb791, data reload: false

query1	1282	948	917	917
query2	6250	1954	1934	1934
query3	10858	4572	4429	4429
query4	33057	23778	23718	23718
query5	3792	606	446	446
query6	273	198	178	178
query7	3987	498	314	314
query8	291	239	233	233
query9	9456	2545	2548	2545
query10	485	334	270	270
query11	18013	15458	15470	15458
query12	169	108	114	108
query13	1560	562	415	415
query14	9224	6958	7283	6958
query15	236	207	194	194
query16	8028	663	514	514
query17	1609	794	608	608
query18	2125	426	314	314
query19	231	191	175	175
query20	124	120	123	120
query21	213	136	118	118
query22	4508	4501	4479	4479
query23	34627	34297	33902	33902
query24	7343	2675	2674	2674
query25	507	527	435	435
query26	773	283	175	175
query27	2117	492	368	368
query28	5514	2203	2189	2189
query29	661	599	487	487
query30	246	188	160	160
query31	984	911	874	874
query32	74	63	56	56
query33	505	365	295	295
query34	746	848	544	544
query35	777	812	755	755
query36	1005	1068	964	964
query37	104	100	69	69
query38	4026	4045	3993	3993
query39	1567	1496	1636	1496
query40	212	118	116	116
query41	50	51	53	51
query42	131	113	109	109
query43	518	523	482	482
query44	1318	827	806	806
query45	187	180	172	172
query46	881	1051	662	662
query47	1980	1995	1950	1950
query48	411	419	348	348
query49	740	503	430	430
query50	709	707	429	429
query51	7317	7369	7285	7285
query52	106	105	94	94
query53	231	255	187	187
query54	542	564	476	476
query55	78	79	81	79
query56	273	267	245	245
query57	1282	1283	1173	1173
query58	247	223	220	220
query59	3072	3358	3178	3178
query60	305	283	261	261
query61	121	129	114	114
query62	838	747	701	701
query63	225	185	192	185
query64	3489	1015	657	657
query65	3348	3292	3293	3292
query66	951	405	299	299
query67	15993	15749	15631	15631
query68	7602	822	527	527
query69	534	311	272	272
query70	1206	1122	1078	1078
query71	409	299	261	261
query72	5882	3833	3835	3833
query73	658	760	347	347
query74	10380	9124	9168	9124
query75	3235	3159	2690	2690
query76	3092	1185	765	765
query77	492	380	273	273
query78	10353	10484	9610	9610
query79	3655	858	578	578
query80	615	519	428	428
query81	531	252	220	220
query82	648	115	99	99
query83	157	157	150	150
query84	249	98	75	75
query85	768	360	306	306
query86	392	304	304	304
query87	4351	4325	4193	4193
query88	5193	2386	2400	2386
query89	412	333	296	296
query90	1793	185	193	185
query91	136	148	109	109
query92	65	55	50	50
query93	2131	883	542	542
query94	696	414	313	313
query95	345	282	277	277
query96	488	610	280	280
query97	3175	3306	3153	3153
query98	226	216	212	212
query99	2133	1435	1342	1342
Total cold run time: 294904 ms
Total hot run time: 197791 ms

doris-robot · 2025-08-07T10:24:54Z

ClickBench: Total hot run time: 29.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 792102371c98beb57feb117bd27e5f252c8cb791, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.24	0.05	0.05
query4	1.64	0.08	0.08
query5	0.50	0.50	0.52
query6	1.14	0.75	0.74
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.57	0.50	0.50
query10	0.56	0.55	0.56
query11	0.17	0.12	0.12
query12	0.17	0.13	0.13
query13	0.61	0.60	0.60
query14	0.79	0.82	0.80
query15	0.86	0.85	0.83
query16	0.38	0.38	0.38
query17	1.01	1.01	1.05
query18	0.18	0.18	0.20
query19	2.00	1.87	1.83
query20	0.02	0.01	0.02
query21	15.37	0.97	0.66
query22	0.77	0.77	0.69
query23	14.85	1.53	0.70
query24	2.25	0.36	0.23
query25	0.14	0.08	0.08
query26	0.28	0.18	0.19
query27	0.09	0.08	0.08
query28	13.36	1.31	0.54
query29	12.68	4.09	3.39
query30	0.26	0.09	0.06
query31	2.83	0.61	0.39
query32	3.24	0.58	0.49
query33	3.01	3.05	3.06
query34	16.58	5.18	4.58
query35	4.65	4.62	4.67
query36	0.62	0.48	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.16
query39	0.05	0.05	0.04
query40	0.16	0.14	0.14
query41	0.10	0.06	0.06
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 102.79 s
Total hot run time: 29.29 s

hello-stephen · 2025-08-07T10:28:58Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-08-07T10:52:15Z

BE UT Coverage Report

Increment line coverage 0.00% (0/46) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	45.43% (12654/27856)
Line Coverage	36.28% (112815/310977)
Region Coverage	35.32% (58315/165111)
Branch Coverage	32.54% (31747/97556)

hello-stephen · 2025-08-07T12:09:05Z

BE Regression && UT Coverage Report

Increment line coverage 19.44% (14/72) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.21% (20975/27524)
Line Coverage	69.62% (216371/310808)
Region Coverage	67.62% (129490/191486)
Branch Coverage	61.19% (67375/110108)

…armup` coredump (#55932) ### What problem does this PR solve? fix coredump introduced in #54284 ``` *** Query id: 0-0 *** *** is nereids: 0 *** *** tablet id: 0 *** *** Aborted at 1757323304 (unix time) try "date -d @1757323304" if you are using GNU date *** *** Current BE git commitID: e2bdf54 *** *** SIGFPE integer divide by zero (@0x562de1b7030d) received by PID 5468 (TID 5727 OR 0x7f480ebd7640) from PID 18446744073201451789; stack trace: *** *** stack smashing detected ***: terminated *** stack smashing detected ***: terminated 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:420 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007F493CE5F520 in /lib/x86_64-linux-gnu/libc.so.6 4# std::_Hashtable<doris::RowsetId, std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> >, std::__detail::_Select1st, std::equal_to<doris::RowsetId>, std::hash<doris::RowsetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_locate(doris::RowsetId const&) const at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:2280 5# doris::CloudTablet::complete_rowset_segment_warmup(doris::RowsetId, doris::Status) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_tablet.cpp:1630 6# std::_Function_handler<void (doris::Status), doris::CloudInternalServiceImpl::warm_up_rowset(google::protobuf::RpcController*, doris::PWarmUpRowsetRequest const*, doris::PWarmUpRowsetResponse*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&, doris::Status&&) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292 7# doris::io::FileCacheBlockDownloader::download_segment_file(doris::io::DownloadFileMeta const&) at /home/zcp/repo_center/doris_master/doris/be/src/io/cache/block_file_cache_downloader.cpp:297 8# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:621 9# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:461 10# start_thread at ./nptl/pthread_create.c:442 11# 0x00007F493CF43850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 ```

zhannngchen requested a review from morrySnow as a code owner August 4, 2025 08:49

zhannngchen mentioned this pull request Aug 4, 2025

[opt](warm up) support delaying compaction commit until rowset warm-up finishes #54275

Closed

16 tasks