branch-3.1: [Fix](mow) Fix mow coreudmp in `BaseTablet::get_rowset_by_ids()` #55539 #55601

github-actions · 2025-09-02T13:21:57Z

Cherry-picked from #55539

@0x0

) ### What problem does this PR solve? ``` *** Query id: 7b4f7bbcecef464c-81b65c1731e755ee *** *** is nereids: 0 *** *** tablet id: 1756432156986 *** *** Aborted at 1756473436 (unix time) try "date -d @1756473436" if you are using GNU date *** *** Current BE git commitID: d26c87d *** *** SIGSEGV unknown detail explain (@0x0) received by PID 333577 (TID 379616 OR 0x7fed07dd9640) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/common/signal_handler.h:421 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007FEE2C7EA520 in /lib/x86_64-linux-gnu/libc.so.6 4# std::_Hashtable<doris::RowsetId, doris::RowsetId, std::allocator<doris::RowsetId>, std::__detail::_Identity, std::equal_to<doris::RowsetId>, std::hash<doris::RowsetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::find(doris::RowsetId const&) const at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/hashtable.h:1587 5# doris::BaseTablet::get_rowset_by_ids(std::unordered_set<doris::RowsetId, std::hash<doris::RowsetId>, std::equal_to<doris::RowsetId>, std::allocator<doris::RowsetId> > const*) at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/olap/base_tablet.cpp:440 6# doris::BaseBetaRowsetWriter::_generate_delete_bitmap(int) at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/olap/rowset/beta_rowset_writer.cpp:332 7# doris::BaseBetaRowsetWriter::add_segment(unsigned int, doris::SegmentStatistics const&, std::shared_ptr<doris::TabletSchema>) at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/olap/rowset/beta_rowset_writer.cpp:1104 8# doris::SegmentCollectorT<doris::BaseBetaRowsetWriter>::add(unsigned int, doris::SegmentStatistics&, std::shared_ptr<doris::TabletSchema>) at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/olap/rowset/segment_creator.h:91 9# doris::SegmentFlusher::_flush_segment_writer(std::unique_ptr<doris::segment_v2::VerticalSegmentWriter, std::default_delete<doris::segment_v2::VerticalSegmentWriter> >&, std::shared_ptr<doris::TabletSchema>, long*) at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/olap/rowset/segment_creator.cpp:263 10# doris::SegmentFlusher::flush_single_block(doris::vectorized::Block const*, int, long*) at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/olap/rowset/segment_creator.cpp:76 11# doris::SegmentCreator::flush_single_block(doris::vectorized::Block const*, int, long*) at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/olap/rowset/segment_creator.cpp:416 12# doris::BaseBetaRowsetWriter::flush_memtable(doris::vectorized::Block*, int, long*) at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/olap/rowset/beta_rowset_writer.cpp:719 13# doris::FlushToken::_do_flush_memtable(doris::MemTable*, int, long*) at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/olap/memtable_flush_executor.cpp:160 14# doris::FlushToken::_flush_memtable(std::shared_ptr<doris::MemTable>, int, long) in /mnt/hdd01/PERFORMANCE_ENV/be/lib/doris_be 15# doris::MemtableFlushTask::run() at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/olap/memtable_flush_executor.cpp:60 16# doris::ThreadPool::dispatch_thread() at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/util/threadpool.cpp:609 17# doris::Thread::supervise_thread(void*) at /mnt/disk3/pipeline/repo/selectdb-core_selectdb-second-branch/be/src/util/thread.cpp:499 18# start_thread at ./nptl/pthread_create.c:442 19# 0x00007FEE2C8CE850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 ```

Thearas · 2025-09-02T13:22:03Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Thearas · 2025-09-02T13:22:07Z

run buildall

doris-robot · 2025-09-02T15:10:49Z

TPC-H: Total hot run time: 32832 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef68f0f5e2561bb6159b4c7084fa7fc9c9c6ba8f, data reload: false

------ Round 1 ----------------------------------
q1	17938	5462	5576	5462
q2	2022	439	284	284
q3	12470	1249	748	748
q4	10281	882	460	460
q5	8905	2431	2147	2147
q6	183	164	132	132
q7	914	731	610	610
q8	9344	1462	1128	1128
q9	5205	5033	4958	4958
q10	6788	2300	1812	1812
q11	481	302	258	258
q12	340	359	214	214
q13	17783	3617	3013	3013
q14	226	235	215	215
q15	511	471	464	464
q16	423	429	375	375
q17	583	876	361	361
q18	7046	6634	6443	6443
q19	1206	948	551	551
q20	327	339	200	200
q21	2856	2209	1992	1992
q22	1102	1048	1005	1005
Total cold run time: 106934 ms
Total hot run time: 32832 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5545	5495	5516	5495
q2	233	334	237	237
q3	2301	2640	2363	2363
q4	1356	1801	1372	1372
q5	4436	5061	5021	5021
q6	179	169	129	129
q7	2123	1970	1842	1842
q8	2647	2870	2730	2730
q9	7300	7335	7261	7261
q10	3091	3230	2734	2734
q11	569	522	498	498
q12	690	785	614	614
q13	3401	3778	3161	3161
q14	292	293	294	293
q15	520	481	468	468
q16	429	515	424	424
q17	1259	1784	1255	1255
q18	7731	7524	7387	7387
q19	821	1172	1086	1086
q20	2075	2036	1922	1922
q21	5413	5082	4532	4532
q22	1052	1094	1030	1030
Total cold run time: 53463 ms
Total hot run time: 51854 ms

doris-robot · 2025-09-02T15:22:58Z

TPC-DS: Total hot run time: 192771 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef68f0f5e2561bb6159b4c7084fa7fc9c9c6ba8f, data reload: false

query1	984	414	397	397
query2	6231	1916	1896	1896
query3	8695	206	198	198
query4	33629	23653	23507	23507
query5	3691	593	438	438
query6	293	197	189	189
query7	4203	497	325	325
query8	303	238	245	238
query9	9321	2607	2598	2598
query10	457	318	251	251
query11	18049	15504	15410	15410
query12	175	110	104	104
query13	1556	552	422	422
query14	9536	6689	7204	6689
query15	255	197	174	174
query16	8095	681	497	497
query17	1549	793	608	608
query18	2137	413	344	344
query19	239	191	172	172
query20	133	123	127	123
query21	207	129	107	107
query22	4597	4444	4458	4444
query23	35140	34528	33946	33946
query24	7736	2728	2732	2728
query25	534	511	447	447
query26	737	291	178	178
query27	2123	490	353	353
query28	5176	2285	2258	2258
query29	703	612	495	495
query30	239	197	168	168
query31	1008	946	846	846
query32	92	69	65	65
query33	520	385	334	334
query34	729	875	534	534
query35	787	810	738	738
query36	1017	1048	980	980
query37	107	101	71	71
query38	4107	3986	3951	3951
query39	1547	1468	1459	1459
query40	209	126	108	108
query41	50	51	47	47
query42	123	110	109	109
query43	512	532	481	481
query44	1290	826	822	822
query45	189	182	173	173
query46	898	1074	679	679
query47	1978	1967	1937	1937
query48	419	427	355	355
query49	738	488	419	419
query50	674	691	428	428
query51	7504	7359	7289	7289
query52	101	99	89	89
query53	230	260	197	197
query54	566	550	486	486
query55	80	80	82	80
query56	271	279	272	272
query57	1274	1293	1203	1203
query58	226	239	224	224
query59	3082	3121	3070	3070
query60	301	297	259	259
query61	119	116	112	112
query62	782	752	738	738
query63	240	199	198	198
query64	3755	993	673	673
query65	3409	3348	3277	3277
query66	796	426	310	310
query67	16138	15836	15732	15732
query68	7914	830	567	567
query69	501	303	266	266
query70	1206	1153	1129	1129
query71	375	295	267	267
query72	5823	3661	3867	3661
query73	654	739	350	350
query74	10216	9364	9180	9180
query75	3206	3148	2718	2718
query76	3008	1173	765	765
query77	509	362	287	287
query78	10316	10354	9573	9573
query79	3504	910	611	611
query80	797	506	428	428
query81	506	261	224	224
query82	606	119	90	90
query83	169	163	147	147
query84	286	109	86	86
query85	786	372	298	298
query86	392	314	280	280
query87	4417	4321	4209	4209
query88	5106	2439	2428	2428
query89	405	332	301	301
query90	1706	190	195	190
query91	134	142	115	115
query92	67	57	53	53
query93	2182	889	558	558
query94	687	412	307	307
query95	347	279	280	279
query96	493	608	287	287
query97	3192	3271	3127	3127
query98	231	208	204	204
query99	1536	1432	1337	1337
Total cold run time: 293868 ms
Total hot run time: 192771 ms

doris-robot · 2025-09-02T15:28:13Z

ClickBench: Total hot run time: 28.93 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef68f0f5e2561bb6159b4c7084fa7fc9c9c6ba8f, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.02	0.03
query3	0.24	0.07	0.07
query4	1.63	0.11	0.11
query5	0.54	0.51	0.51
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.56	0.51	0.49
query10	0.56	0.56	0.57
query11	0.14	0.10	0.10
query12	0.15	0.11	0.11
query13	0.65	0.59	0.59
query14	0.77	0.83	0.80
query15	0.83	0.84	0.83
query16	0.40	0.39	0.38
query17	1.07	1.02	1.08
query18	0.24	0.23	0.23
query19	1.85	1.81	1.83
query20	0.01	0.01	0.01
query21	15.39	0.94	0.57
query22	0.74	0.96	0.64
query23	15.06	1.36	0.53
query24	2.93	1.31	0.98
query25	0.13	0.09	0.17
query26	0.26	0.14	0.13
query27	0.06	0.05	0.04
query28	13.41	1.02	0.43
query29	12.57	3.82	3.20
query30	0.25	0.09	0.09
query31	2.81	0.62	0.38
query32	3.22	0.54	0.45
query33	2.99	3.07	3.08
query34	16.67	5.17	4.55
query35	4.57	4.57	4.55
query36	0.63	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 103.15 s
Total hot run time: 28.93 s

doris-robot · 2025-09-02T15:35:34Z

BE UT Coverage Report

Increment line coverage 53.33% (8/15) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	45.49% (12725/27973)
Line Coverage	36.38% (113480/311932)
Region Coverage	34.00% (64945/191010)
Branch Coverage	31.05% (34095/109814)

morrySnow · 2025-09-04T02:58:09Z

run buildall

doris-robot · 2025-09-04T04:47:30Z

TPC-H: Total hot run time: 32972 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef68f0f5e2561bb6159b4c7084fa7fc9c9c6ba8f, data reload: false

------ Round 1 ----------------------------------
q1	17903	5510	5541	5510
q2	2041	399	292	292
q3	12040	1248	755	755
q4	10285	894	468	468
q5	8856	2431	2162	2162
q6	184	166	131	131
q7	912	779	625	625
q8	9343	1477	1145	1145
q9	5144	4938	4927	4927
q10	6784	2281	1854	1854
q11	462	284	268	268
q12	342	379	214	214
q13	17766	3593	3060	3060
q14	227	233	220	220
q15	520	465	467	465
q16	427	439	392	392
q17	588	874	362	362
q18	7002	6360	6406	6360
q19	1445	969	561	561
q20	334	336	212	212
q21	2837	2199	1991	1991
q22	1078	1034	998	998
Total cold run time: 106520 ms
Total hot run time: 32972 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5511	5486	6224	5486
q2	234	323	231	231
q3	2289	2642	2327	2327
q4	1356	1824	1414	1414
q5	4438	5074	4952	4952
q6	170	162	125	125
q7	2066	1968	1860	1860
q8	2649	2847	2718	2718
q9	7283	7239	7206	7206
q10	3022	3269	2721	2721
q11	569	496	498	496
q12	663	757	627	627
q13	3430	3830	3196	3196
q14	274	301	275	275
q15	513	489	471	471
q16	437	493	430	430
q17	1230	1765	1253	1253
q18	7607	7446	7370	7370
q19	805	1173	1083	1083
q20	2038	2060	1899	1899
q21	5363	4935	4653	4653
q22	1086	1063	1003	1003
Total cold run time: 53033 ms
Total hot run time: 51796 ms

doris-robot · 2025-09-04T04:59:40Z

TPC-DS: Total hot run time: 193128 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef68f0f5e2561bb6159b4c7084fa7fc9c9c6ba8f, data reload: false

query1	931	404	405	404
query2	6254	1913	1867	1867
query3	8688	207	207	207
query4	33094	23894	23647	23647
query5	3754	602	457	457
query6	303	198	188	188
query7	4206	500	333	333
query8	318	246	246	246
query9	9416	2623	2605	2605
query10	463	321	258	258
query11	18242	15565	15206	15206
query12	160	110	108	108
query13	1569	578	421	421
query14	8916	6658	6675	6658
query15	232	193	186	186
query16	8017	648	519	519
query17	1555	776	608	608
query18	2153	416	325	325
query19	198	190	167	167
query20	124	126	121	121
query21	206	139	112	112
query22	4690	4916	4498	4498
query23	35303	34311	34136	34136
query24	8322	2657	2723	2657
query25	513	486	429	429
query26	727	287	182	182
query27	2510	487	363	363
query28	5817	2249	2246	2246
query29	682	593	445	445
query30	239	185	157	157
query31	999	934	887	887
query32	91	63	68	63
query33	544	377	318	318
query34	733	891	532	532
query35	793	835	742	742
query36	1025	1053	969	969
query37	108	102	75	75
query38	4038	4027	3956	3956
query39	1513	1479	1470	1470
query40	213	124	113	113
query41	53	53	49	49
query42	130	107	108	107
query43	511	523	485	485
query44	1295	821	830	821
query45	201	185	174	174
query46	889	1081	689	689
query47	2007	2001	1933	1933
query48	432	449	358	358
query49	763	521	436	436
query50	700	720	444	444
query51	7366	7363	7314	7314
query52	100	97	91	91
query53	228	256	194	194
query54	571	549	512	512
query55	81	82	82	82
query56	252	279	278	278
query57	1295	1261	1205	1205
query58	240	224	222	222
query59	3055	3140	3095	3095
query60	279	286	279	279
query61	116	110	112	110
query62	799	761	714	714
query63	237	198	207	198
query64	3414	1010	640	640
query65	3394	3271	3309	3271
query66	809	418	323	323
query67	16395	15984	15798	15798
query68	7670	836	560	560
query69	495	309	275	275
query70	1216	1103	1106	1103
query71	366	291	261	261
query72	5726	3893	3835	3835
query73	640	751	364	364
query74	10732	9384	8969	8969
query75	3191	3171	2699	2699
query76	3136	1196	765	765
query77	581	372	280	280
query78	10326	10447	9596	9596
query79	2601	899	631	631
query80	662	522	449	449
query81	498	256	219	219
query82	394	117	87	87
query83	172	158	147	147
query84	246	101	79	79
query85	813	371	309	309
query86	378	324	306	306
query87	4405	4364	4272	4272
query88	5530	2437	2424	2424
query89	398	335	300	300
query90	1755	189	193	189
query91	143	141	110	110
query92	69	55	52	52
query93	1949	902	559	559
query94	663	425	323	323
query95	336	280	267	267
query96	495	606	291	291
query97	3186	3245	3187	3187
query98	228	218	199	199
query99	1570	1443	1318	1318
Total cold run time: 293841 ms
Total hot run time: 193128 ms

doris-robot · 2025-09-04T05:04:53Z

ClickBench: Total hot run time: 28.47 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef68f0f5e2561bb6159b4c7084fa7fc9c9c6ba8f, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.51	0.51	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.50	0.50
query10	0.55	0.55	0.58
query11	0.14	0.11	0.11
query12	0.14	0.12	0.11
query13	0.62	0.60	0.59
query14	0.80	0.81	0.80
query15	0.85	0.84	0.82
query16	0.38	0.39	0.38
query17	1.02	1.07	1.00
query18	0.24	0.23	0.23
query19	1.87	1.77	1.82
query20	0.01	0.01	0.01
query21	15.40	0.89	0.58
query22	0.75	0.71	0.55
query23	15.29	1.42	0.51
query24	3.11	0.97	0.64
query25	0.23	0.21	0.11
query26	0.27	0.15	0.15
query27	0.06	0.03	0.04
query28	13.53	1.05	0.44
query29	12.56	3.88	3.17
query30	0.25	0.10	0.06
query31	2.83	0.63	0.39
query32	3.23	0.56	0.48
query33	3.00	3.06	3.06
query34	16.65	5.20	4.62
query35	4.55	4.54	4.57
query36	0.66	0.50	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 103.69 s
Total hot run time: 28.47 s

doris-robot · 2025-09-04T05:11:29Z

BE UT Coverage Report

Increment line coverage 53.33% (8/15) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	45.52% (12744/27996)
Line Coverage	36.37% (113622/312390)
Region Coverage	34.00% (65008/191173)
Branch Coverage	31.04% (34117/109916)

hello-stephen · 2025-09-04T07:53:32Z

BE Regression && UT Coverage Report

Increment line coverage 93.33% (14/15) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.25% (20988/27525)
Line Coverage	69.46% (216273/311341)
Region Coverage	67.36% (129269/191912)
Branch Coverage	60.91% (67252/110412)

hello-stephen · 2025-09-04T08:54:54Z

BE Regression && UT Coverage Report

Increment line coverage 93.33% (14/15) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.25% (20988/27525)
Line Coverage	69.46% (216273/311341)
Region Coverage	67.36% (129269/191912)
Branch Coverage	60.91% (67252/110412)

dataroaring

LGTM

github-actions bot requested a review from morrySnow as a code owner September 2, 2025 13:21

dataroaring closed this Sep 2, 2025

dataroaring reopened this Sep 2, 2025

dataroaring approved these changes Sep 5, 2025

View reviewed changes

morrySnow approved these changes Sep 5, 2025

View reviewed changes

morrySnow merged commit a50a89b into branch-3.1 Sep 5, 2025
22 of 23 checks passed

github-actions bot deleted the auto-pick-55539-branch-3.1 branch September 5, 2025 10:20

morrySnow mentioned this pull request Sep 22, 2025

Release Note 3.1.1 #56289

Closed

branch-3.1: [Fix](mow) Fix mow coreudmp in BaseTablet::get_rowset_by_ids() #55539 #55601

branch-3.1: [Fix](mow) Fix mow coreudmp in BaseTablet::get_rowset_by_ids() #55539 #55601

Uh oh!

Conversation

github-actions bot commented Sep 2, 2025

Uh oh!

Thearas commented Sep 2, 2025

Uh oh!

Thearas commented Sep 2, 2025

Uh oh!

doris-robot commented Sep 2, 2025

Uh oh!

doris-robot commented Sep 2, 2025

Uh oh!

doris-robot commented Sep 2, 2025

Uh oh!

doris-robot commented Sep 2, 2025

BE UT Coverage Report

Uh oh!

morrySnow commented Sep 4, 2025

Uh oh!

doris-robot commented Sep 4, 2025

Uh oh!

doris-robot commented Sep 4, 2025

Uh oh!

doris-robot commented Sep 4, 2025

Uh oh!

doris-robot commented Sep 4, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 4, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Sep 4, 2025

BE Regression && UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

branch-3.1: [Fix](mow) Fix mow coreudmp in `BaseTablet::get_rowset_by_ids()` #55539 #55601

branch-3.1: [Fix](mow) Fix mow coreudmp in `BaseTablet::get_rowset_by_ids()` #55539 #55601