[Bug](fix) try to fix double free materialization status #51518

HappenLee · 2025-06-05T07:17:29Z

What problem does this PR solve?

=================================================================
==8328==ERROR: AddressSanitizer: attempting double-free on 0x60600333cf80 in thread T1682:
    #0 0x5631ccdaed9d in operator delete(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa6d9d) (BuildId: dab4df3df11ecdce)
    #1 0x5631ccdde467 in std::__uniq_ptr_impl>::operator=(std::__uniq_ptr_impl>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:167:2
    #2 0x5631ccdc942d in std::__uniq_ptr_data, true, true>::operator=(std::__uniq_ptr_data, true, true>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:212:61
    #3 0x5631ccdc942d in std::unique_ptr>::operator=(std::unique_ptr>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:371:51
    #4 0x5631ccdc942d in doris::Status::operator=(doris::Status&&) /home/zcp/repo_center/doris_master/doris/be/src/common/status.h:400:22
    #5 0x56320bfc33ba in doris::pipeline::MaterializationCallback::call() /home/zcp/repo_center/doris_master/doris/be/src/pipeline/exec/materialization_sink_operator.cpp:88:39
    #6 0x56320bfc5cdd in doris::AutoReleaseClosure>::Run() /home/zcp/repo_center/doris_master/doris/be/src/util/brpc_closure.h:98:18
    #7 0x5631d03b7b1b in doris::FailureDetectClosure::Run() /home/zcp/repo_center/doris_master/doris/be/src/util/brpc_client_cache.h:83:20
    #8 0x56320dce1943 in brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9d9943) (BuildId: dab4df3df11ecdce)
    #9 0x56320dd0cc4e in brpc::policy::ProcessRpcResponse(brpc::InputMessageBase*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7da04c4e) (BuildId: dab4df3df11ecdce)
    #10 0x56320dd03f76 in brpc::ProcessInputMessage(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9fbf76) (BuildId: dab4df3df11ecdce)
    #11 0x56320dd04920 in brpc::InputMessenger::InputMessageClosure::~InputMessageClosure() (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9fc920) (BuildId: dab4df3df11ecdce)
    #12 0x56320dd052ab in brpc::InputMessenger::OnNewMessages(brpc::Socket*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9fd2ab) (BuildId: dab4df3df11ecdce)
    #13 0x56320de3437d in brpc::Socket::ProcessEvent(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7db2c37d) (BuildId: dab4df3df11ecdce)
    #14 0x56320dc970a6 in bthread::TaskGroup::task_runner(long) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d98f0a6) (BuildId: dab4df3df11ecdce)
    #15 0x56320dc83660 in bthread_make_fcontext (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d97b660) (BuildId: dab4df3df11ecdce)

0x60600333cf80 is located 0 bytes inside of 64-byte region [0x60600333cf80,0x60600333cfc0)
freed by thread T1791 here:
    #0 0x5631ccdaed9d in operator delete(void*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa6d9d) (BuildId: dab4df3df11ecdce)

previously allocated by thread T1623 here:
    #0 0x5631ccdae53d in operator new(unsigned long) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa653d) (BuildId: dab4df3df11ecdce)

Thread T1682 created by T0 here:
    #0 0x5631ccd5bcaa in pthread_create (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3ca53caa) (BuildId: dab4df3df11ecdce)
    #1 0x56320dc907be in bthread::TaskControl::add_workers(int) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9887be) (BuildId: dab4df3df11ecdce)
    #2 0x56320dc7f0ac in bthread_setconcurrency (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9770ac) (BuildId: dab4df3df11ecdce)
    #3 0x56320ddfcc79 in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf4c79) (BuildId: dab4df3df11ecdce)
    #4 0x56320ddfecc9 in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf6cc9) (BuildId: dab4df3df11ecdce)
    #5 0x5631d13708df in doris::BRpcService::start(int, int) /home/zcp/repo_center/doris_master/doris/be/src/service/brpc_service.cpp:94:18
    #6 0x5631ccdb73e2 in main /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:570:28
    #7 0x7faec2201d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

Thread T1791 created by T0 here:
    #0 0x5631ccd5bcaa in pthread_create (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3ca53caa) (BuildId: dab4df3df11ecdce)
    #1 0x56320dc907be in bthread::TaskControl::add_workers(int) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9887be) (BuildId: dab4df3df11ecdce)
    #2 0x56320dc7f0ac in bthread_setconcurrency (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9770ac) (BuildId: dab4df3df11ecdce)
    #3 0x56320ddfcc79 in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf4c79) (BuildId: dab4df3df11ecdce)
    #4 0x56320ddfecc9 in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf6cc9) (BuildId: dab4df3df11ecdce)
    #5 0x5631d13708df in doris::BRpcService::start(int, int) /home/zcp/repo_center/doris_master/doris/be/src/service/brpc_service.cpp:94:18
    #6 0x5631ccdb73e2 in main /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:570:28
    #7 0x7faec2201d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

Thread T1623 created by T0 here:
    #0 0x5631ccd5bcaa in pthread_create (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3ca53caa) (BuildId: dab4df3df11ecdce)
    #1 0x56320dc907be in bthread::TaskControl::add_workers(int) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9887be) (BuildId: dab4df3df11ecdce)
    #2 0x56320dc7f0ac in bthread_setconcurrency (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7d9770ac) (BuildId: dab4df3df11ecdce)
    #3 0x56320ddfcc79 in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf4c79) (BuildId: dab4df3df11ecdce)
    #4 0x56320ddfecc9 in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x7daf6cc9) (BuildId: dab4df3df11ecdce)
    #5 0x5631d13708df in doris::BRpcService::start(int, int) /home/zcp/repo_center/doris_master/doris/be/src/service/brpc_service.cpp:94:18
    #6 0x5631ccdb73e2 in main /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:570:28
    #7 0x7faec2201d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

SUMMARY: AddressSanitizer: double-free (/mnt/hdd01/ci/doris-deploy-master-local/be/lib/doris_be+0x3caa6d9d) (BuildId: dab4df3df11ecdce) in operator delete(void*)

Add a lock when need change the materialization status

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-06-05T07:17:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

HappenLee · 2025-06-05T07:27:28Z

run buildall

github-actions · 2025-06-05T07:29:59Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-06-05T07:30:01Z

PR approved by anyone and no changes requested.

doris-robot · 2025-06-05T08:15:50Z

TPC-H: Total hot run time: 33917 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6301ab5e011a9f404217774f340ba017f6604a11, data reload: false

------ Round 1 ----------------------------------
q1	26259	5079	5033	5033
q2	1952	267	171	171
q3	10315	1231	716	716
q4	10228	997	514	514
q5	7549	2344	2378	2344
q6	183	171	137	137
q7	914	743	619	619
q8	9295	1326	1078	1078
q9	6889	5184	5074	5074
q10	6858	2330	1922	1922
q11	485	281	266	266
q12	335	350	214	214
q13	17787	3630	3065	3065
q14	234	228	209	209
q15	568	493	486	486
q16	423	438	373	373
q17	594	860	358	358
q18	7545	7262	7230	7230
q19	1231	972	562	562
q20	336	342	230	230
q21	3748	3146	2330	2330
q22	1037	1041	986	986
Total cold run time: 114765 ms
Total hot run time: 33917 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5113	5099	5088	5088
q2	238	319	221	221
q3	2162	2648	2306	2306
q4	1391	1779	1307	1307
q5	4459	4384	4396	4384
q6	220	175	130	130
q7	2004	1901	1759	1759
q8	2564	2586	2552	2552
q9	7222	7220	6902	6902
q10	3060	3209	2767	2767
q11	563	498	490	490
q12	701	757	619	619
q13	3513	3882	3262	3262
q14	286	293	281	281
q15	523	484	479	479
q16	465	506	435	435
q17	1135	1551	1368	1368
q18	7824	7567	7417	7417
q19	799	853	849	849
q20	2040	2056	1914	1914
q21	4753	4492	4318	4318
q22	1121	1020	1030	1020
Total cold run time: 52156 ms
Total hot run time: 49868 ms

doris-robot · 2025-06-05T08:28:02Z

TPC-DS: Total hot run time: 192505 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6301ab5e011a9f404217774f340ba017f6604a11, data reload: false

query1	1446	1066	1087	1066
query2	6138	1828	1803	1803
query3	11046	4407	4457	4407
query4	54167	25861	23335	23335
query5	4983	534	458	458
query6	352	209	198	198
query7	4949	494	293	293
query8	304	233	213	213
query9	5991	2674	2681	2674
query10	456	339	288	288
query11	15113	15001	15099	15001
query12	155	110	105	105
query13	1075	528	419	419
query14	10133	6468	6383	6383
query15	212	210	180	180
query16	7063	646	497	497
query17	1106	770	621	621
query18	1596	414	327	327
query19	202	210	168	168
query20	124	126	125	125
query21	208	130	109	109
query22	4390	4345	4329	4329
query23	34114	33201	33486	33201
query24	6617	2434	2433	2433
query25	457	481	424	424
query26	714	268	154	154
query27	2285	515	341	341
query28	3288	2205	2162	2162
query29	577	557	438	438
query30	270	216	193	193
query31	881	865	782	782
query32	72	68	64	64
query33	459	370	328	328
query34	765	858	526	526
query35	803	824	762	762
query36	955	986	895	895
query37	109	101	80	80
query38	4199	4253	4222	4222
query39	1501	1465	1467	1465
query40	209	126	112	112
query41	59	59	56	56
query42	124	106	137	106
query43	491	516	486	486
query44	1324	843	837	837
query45	181	183	171	171
query46	857	1028	644	644
query47	1796	1882	1761	1761
query48	387	418	319	319
query49	650	469	400	400
query50	663	696	430	430
query51	4240	4243	4179	4179
query52	107	107	101	101
query53	240	257	186	186
query54	585	586	536	536
query55	85	85	85	85
query56	307	312	286	286
query57	1156	1209	1113	1113
query58	264	274	256	256
query59	2677	2766	2709	2709
query60	345	327	318	318
query61	125	127	130	127
query62	746	737	668	668
query63	224	197	197	197
query64	1781	1055	716	716
query65	4253	4159	4145	4145
query66	714	410	296	296
query67	15876	15597	15619	15597
query68	6959	889	523	523
query69	545	311	268	268
query70	1217	1128	1137	1128
query71	523	326	294	294
query72	5541	4761	4811	4761
query73	1409	619	349	349
query74	9412	9046	8664	8664
query75	3854	3192	2670	2670
query76	4245	1197	744	744
query77	639	350	296	296
query78	10013	10189	9361	9361
query79	2133	817	585	585
query80	577	512	430	430
query81	481	269	219	219
query82	443	124	100	100
query83	252	252	235	235
query84	289	110	92	92
query85	790	358	317	317
query86	376	305	299	299
query87	4404	4621	4318	4318
query88	3667	2249	2270	2249
query89	417	337	285	285
query90	1813	202	205	202
query91	142	142	121	121
query92	67	60	58	58
query93	1760	936	593	593
query94	655	410	284	284
query95	367	299	290	290
query96	499	577	285	285
query97	2708	2749	2675	2675
query98	246	211	203	203
query99	1441	1414	1301	1301
Total cold run time: 297625 ms
Total hot run time: 192505 ms

doris-robot · 2025-06-05T08:33:20Z

ClickBench: Total hot run time: 28.73 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6301ab5e011a9f404217774f340ba017f6604a11, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.11
query3	0.24	0.20	0.20
query4	1.60	0.20	0.20
query5	0.45	0.44	0.44
query6	1.45	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.51
query10	0.56	0.59	0.57
query11	0.17	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.60	0.60
query14	0.80	0.81	0.80
query15	0.88	0.86	0.84
query16	0.39	0.38	0.38
query17	1.05	1.03	1.01
query18	0.23	0.21	0.21
query19	1.91	1.86	1.84
query20	0.02	0.01	0.01
query21	15.79	0.89	0.56
query22	0.81	1.29	0.66
query23	14.94	1.38	0.66
query24	7.78	0.69	0.36
query25	0.49	0.18	0.06
query26	0.71	0.17	0.15
query27	0.05	0.05	0.05
query28	8.78	0.90	0.44
query29	12.64	4.04	3.38
query30	0.25	0.09	0.07
query31	2.82	0.58	0.38
query32	3.22	0.56	0.47
query33	3.07	3.05	3.07
query34	15.83	5.08	4.47
query35	4.53	4.57	4.48
query36	0.66	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.05	0.04	0.03
Total cold run time: 104.17 s
Total hot run time: 28.73 s

hello-stephen · 2025-06-05T13:31:31Z

BE UT Coverage Report

Increment line coverage 0.00% (0/4) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.33% (15094/26798)
Line Coverage	45.10% (134696/298672)
Region Coverage	44.19% (67720/153250)
Branch Coverage	38.75% (34718/89606)

hello-stephen · 2025-06-05T13:34:36Z

BE Regression && UT Coverage Report

Increment line coverage 50.00% (2/4) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.59% (20994/26379)
Line Coverage	72.69% (217015/298569)
Region Coverage	70.92% (127795/180208)
Branch Coverage	64.61% (66176/102426)

[Bug](fix) try to fix double free materialization status

6301ab5

HappenLee force-pushed the new_cache branch from 1265213 to 6301ab5 Compare June 5, 2025 07:26

yiguolei approved these changes Jun 5, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 5, 2025

github-actions bot added the reviewed label Jun 5, 2025

jacktengg approved these changes Jun 6, 2025

View reviewed changes

yiguolei merged commit 63ad04b into apache:master Jun 6, 2025
24 of 26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Bug](fix) try to fix double free materialization status #51518

[Bug](fix) try to fix double free materialization status #51518

Uh oh!

HappenLee commented Jun 5, 2025

Uh oh!

hello-stephen commented Jun 5, 2025

Uh oh!

HappenLee commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

doris-robot commented Jun 5, 2025

Uh oh!

doris-robot commented Jun 5, 2025

Uh oh!

doris-robot commented Jun 5, 2025

Uh oh!

hello-stephen commented Jun 5, 2025

Uh oh!

hello-stephen commented Jun 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[Bug](fix) try to fix double free materialization status #51518

[Bug](fix) try to fix double free materialization status #51518

Uh oh!

Conversation

HappenLee commented Jun 5, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jun 5, 2025

Uh oh!

HappenLee commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

doris-robot commented Jun 5, 2025

Uh oh!

doris-robot commented Jun 5, 2025

Uh oh!

doris-robot commented Jun 5, 2025

Uh oh!

hello-stephen commented Jun 5, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Jun 5, 2025

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants