[fix](filecache) microbench crash when FileBufferBuilder #56204

freemandealer · 2025-09-18T14:23:59Z

#9 0x000055e72fad6219 in google::LogMessageFatal::~LogMessageFatal() ()
#10 0x000055e72a3f01fc in doris::ThreadMemTrackerMgr::limiter_mem_tracker_sptr (this=0x7f38233fa000) at /mnt/disk2/doris/be/src/runtime/memory/thread_mem_tracker_mgr.h:112
#11 0x000055e72b33efa0 in doris::SwitchThreadMemTrackerLimiter::SwitchThreadMemTrackerLimiter (this=0x7f3796375df7, mem_tracker=...) at /mnt/disk2/doris/be/src/runtime/thread_context.cpp:88
#12 0x000055e72a457f0b in doris::io::FileBufferBuilder::build (this=0x7f3796375e30, buf=0x7f3823375438) at /mnt/disk2/doris/be/src/io/fs/s3_file_bufferpool.cpp:243
#13 0x000055e72a44f448 in doris::io::S3FileWriter::_build_upload_buffer (this=) at /mnt/disk2/doris/be/src/io/fs/s3_file_writer.cpp:196
#14 0x000055e72a44cd71 in doris::io::S3FileWriter::appendv (this=0x7f3823375300, data=0x7f3823392000, data_cnt=200) at /mnt/disk2/doris/be/src/io/fs/s3_file_writer.cpp:248
#15 0x000055e72a355f02 in MicrobenchS3FileWriter::appendv (this=0x7f3823375300, slices=slices@entry=0x7f3823392000, slices_size=slices_size@entry=200, write_bvar=...) at /mnt/disk2/doris/be/src/io/tools/file_cache_microbench.cpp:303
#16 0x000055e72a3557d4 in JobManager::execute_write_tasks(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&, Job&, JobConfig const&)::{lambda()#1}::operator()() const (this=0x7f3819eb1d38) at /mnt/disk2/doris/be/src/io/tools/file_cache_microbench.cpp:1090

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

hello-stephen · 2025-09-18T14:24:06Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

freemandealer · 2025-09-18T14:24:10Z

run buildall

dataroaring

LGTM

github-actions · 2025-09-18T15:09:26Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-09-18T15:09:29Z

PR approved by anyone and no changes requested.

doris-robot · 2025-09-18T15:41:42Z

TPC-H: Total hot run time: 1497 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6fb0993bcc087a973b7b5fad09659687762b3673, data reload: false

------ Round 1 ----------------------------------
q1	17627	27	19	19
q2	2638	26	26	26
q3	10718	16	15	15
q4	11159	16	16	16
q5	8103	16	14	14
q6	249	15	14	14
q7	953	23	25	23
q8	9458	14	12	12
q9	10071	22	12	12
q10	7006	13	11	11
q11	729	21	20	20
q12	528	14	11	11
q13	18668	13	12	12
q14	254	12	11	11
q15	610	12	11	11
q16	1052	978	959	959
q17	572	13	11	11
q18	7913	11	11	11
q19	1146	12	12	12
q20	1949	373	246	246
q21	9196	20	20	20
q22	1142	13	11	11
Total cold run time: 121741 ms
Total hot run time: 1497 ms

----- Round 2, with runtime_filter_mode=off -----
q1	11	10	10	10
q2	19	19	19	19
q3	11	11	11	11
q4	13	10	10	10
q5	11	10	14	10
q6	10	11	11	11
q7	21	19	20	19
q8	11	10	10	10
q9	10	11	12	11
q10	10	10	11	10
q11	18	19	19	19
q12	9	10	12	10
q13	11	11	10	10
q14	10	10	10	10
q15	10	10	10	10
q16	1038	1081	1023	1023
q17	12	10	10	10
q18	10	9	9	9
q19	10	10	10	10
q20	1854	1977	1865	1865
q21	20	19	18	18
q22	10	9	11	9
Total cold run time: 3139 ms
Total hot run time: 3124 ms

doris-robot · 2025-09-18T15:47:11Z

TPC-DS: Total hot run time: 2734 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6fb0993bcc087a973b7b5fad09659687762b3673, data reload: false

query1	1087	20	12	12
query2	7132	16	17	16
query3	7639	12	12	12
query4	26800	12	12	12
query5	4416	14	12	12
query6	367	11	11	11
query7	5443	11	11	11
query8	357	19	17	17
query9	9251	11	10	10
query10	726	12	10	10
query11	15819	11	10	10
query12	181	12	10	10
query13	1715	11	11	11
query14	10809	15	13	13
query15	220	11	10	10
query16	7193	11	10	10
query17	1530	12	10	10
query18	2799	12	11	11
query19	212	12	9	9
query20	136	11	10	10
query21	233	10	10	10
query22	4106	10	10	10
query23	33428	15	12	12
query24	9550	13	12	12
query25	719	10	8	8
query26	1469	10	9	9
query27	3403	10	9	9
query28	6365	10	9	9
query29	1362	10	8	8
query30	619	10	10	10
query31	1758	11	10	10
query32	117	11	10	10
query33	1301	10	8	8
query34	1686	807	511	511
query35	930	11	10	10
query36	982	10	10	10
query37	121	10	10	10
query38	3533	8	7	7
query39	1546	751	722	722
query40	312	10	9	9
query41	77	10	11	10
query42	142	11	10	10
query43	507	10	9	9
query44	1330	10	9	9
query45	195	10	10	10
query46	901	9	8	8
query47	1791	9	9	9
query48	393	10	9	9
query49	1160	11	9	9
query50	754	9	9	9
query51	3883	10	9	9
query52	121	10	10	10
query53	244	10	10	10
query54	712	10	9	9
query55	91	9	7	7
query56	327	12	10	10
query57	1237	9	9	9
query58	317	11	11	11
query59	2738	9	8	8
query60	370	10	9	9
query61	182	10	10	10
query62	842	10	9	9
query63	257	10	10	10
query64	4579	11	9	9
query65	4004	9	8	8
query66	1165	11	10	10
query67	16587	18	9	9
query68	3424	10	10	10
query69	568	10	11	10
query70	1404	9	9	9
query71	398	346	355	346
query72	6968	11	9	9
query73	513	10	9	9
query74	9154	9	9	9
query75	3437	11	10	10
query76	3133	11	10	10
query77	864	12	11	11
query78	9677	15	8	8
query79	999	9	10	9
query80	719	12	12	12
query81	574	10	9	9
query82	207	11	11	11
query83	320	12	11	11
query84	328	11	10	10
query85	1001	10	11	10
query86	356	10	10	10
query87	3896	9	8	8
query88	2816	12	10	10
query89	398	12	13	12
query90	1989	9	8	8
query91	196	9	9	9
query92	88	12	10	10
query93	1031	10	9	9
query94	811	11	9	9
query95	995	11	9	9
query96	437	10	10	10
query97	3055	10	8	8
query98	248	235	219	219
query99	1458	11	10	10
Total cold run time: 283740 ms
Total hot run time: 2734 ms

doris-robot · 2025-09-18T15:51:11Z

ClickBench: Total hot run time: 0.08 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6fb0993bcc087a973b7b5fad09659687762b3673, data reload: false

query1	0.07	0.02	0.01
query2	0.10	0.00	0.00
query3	0.27	0.01	0.01
query4	1.76	0.00	0.00
query5	0.29	0.01	0.01
query6	1.68	0.01	0.00
query7	0.06	0.01	0.00
query8	0.06	0.01	0.00
query9	0.63	0.00	0.00
query10	0.59	0.01	0.00
query11	0.18	0.01	0.01
query12	0.16	0.00	0.00
query13	0.65	0.00	0.00
query14	1.04	0.01	0.01
query15	0.89	0.00	0.01
query16	0.40	0.00	0.00
query17	1.05	0.00	0.00
query18	0.22	0.01	0.00
query19	1.93	0.01	0.00
query20	0.02	0.00	0.00
query21	15.36	0.00	0.00
query22	6.19	0.00	0.01
query23	16.34	0.00	0.00
query24	1.50	0.00	0.01
query25	0.22	0.01	0.01
query26	0.19	0.01	0.00
query27	0.12	0.01	0.01
query28	1.17	0.01	0.00
query29	13.29	0.01	0.00
query30	0.30	0.01	0.00
query31	2.26	0.00	0.00
query32	5.88	0.00	0.00
query33	4.31	0.00	0.01
query34	7.37	0.01	0.00
query35	6.23	0.00	0.01
query36	0.68	0.00	0.00
query37	0.11	0.00	0.00
query38	0.08	0.00	0.00
query39	0.04	0.00	0.00
query40	0.19	0.01	0.00
query41	0.09	0.01	0.01
query42	0.06	0.00	0.01
query43	0.05	0.00	0.01
Total cold run time: 94.08 s
Total hot run time: 0.08 s

hello-stephen · 2025-09-18T17:27:49Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.34% (17594/33614)
Line Coverage	37.55% (159713/425368)
Region Coverage	32.09% (121643/379074)
Branch Coverage	33.43% (53360/159599)

hello-stephen · 2025-09-18T18:29:58Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.08% (23353/32853)
Line Coverage	57.44% (243833/424475)
Region Coverage	52.92% (203361/384250)
Branch Coverage	54.47% (87329/160336)

freemandealer · 2025-09-19T02:17:30Z

run p0

hello-stephen · 2025-09-19T04:31:36Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.08% (23353/32853)
Line Coverage	57.44% (243838/424475)
Region Coverage	52.93% (203372/384250)
Branch Coverage	54.47% (87334/160336)

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

freemandealer · 2025-09-19T13:02:38Z

run buildall

github-actions · 2025-09-19T15:07:23Z

PR approved by at least one committer and no changes requested.

freemandealer · 2025-09-20T06:44:23Z

run buildall

hello-stephen · 2025-09-20T09:34:46Z

BE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.34% (17596/33618)
Line Coverage	37.56% (159694/425185)
Region Coverage	32.12% (121700/378946)
Branch Coverage	33.46% (53330/159387)

hello-stephen · 2025-09-20T10:27:33Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.11% (23363/32857)
Line Coverage	57.48% (243888/424292)
Region Coverage	52.96% (203435/384120)
Branch Coverage	54.51% (87279/160124)

deardeng

LGTM

freemandealer · 2025-09-22T03:03:13Z

run buildall

doris-robot · 2025-09-22T06:18:54Z

BE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.40% (17622/33627)
Line Coverage	37.62% (159971/425195)
Region Coverage	32.12% (121721/378994)
Branch Coverage	33.48% (53385/159430)

hello-stephen · 2025-09-22T06:59:33Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.19% (23397/32866)
Line Coverage	57.55% (244174/424303)
Region Coverage	52.87% (203128/384168)
Branch Coverage	54.50% (87297/160167)

freemandealer · 2025-09-22T07:04:31Z

run performance

dataroaring

LGTM

github-actions · 2025-09-24T03:40:26Z

PR approved by at least one committer and no changes requested.

[fix](filecache) microbench crash when FileBufferBuilder

6fb0993

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

freemandealer requested review from dataroaring and gavinchou as code owners September 18, 2025 14:24

dataroaring previously approved these changes Sep 18, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2025

github-actions bot added the reviewed label Sep 18, 2025

fix memtracker

db9cccf

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

freemandealer dismissed dataroaring’s stale review via db9cccf September 19, 2025 13:02

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 19, 2025

gavinchou previously approved these changes Sep 19, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2025

fix compile

142420a

freemandealer dismissed gavinchou’s stale review via 142420a September 20, 2025 06:44

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 20, 2025

deardeng approved these changes Sep 22, 2025

View reviewed changes

Merge branch 'master' into fix-microbench

a546108

dataroaring approved these changes Sep 24, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 24, 2025

dataroaring merged commit 011b55b into apache:master Sep 24, 2025
25 of 27 checks passed

[fix](filecache) microbench crash when FileBufferBuilder #56204

[fix](filecache) microbench crash when FileBufferBuilder #56204

Uh oh!

Conversation

freemandealer commented Sep 18, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 18, 2025

Uh oh!

freemandealer commented Sep 18, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 18, 2025

Uh oh!

github-actions bot commented Sep 18, 2025

Uh oh!

doris-robot commented Sep 18, 2025

Uh oh!

doris-robot commented Sep 18, 2025

Uh oh!

doris-robot commented Sep 18, 2025

Uh oh!

hello-stephen commented Sep 18, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 18, 2025

BE Regression && UT Coverage Report

Uh oh!

freemandealer commented Sep 19, 2025

Uh oh!

hello-stephen commented Sep 19, 2025

BE Regression && UT Coverage Report

Uh oh!

freemandealer commented Sep 19, 2025

Uh oh!

github-actions bot commented Sep 19, 2025

Uh oh!

freemandealer commented Sep 20, 2025

Uh oh!

hello-stephen commented Sep 20, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 20, 2025

BE Regression && UT Coverage Report

Uh oh!

deardeng left a comment

Choose a reason for hiding this comment

Uh oh!

freemandealer commented Sep 22, 2025

Uh oh!

doris-robot commented Sep 22, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 22, 2025

BE Regression && UT Coverage Report

Uh oh!

freemandealer commented Sep 22, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants