Skip to content

branch-4.0: [Fix](warmup) Fix coredump in CloudTablet::complete_rowset_segment_warmup due to capture by reference #56395 #56458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Sep 26, 2025
Merged

branch-4.0: [Fix](warmup) Fix coredump in CloudTablet::complete_rowset_segment_warmup due to capture by reference #56395 #56458

dataroaring merged 1 commit into from
Sep 26, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Sep 25, 2025

Cherry-picked from #56395

@bobhan1
[Fix](warmup) Fix coredump in `CloudTablet::complete_rowset_segment_w…
c435963 
…armup` due to capture by reference (#56395)

### What problem does this PR solve?
introduced in #54611

```
(gdb) bt
#0  __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ./nptl/pthread_sigmask.c:43
#1  0x00007f7aa6b5d71e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#2  0x00007f7aa6b5e206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#3  <signal handler called>
#4  0x000055e20050d443 in std::_Hashtable<doris::RowsetId, std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> >, std::__detail::_Select1st, std::equal_to<doris::RowsetId>, std::hash<doris::RowsetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node (this=0x7f79e69acde0,
    __bkt=4186920012728959759, __k=..., __code=11200260987994981938) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:2205
#5  std::_Hashtable<doris::RowsetId, std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> >, std::__detail::_Select1st, std::equal_to<doris::RowsetId>, std::hash<doris::RowsetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_locate (this=this@entry=0x7f79e69acde0, __k=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:2283
#6  0x000055e200505a8d in std::_Hashtable<doris::RowsetId, std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> >, std::__detail::_Select1st, std::equal_to<doris::RowsetId>, std::hash<doris::RowsetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find (this=0x7f79e69acde0, __k=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/hashtable.h:1929
#7  std::unordered_map<doris::RowsetId, doris::CloudTablet::RowsetWarmUpInfo, std::hash<doris::RowsetId>, std::equal_to<doris::RowsetId>, std::allocator<std::pair<doris::RowsetId const, doris::CloudTablet::RowsetWarmUpInfo> > >::contains (this=0x7f79e69acde0, __x=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unordered_map.h:999
#8  doris::CloudTablet::complete_rowset_segment_warmup (this=<optimized out>, rowset_id=..., status=..., segment_num=1, inverted_idx_num=0) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_tablet.cpp:1692
#9  0x000055e2005338ba in doris::CloudWarmUpManager::handle_jobs()::$_0::operator()(doris::Status) const (this=0x7f79e8bebad0, st=...) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_warm_up_manager.cpp:243
#10 std::__invoke_impl<void, doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status>(std::__invoke_other, doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status&&) (__f=..., __args=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
#11 std::__invoke_r<void, doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status>(doris::CloudWarmUpManager::handle_jobs()::$_0&, doris::Status&&) (__fn=..., __args=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113
#12 std::_Function_handler<void (doris::Status), doris::CloudWarmUpManager::handle_jobs()::$_0>::_M_invoke(std::_Any_data const&, doris::Status&&) (__functor=..., __args=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
#13 0x000055e200533310 in std::function<void (doris::Status)>::operator()(doris::Status) const (this=0x7f795a3ff1d0, __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
#14 doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0::operator()(doris::Status) const (
    this=0x7f79f0710b60, st=...) at /home/zcp/repo_center/doris_master/doris/be/src/cloud/cloud_warm_up_manager.cpp:149
#15 std::__invoke_impl<void, doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status>(std::__invoke_other, doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status&&) (
    __f=..., __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:63
#16 std::__invoke_r<void, doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status>(doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0&, doris::Status&&) (__fn=..., __args=...)
    at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/invoke.h:113
#17 std::_Function_handler<void (doris::Status), doris::CloudWarmUpManager::submit_download_tasks(std::filesystem::__cxx11::path, long, std::shared_ptr<doris::io::FileSystem>, long, std::shared_ptr<bthread::CountdownEvent>, bool, std::function<void (doris::Status)>)::$_0>::_M_invoke(std::_Any_data const&, doris::Status&&) (__functor=..., __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:292
#18 0x000055e200469cb1 in std::function<void (doris::Status)>::operator()(doris::Status) const (this=0x7f796941cee0, __args=...) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
#19 doris::io::FileCacheBlockDownloader::download_segment_file (this=<optimized out>, meta=...) at /home/zcp/repo_center/doris_master/doris/be/src/io/cache/block_file_cache_downloader.cpp:297
#20 0x000055e1fbfcbe85 in doris::ThreadPool::dispatch_thread (this=0x7f7995a1da00) at /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:614
#21 0x000055e1fbfc0fac in std::function<void ()>::operator()() const (this=0x7f796941cee0) at /usr/local/ldb-toolchain-v0.26/bin/../lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/std_function.h:593
#22 doris::Thread::supervise_thread (arg=0x7f79f13de010) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:460
#23 0x00007f7aa5bf8ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#24 0x00007f7aa5c8a850 in __closefrom_fallback (from=1685031872, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:45
#25 0x0000000000000000 in ?? ()
```

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@Thearas
Copy link
Contributor

Thearas commented Sep 25, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Sep 25, 2025
@Thearas
Copy link
Contributor

Thearas commented Sep 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 25, 2025

ClickBench: Total hot run time: 30.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c435963a6d021edd3b8047539c5a55a4f64323c3, data reload: true

query1	0.04	0.03	0.03
query2	0.11	0.04	0.04
query3	0.28	0.07	0.07
query4	1.72	0.11	0.11
query5	0.26	0.23	0.26
query6	1.24	0.64	0.65
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.59	0.52	0.51
query10	0.57	0.57	0.57
query11	0.17	0.11	0.11
query12	0.15	0.12	0.11
query13	0.64	0.61	0.61
query14	0.80	0.82	0.82
query15	0.87	0.86	0.84
query16	0.40	0.39	0.41
query17	1.05	1.10	1.08
query18	0.20	0.19	0.20
query19	1.92	1.83	1.82
query20	0.03	0.01	0.02
query21	15.45	0.92	0.57
query22	0.79	1.04	0.88
query23	15.05	1.35	0.60
query24	16.29	1.54	0.62
query25	2.24	0.39	0.20
query26	0.19	0.18	0.18
query27	0.21	0.20	0.20
query28	1.78	1.02	0.93
query29	12.75	3.86	3.25
query30	0.31	0.12	0.10
query31	3.01	0.58	0.38
query32	3.33	0.57	0.47
query33	3.09	3.13	3.06
query34	16.75	5.41	4.83
query35	5.14	5.16	5.19
query36	0.71	0.57	0.54
query37	0.11	0.07	0.07
query38	0.06	0.04	0.03
query39	0.04	0.02	0.03
query40	0.19	0.16	0.17
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.07	0.03	0.04
Total cold run time: 108.81 s
Total hot run time: 30.57 s

@dataroaring dataroaring merged commit 44ee3b7 into branch-4.0 Sep 26, 2025
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou Awaiting requested review from gavinchou

@dataroaring dataroaring Awaiting requested review from dataroaring

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @bobhan1