Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](memory) Refactor MemCounter #40542

Merged
merged 4 commits into from
Sep 16, 2024
Merged

[fix](memory) Refactor MemCounter #40542

merged 4 commits into from
Sep 16, 2024

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Sep 9, 2024 •
edited
Loading

  1. Add a new class MemCounter.
  2. MemTracker and MemTrackerLimiter no longer have inheritance and dependencies.
  3. MemTrackerLimiter is used to count process memory, the BE web page /mem_tracker also only displays MemTrackerLimiter.
  4. MemTracker is used to count the memory of operators and some data structures. It is not counted in the process memory and is used for logic control and profile.

In addition, it seems that the crash is caused by memory abnormality, not sure whether this PR can fix the problem, but this will help locate the problem and more elegant.

==6641==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000980e70 at pc 0x559be21880be bp 0x7fd7cfc75070 sp 0x7fd7cfc75068
READ of size 8 at 0x603000980e70 thread T1453 (memory_maintena)
    #0 0x559be21880bd in std::__atomic_base<long>::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
    #1 0x559be21880bd in doris::MemTracker::MemCounter::current_value() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:139:63
    #2 0x559be21880bd in doris::MemTracker::consumption() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:165:56
    #3 0x559be3985133 in doris::MemTrackerLimiter::refresh_global_counter() /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker_limiter.cpp:245:59
    #4 0x559bdfcd52d5 in doris::Daemon::memory_maintenance_thread() /home/zcp/repo_center/doris_master/doris/be/src/common/daemon.cpp:239:13
    #5 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
    #6 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8
    #7 0x7fdf5647584f  misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

0x603000980e74 is located 0 bytes after 20-byte region [0x603000980e60,0x603000980e74)
04:19:48   freed by thread T1491 (Scan_normal [wo) here:
04:19:48       #0 0x559bdfb0dd9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x33546d9d) (BuildId: f170e92ad3c55512)
04:19:48       #1 0x559bdfb1d42b in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:139:2
04:19:48       #2 0x559bdfb1d42b in std::allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:187:27
04:19:48       #3 0x559bdfb1d42b in std::allocator_traits<std::allocator<char>>::deallocate(std::allocator<char>&, char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:492:13
04:19:48       #4 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_destroy(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:237:9
04:19:48       #5 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_dispose() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:232:4
04:19:48       #6 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::~basic_string() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:658:9
04:19:48       #7 0x559bdfb22501 in void std::destroy_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
04:19:48       #8 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
04:19:48       #9 0x559bdfb22501 in void std::_Destroy_aux<false>::__destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
04:19:48       #10 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
04:19:48       #11 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
04:19:48       #12 0x559bdfb22501 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
04:19:48       #13 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9867:1
04:19:48       #14 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9866:36
04:19:48       #15 0x559be0047ee6 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:372:5
04:19:48       #16 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13
04:19:48       #17 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28
04:19:48       #18 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18
04:19:48       #19 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18
04:19:48       #20 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5
04:19:48       #21 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32
04:19:48       #22 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
04:19:48       #23 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
04:19:48       #24 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
04:19:48       #25 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
04:19:48       #26 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
04:19:48       #27 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
04:19:48       #28 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24
04:19:48       #29 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
04:19:48       #30 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8
04:19:48   
04:19:48   previously allocated by thread T1491 (Scan_normal [wo) here:
04:19:48       #0 0x559bdfb0d53d in operator new(unsigned long) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x3354653d) (BuildId: f170e92ad3c55512)
04:19:48       #1 0x559bdfb33f74 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:219:14
04:19:48       #2 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct_aux<char*>(char*, char*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11
04:19:48       #3 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4
04:19:48       #4 0x559bdfb32705 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:451:9
04:19:48       #5 0x559bdfc326e6 in decltype(::new((void*)(0)) std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>(std::declval<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>())) std::construct_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39
04:19:48       #6 0x559bdfc326e6 in void std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:514:4
04:19:48       #7 0x559bdfc326e6 in void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:449:4
04:19:48       #8 0x559be0032206 in doris::DeleteHandler::parse_condition(doris::DeleteSubPredicatePB const&, doris::TCondition*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:300:33
04:19:48       #9 0x559be0047aa3 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:355:9
04:19:48       #10 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13
04:19:48       #11 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28
04:19:48       #12 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18
04:19:48       #13 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18
04:19:48       #14 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5
04:19:48       #15 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32
04:19:48       #16 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
04:19:48       #17 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
04:19:48       #18 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
04:19:48       #19 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
04:19:48       #20 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
04:19:48       #21 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
04:19:48       #22 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24
04:19:48       #23 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
04:19:48       #24 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Sep 9, 2024

run buildall

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Sep 9, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.83% (9384/25478)
Line Coverage: 28.23% (77398/274167)
Region Coverage: 27.63% (39968/144647)
Branch Coverage: 24.25% (20329/83814)
Coverage Report: http://coverage.selectdb-in.cc/coverage/39ee80fccbb2c12c5a6237b40b69552ccfea5bb2_39ee80fccbb2c12c5a6237b40b69552ccfea5bb2/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38156 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39ee80fccbb2c12c5a6237b40b69552ccfea5bb2, data reload: false

------ Round 1 ----------------------------------
q1	17676	5093	4324	4324
q2	2026	189	190	189
q3	11651	983	1163	983
q4	10510	766	825	766
q5	7741	2915	2881	2881
q6	228	137	136	136
q7	960	619	593	593
q8	9312	2103	2119	2103
q9	7269	6559	6586	6559
q10	7002	2208	2186	2186
q11	469	241	242	241
q12	398	228	232	228
q13	17781	3089	3080	3080
q14	294	240	237	237
q15	541	492	470	470
q16	525	432	434	432
q17	992	681	746	681
q18	7433	7006	6916	6916
q19	1397	980	1027	980
q20	721	330	333	330
q21	3919	3140	2835	2835
q22	1107	1010	1006	1006
Total cold run time: 109952 ms
Total hot run time: 38156 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4379	4304	4279	4279
q2	388	282	257	257
q3	2911	2676	2639	2639
q4	1955	1689	1651	1651
q5	5710	5704	5788	5704
q6	233	134	133	133
q7	2198	1827	1844	1827
q8	3326	3448	3518	3448
q9	8950	8908	8885	8885
q10	3583	3367	3373	3367
q11	616	523	522	522
q12	836	652	655	652
q13	13715	3183	3261	3183
q14	331	287	303	287
q15	528	505	492	492
q16	560	483	496	483
q17	1867	1544	1536	1536
q18	8250	7982	8044	7982
q19	1742	1594	1423	1423
q20	2154	1906	1887	1887
q21	5790	5559	5577	5559
q22	1122	1032	1040	1032
Total cold run time: 71144 ms
Total hot run time: 57228 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192332 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39ee80fccbb2c12c5a6237b40b69552ccfea5bb2, data reload: false

query1	1256	898	904	898
query2	6385	1929	1851	1851
query3	10601	4081	4002	4002
query4	59353	24292	23086	23086
query5	5486	515	496	496
query6	416	164	169	164
query7	5900	303	297	297
query8	325	230	228	228
query9	9007	2489	2479	2479
query10	505	278	264	264
query11	19028	15048	15165	15048
query12	164	103	125	103
query13	1595	382	387	382
query14	11323	7337	7299	7299
query15	233	169	171	169
query16	7613	491	451	451
query17	1195	586	550	550
query18	1994	302	300	300
query19	300	143	147	143
query20	118	111	111	111
query21	199	101	105	101
query22	4702	4586	4373	4373
query23	34418	33571	33501	33501
query24	5863	2938	2944	2938
query25	518	380	383	380
query26	688	151	149	149
query27	1782	275	276	275
query28	3718	2064	2056	2056
query29	697	403	414	403
query30	238	150	149	149
query31	934	743	774	743
query32	88	50	52	50
query33	461	291	276	276
query34	883	488	483	483
query35	798	730	715	715
query36	1058	910	945	910
query37	146	87	82	82
query38	3940	3883	3843	3843
query39	1425	1391	1392	1391
query40	195	118	115	115
query41	48	44	46	44
query42	116	100	98	98
query43	501	475	456	456
query44	1110	749	737	737
query45	207	170	174	170
query46	1087	763	730	730
query47	1881	1753	1818	1753
query48	368	298	295	295
query49	800	470	468	468
query50	836	417	430	417
query51	7068	6974	6872	6872
query52	100	88	87	87
query53	261	185	184	184
query54	577	462	462	462
query55	80	75	75	75
query56	286	313	260	260
query57	1207	1097	1042	1042
query58	213	238	228	228
query59	3009	2883	2785	2785
query60	292	261	258	258
query61	103	96	100	96
query62	734	661	656	656
query63	221	186	191	186
query64	2836	670	719	670
query65	3240	3123	3144	3123
query66	678	342	341	341
query67	15490	15322	15125	15125
query68	2989	594	582	582
query69	397	289	292	289
query70	1175	1061	1144	1061
query71	348	274	271	271
query72	5323	4032	3979	3979
query73	762	320	324	320
query74	9235	8780	8711	8711
query75	3352	2707	2691	2691
query76	1376	1002	1010	1002
query77	525	335	316	316
query78	10547	9104	9023	9023
query79	1045	545	536	536
query80	751	502	515	502
query81	516	237	231	231
query82	250	140	144	140
query83	177	151	150	150
query84	254	77	135	77
query85	742	292	279	279
query86	320	297	295	295
query87	4434	4189	4301	4189
query88	3120	2287	2279	2279
query89	384	280	284	280
query90	1781	190	189	189
query91	126	98	100	98
query92	59	49	50	49
query93	1092	541	539	539
query94	598	855	301	301
query95	346	255	255	255
query96	591	266	267	266
query97	3179	3059	3082	3059
query98	214	210	201	201
query99	1655	1300	1304	1300
Total cold run time: 306560 ms
Total hot run time: 192332 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39ee80fccbb2c12c5a6237b40b69552ccfea5bb2, data reload: false

query1	0.04	0.04	0.04
query2	0.09	0.04	0.05
query3	0.23	0.05	0.06
query4	1.67	0.09	0.08
query5	0.51	0.50	0.48
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.56	0.55	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	1.44	1.40	1.43
query15	0.83	0.81	0.81
query16	0.35	0.37	0.36
query17	1.07	1.06	1.00
query18	0.21	0.20	0.21
query19	1.89	1.72	1.88
query20	0.02	0.01	0.01
query21	15.38	0.67	0.65
query22	3.76	7.60	1.96
query23	18.29	1.38	1.30
query24	2.09	0.25	0.22
query25	0.17	0.08	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.18	1.03	0.99
query29	12.67	3.36	3.32
query30	0.24	0.06	0.05
query31	2.88	0.40	0.40
query32	3.27	0.48	0.48
query33	2.96	3.01	3.01
query34	17.02	4.39	4.35
query35	4.47	4.38	4.44
query36	0.67	0.48	0.48
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.03
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.71 s
Total hot run time: 31.34 s

@xinyiZzz
Copy link
Contributor Author

run buildall

1 similar comment
@xinyiZzz
Copy link
Contributor Author

run buildall

@xinyiZzz xinyiZzz changed the title [fix](memory) Fix MemCounter null pointer [fix](memory) Refactor MemCounter Sep 13, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 43018 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7b2b7faa5c2e4b8a3881e8cc4766a6cd385f9ca0, data reload: false

------ Round 1 ----------------------------------
q1	17610	7250	7222	7222
q2	2055	188	179	179
q3	10517	1261	1375	1261
q4	10276	985	1028	985
q5	7748	3208	3168	3168
q6	238	148	146	146
q7	1038	633	628	628
q8	9453	2048	2066	2048
q9	6746	6313	6296	6296
q10	7035	2516	2518	2516
q11	423	262	258	258
q12	409	235	228	228
q13	17771	3021	3004	3004
q14	286	245	259	245
q15	570	519	533	519
q16	517	431	438	431
q17	1016	964	980	964
q18	7511	6874	6849	6849
q19	1380	1240	1236	1236
q20	613	333	332	332
q21	3944	3518	3558	3518
q22	1094	998	985	985
Total cold run time: 108250 ms
Total hot run time: 43018 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7167	7272	7147	7147
q2	352	241	232	232
q3	3099	3101	3036	3036
q4	2097	2119	2070	2070
q5	5674	5606	5745	5606
q6	249	151	147	147
q7	2145	1763	1775	1763
q8	3390	3437	3423	3423
q9	8735	8964	8840	8840
q10	3505	3602	3603	3602
q11	579	473	474	473
q12	803	651	659	651
q13	9370	3191	3210	3191
q14	331	290	280	280
q15	594	537	538	537
q16	509	473	450	450
q17	1817	1744	1731	1731
q18	8584	8073	8087	8073
q19	1797	1783	1759	1759
q20	2124	1898	1885	1885
q21	6005	5586	5701	5586
q22	1105	1044	1070	1044
Total cold run time: 70031 ms
Total hot run time: 61526 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199505 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7b2b7faa5c2e4b8a3881e8cc4766a6cd385f9ca0, data reload: false

query1	1259	892	869	869
query2	6417	1797	1717	1717
query3	10661	3901	3864	3864
query4	55671	25138	23886	23886
query5	5003	542	534	534
query6	310	171	164	164
query7	5625	305	294	294
query8	270	222	222	222
query9	6106	2584	2593	2584
query10	391	293	276	276
query11	16089	15504	15648	15504
query12	155	100	101	100
query13	1437	384	367	367
query14	10315	6812	7271	6812
query15	207	175	188	175
query16	7007	507	470	470
query17	1194	603	587	587
query18	1846	308	304	304
query19	220	157	153	153
query20	126	118	120	118
query21	214	107	110	107
query22	4815	4569	4396	4396
query23	34808	34223	33894	33894
query24	6180	3131	3117	3117
query25	529	410	411	410
query26	609	156	157	156
query27	1562	279	279	279
query28	2829	2107	2085	2085
query29	699	428	433	428
query30	225	158	155	155
query31	962	777	753	753
query32	75	55	54	54
query33	480	304	294	294
query34	901	485	466	466
query35	847	784	714	714
query36	1063	914	953	914
query37	140	89	85	85
query38	3986	3983	3972	3972
query39	1461	1394	1400	1394
query40	210	117	117	117
query41	47	50	46	46
query42	123	95	98	95
query43	486	439	453	439
query44	1174	786	778	778
query45	201	172	169	169
query46	1115	847	822	822
query47	1929	1852	1835	1835
query48	370	295	285	285
query49	748	522	433	433
query50	925	430	426	426
query51	7074	6917	6909	6909
query52	98	86	86	86
query53	253	176	176	176
query54	565	455	470	455
query55	78	72	76	72
query56	271	269	260	260
query57	1228	1096	1112	1096
query58	219	237	249	237
query59	2773	2573	2903	2573
query60	290	277	284	277
query61	99	94	93	93
query62	799	673	668	668
query63	218	188	186	186
query64	1668	692	642	642
query65	3258	3161	3146	3146
query66	633	307	303	303
query67	16081	15702	15424	15424
query68	2026	866	898	866
query69	433	326	330	326
query70	1158	1162	1180	1162
query71	346	343	340	340
query72	5137	3372	3387	3372
query73	605	578	583	578
query74	9316	9408	9210	9210
query75	3147	3014	3026	3014
query76	951	846	846	846
query77	517	432	395	395
query78	9621	9365	9267	9267
query79	927	912	873	873
query80	827	858	817	817
query81	460	260	266	260
query82	265	267	262	262
query83	191	190	192	190
query84	235	106	105	105
query85	627	428	388	388
query86	317	324	310	310
query87	4399	4493	4361	4361
query88	4112	4046	4019	4019
query89	375	370	367	367
query90	1379	311	305	305
query91	120	123	127	123
query92	83	78	79	78
query93	1056	1044	1044	1044
query94	534	364	386	364
query95	448	431	425	425
query96	470	466	469	466
query97	3199	3118	3160	3118
query98	231	234	230	230
query99	1669	1303	1310	1303
Total cold run time: 290808 ms
Total hot run time: 199505 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7b2b7faa5c2e4b8a3881e8cc4766a6cd385f9ca0, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.04
query4	1.67	0.07	0.06
query5	0.50	0.50	0.50
query6	1.13	0.73	0.72
query7	0.03	0.02	0.01
query8	0.06	0.05	0.05
query9	0.55	0.51	0.51
query10	0.58	0.57	0.56
query11	0.18	0.14	0.12
query12	0.16	0.13	0.12
query13	0.63	0.61	0.61
query14	1.47	1.45	1.51
query15	0.90	0.88	0.89
query16	0.37	0.37	0.36
query17	1.02	1.03	1.02
query18	0.17	0.16	0.17
query19	1.88	1.78	1.82
query20	0.01	0.00	0.01
query21	15.42	0.67	0.66
query22	4.58	7.98	1.18
query23	17.85	1.38	1.39
query24	2.28	0.22	0.22
query25	0.18	0.08	0.09
query26	0.29	0.19	0.18
query27	0.09	0.07	0.08
query28	13.19	1.12	1.11
query29	12.59	3.35	3.35
query30	0.24	0.06	0.05
query31	2.87	0.42	0.42
query32	3.22	0.49	0.51
query33	3.00	3.04	3.06
query34	15.48	4.33	4.31
query35	4.33	4.34	4.35
query36	0.69	0.49	0.48
query37	0.18	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.05
Total cold run time: 108.7 s
Total hot run time: 30.99 s

@xinyiZzz
Copy link
Contributor Author

run buildall

@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 43210 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01933ec23d7816e11e91841b7fe4b0ad8826aabf, data reload: false

------ Round 1 ----------------------------------
q1	17603	7436	7232	7232
q2	2034	190	193	190
q3	10815	1269	1357	1269
q4	10253	966	1036	966
q5	7710	3191	3154	3154
q6	246	153	158	153
q7	1047	642	639	639
q8	9482	2041	2048	2041
q9	6772	6347	6294	6294
q10	7031	2501	2636	2501
q11	433	254	262	254
q12	417	235	239	235
q13	17770	3045	3038	3038
q14	284	249	252	249
q15	583	522	522	522
q16	674	614	635	614
q17	992	950	969	950
q18	7508	6825	6955	6825
q19	1376	1244	1240	1240
q20	617	342	320	320
q21	3938	3537	3591	3537
q22	1095	1006	987	987
Total cold run time: 108680 ms
Total hot run time: 43210 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7175	7215	7226	7215
q2	356	252	244	244
q3	3076	3138	3053	3053
q4	2086	2149	2029	2029
q5	5759	5605	5784	5605
q6	244	154	156	154
q7	2196	1784	1819	1784
q8	3394	3481	3465	3465
q9	8924	8876	8861	8861
q10	3478	3611	3585	3585
q11	584	496	478	478
q12	819	627	611	611
q13	8815	3210	3207	3207
q14	320	310	291	291
q15	586	545	541	541
q16	696	679	656	656
q17	1845	1793	1786	1786
q18	8642	8169	8019	8019
q19	1788	1758	1745	1745
q20	2137	1904	1878	1878
q21	6063	5696	5751	5696
q22	1106	1044	1009	1009
Total cold run time: 70089 ms
Total hot run time: 61912 ms

@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 43483 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 12045c6b817aea4755ba2c37fdf77656d40f98d4, data reload: false

------ Round 1 ----------------------------------
q1	18000	7381	7268	7268
q2	2629	220	189	189
q3	11389	1408	1399	1399
q4	10331	1029	1035	1029
q5	7770	3276	3228	3228
q6	255	156	160	156
q7	1078	659	641	641
q8	9557	2064	2050	2050
q9	6851	6293	6293	6293
q10	7024	2534	2497	2497
q11	429	251	253	251
q12	411	230	226	226
q13	17754	3034	3036	3034
q14	281	247	251	247
q15	573	535	520	520
q16	654	613	605	605
q17	986	956	952	952
q18	7503	6815	6849	6815
q19	1385	1257	1242	1242
q20	627	337	328	328
q21	3898	3531	3525	3525
q22	1100	988	989	988
Total cold run time: 110485 ms
Total hot run time: 43483 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7183	7184	7258	7184
q2	338	235	224	224
q3	2944	2889	2902	2889
q4	1957	1952	1985	1952
q5	5446	5388	5437	5388
q6	240	150	149	149
q7	2036	1664	1681	1664
q8	3244	3337	3281	3281
q9	8394	8418	8481	8418
q10	3394	3469	3452	3452
q11	573	471	474	471
q12	786	585	593	585
q13	5672	3063	3046	3046
q14	293	269	265	265
q15	570	518	522	518
q16	666	644	643	643
q17	1750	1724	1704	1704
q18	7845	7606	7629	7606
q19	1715	1702	1690	1690
q20	2053	1807	1819	1807
q21	5723	5460	5400	5400
q22	1082	1011	997	997
Total cold run time: 63904 ms
Total hot run time: 59333 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195746 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 12045c6b817aea4755ba2c37fdf77656d40f98d4, data reload: false

query1	904	394	394	394
query2	6489	1961	1885	1885
query3	6655	217	223	217
query4	27830	23928	24155	23928
query5	5362	532	559	532
query6	274	188	173	173
query7	4599	309	313	309
query8	297	231	224	224
query9	8537	2621	2639	2621
query10	487	295	281	281
query11	16217	15526	15837	15526
query12	168	105	106	105
query13	1708	408	394	394
query14	11571	6368	6907	6368
query15	212	170	179	170
query16	7526	457	447	447
query17	1550	572	543	543
query18	1824	286	293	286
query19	193	146	153	146
query20	118	111	112	111
query21	213	107	106	106
query22	4363	4393	4246	4246
query23	34372	34094	33813	33813
query24	9642	3073	3039	3039
query25	611	397	423	397
query26	720	155	158	155
query27	2145	281	282	281
query28	5805	2462	2423	2423
query29	870	411	410	410
query30	311	162	154	154
query31	1004	809	812	809
query32	107	56	59	56
query33	733	307	289	289
query34	913	493	486	486
query35	861	727	718	718
query36	1068	914	912	912
query37	164	102	90	90
query38	3981	3950	4000	3950
query39	1489	1458	1393	1393
query40	206	117	114	114
query41	49	48	46	46
query42	123	98	100	98
query43	499	448	496	448
query44	1247	822	783	783
query45	198	168	172	168
query46	1123	833	826	826
query47	1898	1792	1793	1792
query48	382	305	299	299
query49	1080	480	453	453
query50	914	439	435	435
query51	7010	6895	6863	6863
query52	100	93	94	93
query53	258	192	188	188
query54	779	471	474	471
query55	83	81	79	79
query56	297	277	287	277
query57	1217	1079	1085	1079
query58	253	245	247	245
query59	2882	2686	2703	2686
query60	328	298	292	292
query61	140	126	126	126
query62	911	662	675	662
query63	223	194	194	194
query64	4808	785	769	769
query65	3281	3190	3186	3186
query66	1367	308	315	308
query67	15980	15546	15305	15305
query68	3459	876	880	876
query69	448	340	339	339
query70	1167	1140	1153	1140
query71	362	358	344	344
query72	6246	3497	3617	3497
query73	650	593	591	591
query74	9272	9236	9074	9074
query75	3127	2992	2978	2978
query76	1904	847	845	845
query77	463	407	407	407
query78	10074	9289	9261	9261
query79	925	898	890	890
query80	886	855	838	838
query81	459	281	284	281
query82	416	402	403	402
query83	197	198	223	198
query84	234	111	113	111
query85	651	413	401	401
query86	330	333	307	307
query87	4377	4309	4440	4309
query88	4194	4126	4091	4091
query89	390	376	376	376
query90	1300	321	321	321
query91	136	134	135	134
query92	80	75	74	74
query93	1045	1047	1048	1047
query94	613	414	394	394
query95	462	428	436	428
query96	476	525	475	475
query97	3142	3098	3097	3097
query98	240	233	230	230
query99	1563	1321	1318	1318
Total cold run time: 278817 ms
Total hot run time: 195746 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.89% (9462/25648)
Line Coverage: 28.25% (77775/275344)
Region Coverage: 27.66% (40183/145265)
Branch Coverage: 24.27% (20417/84120)
Coverage Report: http://coverage.selectdb-in.cc/coverage/12045c6b817aea4755ba2c37fdf77656d40f98d4_12045c6b817aea4755ba2c37fdf77656d40f98d4/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 32.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 12045c6b817aea4755ba2c37fdf77656d40f98d4, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.06	0.07
query5	0.49	0.49	0.49
query6	1.14	0.72	0.72
query7	0.01	0.01	0.01
query8	0.06	0.05	0.05
query9	0.58	0.52	0.51
query10	0.58	0.58	0.57
query11	0.17	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.61
query14	2.99	2.97	3.08
query15	0.94	0.89	0.89
query16	0.37	0.36	0.37
query17	1.02	1.02	1.03
query18	0.23	0.20	0.20
query19	1.91	1.97	1.76
query20	0.01	0.01	0.02
query21	15.35	0.70	0.69
query22	4.01	8.28	1.26
query23	18.05	1.29	1.39
query24	2.18	0.24	0.22
query25	0.19	0.09	0.08
query26	0.31	0.18	0.19
query27	0.08	0.08	0.08
query28	13.18	1.15	1.10
query29	12.52	3.35	3.31
query30	0.26	0.05	0.05
query31	2.89	0.42	0.41
query32	3.25	0.49	0.50
query33	2.99	3.04	3.07
query34	15.42	4.32	4.33
query35	4.32	4.33	4.35
query36	0.68	0.49	0.48
query37	0.19	0.17	0.16
query38	0.16	0.14	0.16
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 109.75 s
Total hot run time: 32.52 s

@yiguolei
Copy link
Contributor

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.91% (9468/25652)
Line Coverage: 28.26% (77836/275380)
Region Coverage: 27.68% (40216/145286)
Branch Coverage: 24.29% (20432/84132)
Coverage Report: http://coverage.selectdb-in.cc/coverage/30b4aa67da0991be433420aabfa8578b68438f24_30b4aa67da0991be433420aabfa8578b68438f24/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 43222 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30b4aa67da0991be433420aabfa8578b68438f24, data reload: false

------ Round 1 ----------------------------------
q1	17572	7338	7236	7236
q2	2048	198	179	179
q3	10929	1243	1446	1243
q4	10224	1017	1053	1017
q5	7722	3200	3183	3183
q6	246	160	156	156
q7	1057	655	631	631
q8	9449	2036	2045	2036
q9	6793	6328	6311	6311
q10	7047	2501	2544	2501
q11	428	254	251	251
q12	421	226	227	226
q13	17756	3036	3028	3028
q14	286	249	256	249
q15	591	544	528	528
q16	674	614	615	614
q17	1013	958	956	956
q18	7600	6871	6773	6773
q19	1384	1248	1246	1246
q20	616	330	335	330
q21	3916	3536	3576	3536
q22	1072	1026	992	992
Total cold run time: 108844 ms
Total hot run time: 43222 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7200	7181	7403	7181
q2	350	236	236	236
q3	3075	3115	3089	3089
q4	2116	2116	2040	2040
q5	5769	5604	5751	5604
q6	240	156	153	153
q7	2171	1784	1786	1784
q8	3397	3436	3445	3436
q9	8939	8918	8849	8849
q10	3441	3580	3568	3568
q11	592	485	494	485
q12	809	629	629	629
q13	9400	3208	3193	3193
q14	302	290	286	286
q15	562	548	550	548
q16	682	649	647	647
q17	1834	1776	1794	1776
q18	8489	8030	7940	7940
q19	1797	1772	1741	1741
q20	2169	1873	1878	1873
q21	5924	5652	5627	5627
q22	1118	1064	1068	1064
Total cold run time: 70376 ms
Total hot run time: 61749 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 074c450e5ee3a7ccf6d100683ee3cfb2b9a1bdef, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.03	0.03
query3	0.22	0.06	0.06
query4	1.65	0.09	0.10
query5	0.52	0.53	0.50
query6	1.16	0.72	0.73
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.56	0.52	0.49
query10	0.58	0.58	0.56
query11	0.13	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	3.03	3.05	2.98
query15	0.92	0.85	0.88
query16	0.37	0.38	0.39
query17	1.05	1.04	0.98
query18	0.18	0.17	0.18
query19	1.92	1.89	1.94
query20	0.01	0.01	0.01
query21	15.38	0.65	0.65
query22	3.09	4.08	1.69
query23	17.63	1.08	0.98
query24	2.47	0.38	0.44
query25	0.35	0.08	0.05
query26	0.16	0.14	0.13
query27	0.03	0.03	0.03
query28	11.80	1.01	0.98
query29	12.56	3.35	3.30
query30	0.25	0.06	0.06
query31	2.87	0.37	0.37
query32	3.29	0.46	0.46
query33	2.99	3.03	3.02
query34	15.44	4.27	4.26
query35	4.33	4.27	4.33
query36	0.71	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.02	0.02	0.03
query40	0.15	0.12	0.13
query41	0.07	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.01 s
Total hot run time: 32 s

@yiguolei
Copy link
Contributor

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.31% (9570/25651)
Line Coverage: 28.68% (79059/275616)
Region Coverage: 28.16% (40939/145357)
Branch Coverage: 24.79% (20869/84168)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6bf216f8d957f62fbfec919689f983fe9b259cfa_6bf216f8d957f62fbfec919689f983fe9b259cfa/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41793 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6bf216f8d957f62fbfec919689f983fe9b259cfa, data reload: false

------ Round 1 ----------------------------------
q1	17575	7391	7339	7339
q2	2033	178	173	173
q3	10586	1094	1187	1094
q4	10587	758	759	758
q5	7742	3104	3082	3082
q6	238	153	156	153
q7	1014	618	624	618
q8	9426	2056	2037	2037
q9	6849	6396	6465	6396
q10	6979	2305	2309	2305
q11	431	251	253	251
q12	404	227	224	224
q13	17772	2985	3014	2985
q14	244	210	227	210
q15	589	524	513	513
q16	663	645	616	616
q17	951	798	857	798
q18	7324	6732	6684	6684
q19	1407	1023	1106	1023
q20	574	289	277	277
q21	4002	3327	3232	3232
q22	1091	1025	1028	1025
Total cold run time: 108481 ms
Total hot run time: 41793 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7312	7331	7324	7324
q2	324	237	233	233
q3	3145	3068	2987	2987
q4	2093	1853	1767	1767
q5	5699	5693	5789	5693
q6	236	148	149	148
q7	2220	1828	1852	1828
q8	3354	3527	3466	3466
q9	8909	8936	8943	8936
q10	3501	3530	3522	3522
q11	598	487	484	484
q12	851	635	601	601
q13	8549	3375	3337	3337
q14	329	281	297	281
q15	568	540	556	540
q16	728	689	687	687
q17	1849	1663	1615	1615
q18	8451	8499	7754	7754
q19	2749	1595	1536	1536
q20	2150	1891	1938	1891
q21	5734	5427	5232	5232
q22	1179	1091	1029	1029
Total cold run time: 70528 ms
Total hot run time: 60891 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199265 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bf216f8d957f62fbfec919689f983fe9b259cfa, data reload: false

query1	1262	866	896	866
query2	6506	2109	2016	2016
query3	10764	3836	3950	3836
query4	65247	29257	23465	23465
query5	5069	478	468	468
query6	420	186	182	182
query7	5483	316	294	294
query8	340	230	226	226
query9	8392	2639	2641	2639
query10	466	281	283	281
query11	17509	15140	15759	15140
query12	161	112	102	102
query13	1509	418	432	418
query14	10845	7818	7728	7728
query15	196	194	172	172
query16	6732	481	487	481
query17	1141	602	592	592
query18	1041	309	308	308
query19	198	144	151	144
query20	134	119	113	113
query21	203	102	107	102
query22	4639	4446	4739	4446
query23	34687	33817	33913	33817
query24	6009	2901	2881	2881
query25	494	382	390	382
query26	610	155	161	155
query27	1587	283	279	279
query28	4043	2440	2408	2408
query29	634	407	415	407
query30	237	151	154	151
query31	948	772	823	772
query32	72	53	54	53
query33	439	287	288	287
query34	925	504	481	481
query35	848	711	725	711
query36	1034	949	923	923
query37	147	87	86	86
query38	4083	3935	3926	3926
query39	1450	1420	1363	1363
query40	203	94	94	94
query41	49	45	43	43
query42	119	93	95	93
query43	532	484	486	484
query44	1123	804	802	802
query45	190	159	164	159
query46	1119	782	761	761
query47	1952	1810	1823	1810
query48	458	374	378	374
query49	679	402	392	392
query50	840	426	406	406
query51	6989	6831	7126	6831
query52	100	86	87	86
query53	249	179	175	175
query54	564	484	458	458
query55	78	76	75	75
query56	281	250	256	250
query57	1196	1076	1088	1076
query58	219	228	245	228
query59	3136	3009	2922	2922
query60	276	261	270	261
query61	105	98	103	98
query62	754	672	653	653
query63	217	188	187	187
query64	1308	650	651	650
query65	3276	3188	3196	3188
query66	603	308	303	303
query67	16067	15533	15729	15533
query68	1251	852	837	837
query69	443	346	352	346
query70	1198	1172	1189	1172
query71	344	342	336	336
query72	4735	3555	3614	3555
query73	586	574	577	574
query74	9060	8924	8954	8924
query75	3050	2940	2959	2940
query76	943	862	852	852
query77	464	353	361	353
query78	9371	9160	9401	9160
query79	910	879	849	849
query80	572	569	568	568
query81	452	255	257	255
query82	228	229	225	225
query83	161	169	154	154
query84	207	104	100	100
query85	661	373	367	367
query86	305	314	315	314
query87	4401	4254	4414	4254
query88	4182	4025	4022	4022
query89	366	354	353	353
query90	1292	302	302	302
query91	161	164	164	164
query92	72	73	71	71
query93	889	870	870	870
query94	520	369	367	367
query95	412	433	449	433
query96	477	483	479	479
query97	3143	3083	3194	3083
query98	231	222	223	222
query99	1349	1307	1271	1271
Total cold run time: 301622 ms
Total hot run time: 199265 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6bf216f8d957f62fbfec919689f983fe9b259cfa, data reload: false

query1	0.05	0.05	0.04
query2	0.05	0.03	0.03
query3	0.22	0.06	0.06
query4	1.64	0.10	0.10
query5	0.52	0.52	0.51
query6	1.13	0.74	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.49	0.50
query10	0.54	0.57	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.61	0.59
query14	2.99	2.93	3.09
query15	0.89	0.83	0.83
query16	0.39	0.39	0.38
query17	1.00	0.98	1.08
query18	0.19	0.20	0.20
query19	1.99	1.90	1.98
query20	0.01	0.01	0.00
query21	15.35	0.60	0.60
query22	2.67	2.34	1.99
query23	17.32	0.85	0.83
query24	3.02	1.28	1.64
query25	0.21	0.22	0.14
query26	0.59	0.14	0.13
query27	0.04	0.04	0.04
query28	10.03	1.11	1.08
query29	12.56	3.23	3.22
query30	0.24	0.06	0.06
query31	2.87	0.38	0.37
query32	3.28	0.48	0.47
query33	3.00	3.02	3.04
query34	16.92	4.38	4.38
query35	4.42	4.40	4.41
query36	0.64	0.46	0.48
query37	0.08	0.06	0.06
query38	0.04	0.02	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.73 s
Total hot run time: 33.32 s

HappenLee
HappenLee approved these changes Sep 16, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HappenLee HappenLee merged commit b58570a into apache:master Sep 16, 2024
22 of 26 checks passed
@yiguolei yiguolei mentioned this pull request Sep 18, 2024
Merged
yiguolei added a commit that referenced this pull request Sep 18, 2024
@yiguolei @Doris-Extras
[bugfix](core) fmt lost one parameter (#40914)
241cb7a 
## Proposed changes

bug is introduced by #40542

<!--Describe your changes.-->

---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
@yiguolei yiguolei mentioned this pull request Sep 19, 2024
Merged
yiguolei added a commit that referenced this pull request Sep 19, 2024
@yiguolei @Doris-Extras
[bugfix](cache) the cache capacity is wrong when memusage > soft meml…
087f23a 
…imit (#40961)

## Proposed changes

bug is introduced by #40542

Co-authored-by: yiguolei <yiguolei@gmail.com>
yiguolei added a commit that referenced this pull request Sep 19, 2024
@yiguolei @Doris-Extras
[bugfix](cache) the cache capacity is wrong when memusage > soft meml…
330e0d3 
…imit (#40961)

## Proposed changes

bug is introduced by #40542

Co-authored-by: yiguolei <yiguolei@gmail.com>
yiguolei added a commit that referenced this pull request Sep 23, 2024
@yiguolei @Doris-Extras
[bugfix](core) fmt lost one parameter (#40914)
0a35a87 
## Proposed changes

bug is introduced by #40542

<!--Describe your changes.-->

---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
feifeifeimoon pushed a commit to feifeifeimoon/doris that referenced this pull request Sep 24, 2024
@yiguolei @Doris-Extras @feifeifeimoon
[bugfix](cache) the cache capacity is wrong when memusage > soft meml…
afdcdad 
…imit (apache#40961)

## Proposed changes

bug is introduced by apache#40542

Co-authored-by: yiguolei <yiguolei@gmail.com>
@feifeifeimoon feifeifeimoon mentioned this pull request Sep 24, 2024
Closed
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@yiguolei @Doris-Extras @dataroaring
[bugfix](cache) the cache capacity is wrong when memusage > soft meml…
cc30015 
…imit (#40961)

## Proposed changes

bug is introduced by #40542

Co-authored-by: yiguolei <yiguolei@gmail.com>
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Oct 16, 2024
@xinyiZzz
[fix](memory) Refactor MemCounter (apache#40542)
986350b 
1. Add a new class MemCounter.
2. MemTracker and MemTrackerLimiter no longer have inheritance and
dependencies.
3. MemTrackerLimiter is used to count process memory, the BE web page
`/mem_tracker` also only displays MemTrackerLimiter.
4. MemTracker is used to count the memory of operators and some data
structures. It is not counted in the process memory and is used for
logic control and profile.

In addition, it seems that the crash is caused by memory abnormality,
not sure whether this PR can fix the problem, but this will help locate
the problem and more elegant.

```
==6641==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000980e70 at pc 0x559be21880be bp 0x7fd7cfc75070 sp 0x7fd7cfc75068
READ of size 8 at 0x603000980e70 thread T1453 (memory_maintena)
    #0 0x559be21880bd in std::__atomic_base<long>::load(std::memory_order) const /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
    #1 0x559be21880bd in doris::MemTracker::MemCounter::current_value() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:139:63
    apache#2 0x559be21880bd in doris::MemTracker::consumption() const /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker.h:165:56
    apache#3 0x559be3985133 in doris::MemTrackerLimiter::refresh_global_counter() /home/zcp/repo_center/doris_master/doris/be/src/runtime/memory/mem_tracker_limiter.cpp:245:59
    apache#4 0x559bdfcd52d5 in doris::Daemon::memory_maintenance_thread() /home/zcp/repo_center/doris_master/doris/be/src/common/daemon.cpp:239:13
    apache#5 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
    apache#6 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8
    apache#7 0x7fdf5647584f  misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

0x603000980e74 is located 0 bytes after 20-byte region [0x603000980e60,0x603000980e74)
04:19:48   freed by thread T1491 (Scan_normal [wo) here:
04:19:48       #0 0x559bdfb0dd9d in operator delete(void*) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x33546d9d) (BuildId: f170e92ad3c55512)
04:19:48       #1 0x559bdfb1d42b in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:139:2
04:19:48       apache#2 0x559bdfb1d42b in std::allocator<char>::deallocate(char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:187:27
04:19:48       apache#3 0x559bdfb1d42b in std::allocator_traits<std::allocator<char>>::deallocate(std::allocator<char>&, char*, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:492:13
04:19:48       apache#4 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_destroy(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:237:9
04:19:48       apache#5 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_dispose() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:232:4
04:19:48       apache#6 0x559bdfb1d42b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::~basic_string() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:658:9
04:19:48       apache#7 0x559bdfb22501 in void std::destroy_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
04:19:48       apache#8 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
04:19:48       apache#9 0x559bdfb22501 in void std::_Destroy_aux<false>::__destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
04:19:48       apache#10 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
04:19:48       apache#11 0x559bdfb22501 in void std::_Destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
04:19:48       apache#12 0x559bdfb22501 in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::~vector() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
04:19:48       apache#13 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9867:1
04:19:48       apache#14 0x559be5515262 in doris::TCondition::~TCondition() /home/zcp/repo_center/doris_master/doris/gensrc/build/gen_cpp/PaloInternalService_types.cpp:9866:36
04:19:48       apache#15 0x559be0047ee6 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:372:5
04:19:48       apache#16 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13
04:19:48       apache#17 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28
04:19:48       apache#18 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18
04:19:48       apache#19 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18
04:19:48       apache#20 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5
04:19:48       apache#21 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32
04:19:48       apache#22 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
04:19:48       apache#23 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
04:19:48       apache#24 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
04:19:48       apache#25 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
04:19:48       apache#26 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
04:19:48       apache#27 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
04:19:48       apache#28 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24
04:19:48       apache#29 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
04:19:48       apache#30 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8
04:19:48 
04:19:48   previously allocated by thread T1491 (Scan_normal [wo) here:
04:19:48       #0 0x559bdfb0d53d in operator new(unsigned long) (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x3354653d) (BuildId: f170e92ad3c55512)
04:19:48       #1 0x559bdfb33f74 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:219:14
04:19:48       apache#2 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct_aux<char*>(char*, char*, std::__false_type) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:247:11
04:19:48       apache#3 0x559bdfb32705 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char*>(char*, char*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:266:4
04:19:48       apache#4 0x559bdfb32705 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:451:9
04:19:48       apache#5 0x559bdfc326e6 in decltype(::new((void*)(0)) std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>(std::declval<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>())) std::construct_at<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39
04:19:48       apache#6 0x559bdfc326e6 in void std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:514:4
04:19:48       apache#7 0x559bdfc326e6 in void std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::_M_realloc_insert<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&>(__gnu_cxx::__normal_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:449:4
04:19:48       apache#8 0x559be0032206 in doris::DeleteHandler::parse_condition(doris::DeleteSubPredicatePB const&, doris::TCondition*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:300:33
04:19:48       apache#9 0x559be0047aa3 in doris::Status doris::DeleteHandler::_parse_column_pred<doris::DeleteSubPredicatePB>(std::shared_ptr<doris::TabletSchema>, std::shared_ptr<doris::TabletSchema>, google::protobuf::RepeatedPtrField<doris::DeleteSubPredicatePB> const&, doris::DeleteConditions*) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:355:9
04:19:48       apache#10 0x559be003ba59 in doris::DeleteHandler::init(std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetMeta>, std::allocator<std::shared_ptr<doris::RowsetMeta>>> const&, long, bool) /home/zcp/repo_center/doris_master/doris/be/src/olap/delete_handler.cpp:404:13
04:19:48       apache#11 0x559be30ed6e3 in doris::TabletReader::_init_delete_condition(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:636:28
04:19:48       apache#12 0x559be30e23d1 in doris::TabletReader::_init_params(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:290:18
04:19:48       apache#13 0x559be30e1592 in doris::TabletReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/olap/tablet_reader.cpp:124:18
04:19:48       apache#14 0x559c129840c8 in doris::vectorized::BlockReader::init(doris::TabletReader::ReaderParams const&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:210:5
04:19:48       apache#15 0x559c15a4ea09 in doris::vectorized::NewOlapScanner::open(doris::RuntimeState*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:232:32
04:19:48       apache#16 0x559bfb4acd94 in doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:236:5
04:19:48       apache#17 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:176:21
04:19:48       apache#18 0x559bfb4b10be in doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:175:31
04:19:48       apache#19 0x559bfb4b10be in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
04:19:48       apache#20 0x559bfb4b10be in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&>(doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
04:19:48       apache#21 0x559bfb4b10be in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::'lambda'()>::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
04:19:48       apache#22 0x559be410ed1b in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:543:24
04:19:48       apache#23 0x559be40e6c17 in doris::Thread::supervise_thread(void*) /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
04:19:48       apache#24 0x7fdf563e3ac2 in start_thread nptl/pthread_create.c:442:8
```
xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Oct 16, 2024
@yiguolei @Doris-Extras @xinyiZzz
[bugfix](core) fmt lost one parameter (apache#40914)
6466e61 
## Proposed changes

bug is introduced by apache#40542

<!--Describe your changes.-->

---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
@xinyiZzz xinyiZzz mentioned this pull request Oct 16, 2024
Merged
yiguolei added a commit that referenced this pull request Oct 18, 2024
@xinyiZzz @yiguolei @Doris-Extras
[pick](branch-3.0) pick #40542 #40914 (#41973)
603823a 
pick #40542 #40914

---------

Co-authored-by: yiguolei <676222867@qq.com>
Co-authored-by: yiguolei <yiguolei@gmail.com>
zzzxl1993 pushed a commit to zzzxl1993/doris that referenced this pull request Oct 24, 2024
@yiguolei @Doris-Extras @zzzxl1993
[bugfix](cache) the cache capacity is wrong when memusage > soft meml…
a97f6c8 
…imit (apache#40961)

bug is introduced by apache#40542

Co-authored-by: yiguolei <yiguolei@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HappenLee HappenLee HappenLee approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@xinyiZzz @doris-robot @yiguolei @HappenLee