Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed race condition in BackgroundProcessingPool #3680

Merged
merged 1 commit into from
Nov 27, 2018
Merged

Fixed race condition in BackgroundProcessingPool #3680

merged 1 commit into from
Nov 27, 2018

Conversation

alexey-milovidov
Copy link
Member

No description provided.

@alexey-milovidov
Copy link
Member Author 

==844973==ERROR: AddressSanitizer: heap-use-after-free on address 0x6060012dbca0 at pc 0x7f28eefb2d5d bp 0x7f2847a576b0 sp 0x7f2847a576a8
READ of size 8 at 0x6060012dbca0 thread T26528 (TCPHandler)
    #0 0x7f28eefb2d5c in Poco::Timestamp::Timestamp(Poco::Timestamp const&) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Timestamp.cpp:164:14
    #1 0x7f28fce9fc2c in DB::BackgroundProcessingPoolTaskInfo::wake() /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Storages/MergeTree/BackgroundProcessingPool.cpp:41:37
    #2 0x7f28fd0156c2 in DB::MergeTreeBlockOutputStream::write(DB::Block const&) /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Storages/MergeTree/MergeTreeBlockOutputStream.cpp:30:41
    #3 0x7f28f2da937e in DB::PushingToViewsBlockOutputStream::write(DB::Block const&) /home/proller/ClickHouse3/build_clang_asan/../dbms/src/DataStreams/PushingToViewsBlockOutputStream.cpp:76:17
    #4 0x7f28f2e34bac in DB::SquashingBlockOutputStream::finalize() /home/proller/ClickHouse3/build_clang_asan/../dbms/src/DataStreams/SquashingBlockOutputStream.cpp:30:17
    #5 0x7f28f2e34f88 in DB::SquashingBlockOutputStream::writeSuffix() /home/proller/ClickHouse3/build_clang_asan/../dbms/src/DataStreams/SquashingBlockOutputStream.cpp:50:5
    #6 0x7f28f2ad3d71 in DB::AddingDefaultBlockOutputStream::writeSuffix() /home/proller/ClickHouse3/build_clang_asan/../dbms/src/DataStreams/AddingDefaultBlockOutputStream.cpp:92:13
    #7 0x7f28f2b7a631 in DB::CountingBlockOutputStream::writeSuffix() /home/proller/ClickHouse3/build_clang_asan/../dbms/src/DataStreams/CountingBlockOutputStream.h:37:67
    #8 0x80a351d in DB::TCPHandler::processInsertQuery(DB::Settings const&) /home/proller/ClickHouse3/build_clang_asan/../dbms/programs/server/TCPHandler.cpp:367:19
    #9 0x80950dc in DB::TCPHandler::runImpl() /home/proller/ClickHouse3/build_clang_asan/../dbms/programs/server/TCPHandler.cpp:198:17
    #10 0x80b6364 in DB::TCPHandler::run() /home/proller/ClickHouse3/build_clang_asan/../dbms/programs/server/TCPHandler.cpp:913:9
    #11 0x7f28f1714380 in Poco::Net::TCPServerConnection::start() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3
    #12 0x7f28f1715c13 in Poco::Net::TCPServerDispatcher::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Net/src/TCPServerDispatcher.cpp:114:20
    #13 0x7f28eef921d1 in Poco::PooledThread::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/ThreadPool.cpp:214:14
    #14 0x7f28eef84a89 in Poco::(anonymous namespace)::RunnableHolder::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread.cpp:43:11
    #15 0x7f28eef81798 in Poco::ThreadImpl::runnableEntry(void*) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread_STD.cpp:139:27
    #16 0x7f28eef8a2d3 in decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<Poco::ThreadImpl*>(fp0))) std::__1::__invoke<void* (*)(void*), Poco::ThreadImpl*>(void* (*&&)(void*), Poco::ThreadImpl*&&) /usr/include/c++/v1/type_traits:4482:1
    #17 0x7f28eef8a2d3 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*>&, std::__1::__tuple_indices<2ul>) /usr/include/c++/v1/thread:342
    #18 0x7f28eef8a2d3 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*> >(void*) /usr/include/c++/v1/thread:352
    #19 0x7f28ed60b6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #20 0x7f28ec97288e in clone /build/glibc-OTsEL5/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

0x6060012dbca0 is located 32 bytes inside of 56-byte region [0x6060012dbc80,0x6060012dbcb8)
freed by thread T269 (TCPHandler) here:
    #0 0x7d94638 in operator delete(void*) (/opt/home/proller/ClickHouse3/build_clang_asan/dbms/programs/clickhouse+0x7d94638)
    #1 0x7f28fceb5231 in std::__1::__libcpp_deallocate(void*) /usr/include/c++/v1/new:236:3
    #2 0x7f28fceb5231 in std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*> >::deallocate(std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*>*, unsigned long) /usr/include/c++/v1/memory:1796
    #3 0x7f28fceb5231 in std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*> > >::deallocate(std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*> >&, std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*>*, unsigned long) /usr/include/c++/v1/memory:1555
    #4 0x7f28fceb5231 in std::__1::__tree<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, std::__1::__map_value_compare<Poco::Timestamp, std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, std::__1::less<Poco::Timestamp>, true>, std::__1::allocator<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> > > >::erase(std::__1::__tree_const_iterator<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*>*, long>) /usr/include/c++/v1/__tree:2370
    #5 0x7f28fcea5982 in std::__1::multimap<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo>, std::__1::less<Poco::Timestamp>, std::__1::allocator<std::__1::pair<Poco::Timestamp const, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> > > >::erase(std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*>*, long> >) /usr/include/c++/v1/map:1746:56
    #6 0x7f28fcea5982 in DB::BackgroundProcessingPool::removeTask(std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> const&) /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Storages/MergeTree/BackgroundProcessingPool.cpp:95
    #7 0x7f28fc95e065 in DB::StorageMergeTree::shutdown() /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Storages/StorageMergeTree.cpp:100:25
    #8 0x7f28fb8cc148 in DB::InterpreterDropQuery::executeToTable(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, DB::ASTDropQuery::Kind, bool, bool) /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Interpreters/InterpreterDropQuery.cpp:89:40
    #9 0x7f28fb8c8f2a in DB::InterpreterDropQuery::execute() /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Interpreters/InterpreterDropQuery.cpp:42:16
    #10 0x7f28fc2c1b99 in DB::executeQueryImpl(char const*, char const*, DB::Context&, bool, DB::QueryProcessingStage::Enum) /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Interpreters/executeQuery.cpp:206:28
    #11 0x7f28fc2be754 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum) /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Interpreters/executeQuery.cpp:416:38
    #12 0x8094e44 in DB::TCPHandler::runImpl() /home/proller/ClickHouse3/build_clang_asan/../dbms/programs/server/TCPHandler.cpp:188:24
    #13 0x80b6364 in DB::TCPHandler::run() /home/proller/ClickHouse3/build_clang_asan/../dbms/programs/server/TCPHandler.cpp:913:9
    #14 0x7f28f1714380 in Poco::Net::TCPServerConnection::start() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3
    #15 0x7f28f1715c13 in Poco::Net::TCPServerDispatcher::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Net/src/TCPServerDispatcher.cpp:114:20
    #16 0x7f28eef921d1 in Poco::PooledThread::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/ThreadPool.cpp:214:14
    #17 0x7f28eef84a89 in Poco::(anonymous namespace)::RunnableHolder::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread.cpp:43:11
    #18 0x7f28eef81798 in Poco::ThreadImpl::runnableEntry(void*) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread_STD.cpp:139:27
    #19 0x7f28eef8a2d3 in decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<Poco::ThreadImpl*>(fp0))) std::__1::__invoke<void* (*)(void*), Poco::ThreadImpl*>(void* (*&&)(void*), Poco::ThreadImpl*&&) /usr/include/c++/v1/type_traits:4482:1
    #20 0x7f28eef8a2d3 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*>&, std::__1::__tuple_indices<2ul>) /usr/include/c++/v1/thread:342
    #21 0x7f28eef8a2d3 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*> >(void*) /usr/include/c++/v1/thread:352
    #22 0x7f28ed60b6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

previously allocated by thread T35 (BackgrProcPool) here:
    #0 0x7d93840 in operator new(unsigned long) (/opt/home/proller/ClickHouse3/build_clang_asan/dbms/programs/clickhouse+0x7d93840)
    #1 0x7f28fceb624e in std::__1::__allocate(unsigned long) /usr/include/c++/v1/new:228:10
    #2 0x7f28fceb624e in std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*> >::allocate(unsigned long, void const*) /usr/include/c++/v1/memory:1793
    #3 0x7f28fceb624e in std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*> > >::allocate(std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*> >&, unsigned long) /usr/include/c++/v1/memory:1547
    #4 0x7f28fceb624e in std::__1::unique_ptr<std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*>, std::__1::__tree_node_destructor<std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*> > > > std::__1::__tree<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, std::__1::__map_value_compare<Poco::Timestamp, std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, std::__1::less<Poco::Timestamp>, true>, std::__1::allocator<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> > > >::__construct_node<Poco::Timestamp&, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo>&>(Poco::Timestamp&, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo>&) /usr/include/c++/v1/__tree:2191
    #5 0x7f28fceb54c0 in std::__1::__tree_iterator<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*>*, long> std::__1::__tree<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, std::__1::__map_value_compare<Poco::Timestamp, std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, std::__1::less<Poco::Timestamp>, true>, std::__1::allocator<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> > > >::__emplace_multi<Poco::Timestamp&, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo>&>(Poco::Timestamp&, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo>&) /usr/include/c++/v1/__tree:2240:25
    #6 0x7f28fceaba20 in std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, std::__1::__tree_node<std::__1::__value_type<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> >, void*>*, long> > std::__1::multimap<Poco::Timestamp, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo>, std::__1::less<Poco::Timestamp>, std::__1::allocator<std::__1::pair<Poco::Timestamp const, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo> > > >::emplace<Poco::Timestamp&, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo>&>(Poco::Timestamp&, std::__1::shared_ptr<DB::BackgroundProcessingPoolTaskInfo>&) /usr/include/c++/v1/map:1692:24
    #7 0x7f28fceaba20 in DB::BackgroundProcessingPool::threadFunction() /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Storages/MergeTree/BackgroundProcessingPool.cpp:217
    #8 0x7f28fceac92c in DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_0::operator()() const /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Storages/MergeTree/BackgroundProcessingPool.cpp:63:39
    #9 0x7f28fceac55c in decltype(std::__1::forward<DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_0>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_0>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_0&&) /usr/include/c++/v1/type_traits:4482:1
    #10 0x7f28fceac55c in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_0>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_0>&, std::__1::__tuple_indices<>) /usr/include/c++/v1/thread:342
    #11 0x7f28fceac55c in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_0> >(void*) /usr/include/c++/v1/thread:352
    #12 0x7f28ed60b6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

Thread T26528 (TCPHandler) created by T45 here:
    #0 0x7ca2500 in __interceptor_pthread_create (/opt/home/proller/ClickHouse3/build_clang_asan/dbms/programs/clickhouse+0x7ca2500)
    #1 0x7f28eef860da in std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /usr/include/c++/v1/__threading_support:327:10
    #2 0x7f28eef860da in std::__1::thread::thread<void* (&)(void*), Poco::ThreadImpl*, void>(void* (&)(void*), Poco::ThreadImpl*&&) /usr/include/c++/v1/thread:368
    #3 0x7f28eef80722 in Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable> >) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread_STD.cpp:58:53
    #4 0x7f28eef83f6f in Poco::Thread::start(Poco::Runnable&) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread.cpp:116:2
    #5 0x7f28eef90bfb in Poco::PooledThread::start(int) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/ThreadPool.cpp:88:10
    #6 0x7f28eef9d0db in Poco::ThreadPool::getThread() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/ThreadPool.cpp:525:14
    #7 0x7f28eef9d8ca in Poco::ThreadPool::startWithPriority(Poco::Thread::Priority, Poco::Runnable&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/ThreadPool.cpp:429:2
    #8 0x7f28f17177b9 in Poco::Net::TCPServerDispatcher::enqueue(Poco::Net::StreamSocket const&) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Net/src/TCPServerDispatcher.cpp:145:17
    #9 0x7f28f1713270 in Poco::Net::TCPServer::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Net/src/TCPServer.cpp:148:21
    #10 0x7f28eef84a89 in Poco::(anonymous namespace)::RunnableHolder::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread.cpp:43:11
    #11 0x7f28eef81798 in Poco::ThreadImpl::runnableEntry(void*) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread_STD.cpp:139:27
    #12 0x7f28eef8a2d3 in decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<Poco::ThreadImpl*>(fp0))) std::__1::__invoke<void* (*)(void*), Poco::ThreadImpl*>(void* (*&&)(void*), Poco::ThreadImpl*&&) /usr/include/c++/v1/type_traits:4482:1
    #13 0x7f28eef8a2d3 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*>&, std::__1::__tuple_indices<2ul>) /usr/include/c++/v1/thread:342
    #14 0x7f28eef8a2d3 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*> >(void*) /usr/include/c++/v1/thread:352
    #15 0x7f28ed60b6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

Thread T45 created by T0 here:
    #0 0x7ca2500 in __interceptor_pthread_create (/opt/home/proller/ClickHouse3/build_clang_asan/dbms/programs/clickhouse+0x7ca2500)
    #1 0x7f28eef860da in std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /usr/include/c++/v1/__threading_support:327:10
    #2 0x7f28eef860da in std::__1::thread::thread<void* (&)(void*), Poco::ThreadImpl*, void>(void* (&)(void*), Poco::ThreadImpl*&&) /usr/include/c++/v1/thread:368
    #3 0x7f28eef80722 in Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable> >) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread_STD.cpp:58:53
    #4 0x7f28eef83f6f in Poco::Thread::start(Poco::Runnable&) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread.cpp:116:2
    #5 0x7f28f1712d57 in Poco::Net::TCPServer::start() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Net/src/TCPServer.cpp:111:10
    #6 0x7de2eac in DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /home/proller/ClickHouse3/build_clang_asan/../dbms/programs/server/Server.cpp:630:21
    #7 0x7f28f02a312e in Poco::Util::Application::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Util/src/Application.cpp:335:8
    #8 0x7f28f03b03e4 in Poco::Util::ServerApplication::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Util/src/ServerApplication.cpp:94:22
    #9 0x7f28f03b07d0 in Poco::Util::ServerApplication::run(int, char**) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Util/src/ServerApplication.cpp:618:9
    #10 0x7dee591 in mainEntryClickHouseServer(int, char**) /home/proller/ClickHouse3/build_clang_asan/../dbms/programs/server/Server.cpp:728:20
    #11 0x7d98470 in main /home/proller/ClickHouse3/build_clang_asan/../dbms/programs/main.cpp:181:12
    #12 0x7f28ec872b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

Thread T269 (TCPHandler) created by T45 here:
    #0 0x7ca2500 in __interceptor_pthread_create (/opt/home/proller/ClickHouse3/build_clang_asan/dbms/programs/clickhouse+0x7ca2500)
    #1 0x7f28eef860da in std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /usr/include/c++/v1/__threading_support:327:10
    #2 0x7f28eef860da in std::__1::thread::thread<void* (&)(void*), Poco::ThreadImpl*, void>(void* (&)(void*), Poco::ThreadImpl*&&) /usr/include/c++/v1/thread:368
    #3 0x7f28eef80722 in Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable> >) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread_STD.cpp:58:53
    #4 0x7f28eef83f6f in Poco::Thread::start(Poco::Runnable&) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread.cpp:116:2
    #5 0x7f28eef90bfb in Poco::PooledThread::start(int) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/ThreadPool.cpp:88:10
    #6 0x7f28eef9d0db in Poco::ThreadPool::getThread() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/ThreadPool.cpp:525:14
    #7 0x7f28eef9d8ca in Poco::ThreadPool::startWithPriority(Poco::Thread::Priority, Poco::Runnable&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/ThreadPool.cpp:429:2
    #8 0x7f28f17177b9 in Poco::Net::TCPServerDispatcher::enqueue(Poco::Net::StreamSocket const&) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Net/src/TCPServerDispatcher.cpp:145:17
    #9 0x7f28f1713270 in Poco::Net::TCPServer::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Net/src/TCPServer.cpp:148:21
    #10 0x7f28eef84a89 in Poco::(anonymous namespace)::RunnableHolder::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread.cpp:43:11
    #11 0x7f28eef81798 in Poco::ThreadImpl::runnableEntry(void*) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Thread_STD.cpp:139:27
    #12 0x7f28eef8a2d3 in decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<Poco::ThreadImpl*>(fp0))) std::__1::__invoke<void* (*)(void*), Poco::ThreadImpl*>(void* (*&&)(void*), Poco::ThreadImpl*&&) /usr/include/c++/v1/type_traits:4482:1
    #13 0x7f28eef8a2d3 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*>&, std::__1::__tuple_indices<2ul>) /usr/include/c++/v1/thread:342
    #14 0x7f28eef8a2d3 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*> >(void*) /usr/include/c++/v1/thread:352
    #15 0x7f28ed60b6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

Thread T35 (BackgrProcPool) created by T0 here:
    #0 0x7ca2500 in __interceptor_pthread_create (/opt/home/proller/ClickHouse3/build_clang_asan/dbms/programs/clickhouse+0x7ca2500)
    #1 0x7f28fcea3e1b in std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /usr/include/c++/v1/__threading_support:327:10
    #2 0x7f28fcea3e1b in std::__1::thread::thread<DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_0, void>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_0&&) /usr/include/c++/v1/thread:368
    #3 0x7f28fcea3196 in DB::BackgroundProcessingPool::BackgroundProcessingPool(int) /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Storages/MergeTree/BackgroundProcessingPool.cpp:63:18
    #4 0x7f28fb44b619 in std::__1::__compressed_pair_elem<DB::BackgroundProcessingPool, 1, false>::__compressed_pair_elem<DB::SettingInt<unsigned long>&, 0ul>(std::__1::piecewise_construct_t, std::__1::tuple<DB::SettingInt<unsigned long>&>, std::__1::__tuple_indices<0ul>) /usr/include/c++/v1/memory:2103:9
    #5 0x7f28fb44b619 in std::__1::__compressed_pair<std::__1::allocator<DB::BackgroundProcessingPool>, DB::BackgroundProcessingPool>::__compressed_pair<std::__1::allocator<DB::BackgroundProcessingPool>&, DB::SettingInt<unsigned long>&>(std::__1::piecewise_construct_t, std::__1::tuple<std::__1::allocator<DB::BackgroundProcessingPool>&>, std::__1::tuple<DB::SettingInt<unsigned long>&>) /usr/include/c++/v1/memory:2205
    #6 0x7f28fb44b619 in std::__1::__shared_ptr_emplace<DB::BackgroundProcessingPool, std::__1::allocator<DB::BackgroundProcessingPool> >::__shared_ptr_emplace<DB::SettingInt<unsigned long>&>(std::__1::allocator<DB::BackgroundProcessingPool>, DB::SettingInt<unsigned long>&) /usr/include/c++/v1/memory:3618
    #7 0x7f28fb44b619 in std::__1::shared_ptr<DB::BackgroundProcessingPool> std::__1::shared_ptr<DB::BackgroundProcessingPool>::make_shared<DB::SettingInt<unsigned long>&>(DB::SettingInt<unsigned long>&) /usr/include/c++/v1/memory:4277
    #8 0x7f28fb38e9cf in std::__1::enable_if<!(is_array<DB::BackgroundProcessingPool>::value), std::__1::shared_ptr<DB::BackgroundProcessingPool> >::type std::__1::make_shared<DB::BackgroundProcessingPool, DB::SettingInt<unsigned long>&>(DB::SettingInt<unsigned long>&) /usr/include/c++/v1/memory:4656:12
    #9 0x7f28fb38e9cf in DB::Context::getBackgroundPool() /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Interpreters/Context.cpp:1344
    #10 0x7f28fb59aae5 in DB::DNSCacheUpdater::DNSCacheUpdater(DB::Context&) /home/proller/ClickHouse3/build_clang_asan/../dbms/src/Interpreters/DNSCacheUpdater.cpp:59:40
    #11 0x7dcd21b in std::__1::__unique_if<DB::DNSCacheUpdater>::__unique_single std::__1::make_unique<DB::DNSCacheUpdater, DB::Context&>(DB::Context&) /usr/include/c++/v1/memory:3078:32
    #12 0x7dcd21b in DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /home/proller/ClickHouse3/build_clang_asan/../dbms/programs/server/Server.cpp:410
    #13 0x7f28f02a312e in Poco::Util::Application::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Util/src/Application.cpp:335:8
    #14 0x7f28f03b03e4 in Poco::Util::ServerApplication::run() /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Util/src/ServerApplication.cpp:94:22
    #15 0x7f28f03b07d0 in Poco::Util::ServerApplication::run(int, char**) /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Util/src/ServerApplication.cpp:618:9
    #16 0x7dee591 in mainEntryClickHouseServer(int, char**) /home/proller/ClickHouse3/build_clang_asan/../dbms/programs/server/Server.cpp:728:20
    #17 0x7d98470 in main /home/proller/ClickHouse3/build_clang_asan/../dbms/programs/main.cpp:181:12
    #18 0x7f28ec872b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

SUMMARY: AddressSanitizer: heap-use-after-free /opt/home/proller/ClickHouse3/build_clang_asan/../contrib/poco/Foundation/src/Timestamp.cpp:164:14 in Poco::Timestamp::Timestamp(Poco::Timestamp const&)
Shadow bytes around the buggy address:
  0x0c0c80253740: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c80253750: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x0c0c80253760: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x0c0c80253770: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c80253780: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
=>0x0c0c80253790: fd fd fd fd[fd]fd fd fa fa fa fa fa fa fa fa fa
  0x0c0c802537a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c802537b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c802537c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c802537d0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c802537e0: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==844973==ABORTING

@alexey-milovidov alexey-milovidov merged commit b77043c into master Nov 27, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ztlpn ztlpn ztlpn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@alexey-milovidov @ztlpn