clickhouse does not terminates if bind to tcp_port fails

[azat]

If bind to the tcp_port fails, clickhouse does not exists:

• /var/log/clickhouse-server/clickhouse-server.err.log

2019.02.28 13:50:09.188218 [ 1 ] {} <Error> Application: Net Exception: Address already in use: 0.0.0.0:9010
2019.03.01 07:36:21.980763 [ 19 ] {} <Error> void Coordination::ZooKeeper::receiveThread(): Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expe

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x6559076]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3089cd2]
2. /usr/bin/clickhouse-server() [0x61cc7c0]
3. /usr/bin/clickhouse-server(Coordination::ZooKeeper::receiveEvent()+0x34) [0x61cd5a4]
4. /usr/bin/clickhouse-server(Coordination::ZooKeeper::receiveThread()+0x2b6) [0x61cec36]
5. /usr/bin/clickhouse-server() [0x61cee8a]
6. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)+0x19f) [0x655eb6f]
7. /usr/bin/clickhouse-server() [0x70d83df]
8. /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7facf802efa3]
9. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7facf7dd280f]

2019.03.01 07:36:21.980880 [ 19 ] {} <Error> void Coordination::ZooKeeper::finalize(bool, bool): Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket
2019.03.01 07:36:22.021182 [ 21 ] {24159d40-d5c7-4ad3-9247-85aa4b23d323} <Error> void DB::DDLWorker::run(): Code: 439, e.displayText() = DB::Exception: Cannot schedule a task, Stack

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x6559076]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3089cd2]
2. /usr/bin/clickhouse-server() [0x655e336]
3. /usr/bin/clickhouse-server(void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, int, std::optional<unsigned long>)+0x589) [0x65619c9]
4. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, int, unsigned long)+0x4d) [0x6561c6d]
5. /usr/bin/clickhouse-server(Coordination::ZooKeeper::ZooKeeper(std::vector<Poco::Net::SocketAddress, std::allocator<Poco::Net::SocketAddress> > const&, std::__cxx11::basic_string<c
6. /usr/bin/clickhouse-server(zkutil::ZooKeeper::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::ch
7. /usr/bin/clickhouse-server(zkutil::ZooKeeper::startNewSession() const+0xd5) [0x61c6475]
8. /usr/bin/clickhouse-server(DB::Context::getZooKeeper() const+0xc0) [0x5af58a0]
9. /usr/bin/clickhouse-server() [0x2f3cc60]
10. /usr/bin/clickhouse-server(ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::{lambda()#1}::ope
11. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)+0x19f) [0x655eb6f]
12. /usr/bin/clickhouse-server() [0x70d83df]
13. /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7facf802efa3]
14. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7facf7dd280f]

2019.03.01 07:36:27.025581 [ 21 ] {24159d40-d5c7-4ad3-9247-85aa4b23d323} <Error> void DB::DDLWorker::run(): Code: 439, e.displayText() = DB::Exception: Cannot schedule a task, Stack

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x6559076]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3089cd2]
2. /usr/bin/clickhouse-server() [0x655e336]
3. /usr/bin/clickhouse-server(void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, int, std::optional<unsigned long>)+0x589) [0x65619c9]
4. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, int, unsigned long)+0x4d) [0x6561c6d]
5. /usr/bin/clickhouse-server(Coordination::ZooKeeper::ZooKeeper(std::vector<Poco::Net::SocketAddress, std::allocator<Poco::Net::SocketAddress> > const&, std::__cxx11::basic_string<c
6. /usr/bin/clickhouse-server(zkutil::ZooKeeper::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::ch
7. /usr/bin/clickhouse-server(zkutil::ZooKeeper::startNewSession() const+0xd5) [0x61c6475]
8. /usr/bin/clickhouse-server(DB::Context::getZooKeeper() const+0xc0) [0x5af58a0]
9. /usr/bin/clickhouse-server() [0x2f3cc60]
10. /usr/bin/clickhouse-server(ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::{lambda()#1}::ope
11. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)+0x19f) [0x655eb6f]
12. /usr/bin/clickhouse-server() [0x70d83df]
13. /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7facf802efa3]

...

2019.03.01 07:44:31.920536 [ 21 ] {24159d40-d5c7-4ad3-9247-85aa4b23d323} <Error> void DB::DDLWorker::run(): Code: 439, e.displayText() = DB::Exception: Cannot schedule a task, Stack

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x6559076]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3089cd2]
2. /usr/bin/clickhouse-server() [0x655e336]
3. /usr/bin/clickhouse-server(void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, int, std::optional<unsigned long>)+0x589) [0x65619c9]
4. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::scheduleOrThrow(std::function<void ()>, int, unsigned long)+0x4d) [0x6561c6d]
5. /usr/bin/clickhouse-server(Coordination::ZooKeeper::ZooKeeper(std::vector<Poco::Net::SocketAddress, std::allocator<Poco::Net::SocketAddress> > const&, std::__cxx11::basic_string<c
6. /usr/bin/clickhouse-server(zkutil::ZooKeeper::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::ch
7. /usr/bin/clickhouse-server(zkutil::ZooKeeper::startNewSession() const+0xd5) [0x61c6475]
8. /usr/bin/clickhouse-server(DB::Context::getZooKeeper() const+0xc0) [0x5af58a0]
9. /usr/bin/clickhouse-server() [0x2f3cc60]
10. /usr/bin/clickhouse-server(ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::{lambda()#1}::ope
11. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)+0x19f) [0x655eb6f]
12. /usr/bin/clickhouse-server() [0x70d83df]
13. /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7facf802efa3]
14. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7facf7dd280f]

# gdb -q -p 28890
(gdb) bt
#0  0x00007facf8030495 in __GI___pthread_timedjoin_ex (threadid=140377195542272, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:89
#1  0x00000000070d8603 in std::thread::join() ()
#2  0x000000000655f4b9 in ThreadPoolImpl<std::thread>::finalize() ()
#3  0x000000000655f54d in ThreadPoolImpl<std::thread>::~ThreadPoolImpl() ()
#4  0x00007facf7d12e9c in __run_exit_handlers (status=70, listp=0x7facf7e94718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
    at exit.c:108
#5  0x00007facf7d12fca in __GI_exit (status=<optimized out>) at exit.c:139
#6  0x00007facf7cfd0a2 in __libc_start_main (main=0x306c0c0 <main>, argc=2, argv=0x7ffded6a92e8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7ffded6a92d8) at ../csu/libc-start.c:342
#7  0x000000000308696a in _start ()
(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7facf7cd4b80 (LWP 28890) "clickhouse-serv" 0x00007facf8030495 in __GI___pthread_timedjoin_ex (threadid=140377195542272, thread_return=0x0, abstime=0x0,
    block=<optimized out>) at pthread_join_common.c:89
  2    Thread 0x7facf717f700 (LWP 28902) "clickhouse-serv" futex_wait_cancelable (private=0, expected=0, futex_word=0x7facf7971640) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7facf697e700 (LWP 28903) "clickhouse-serv" futex_wait_cancelable (private=0, expected=0, futex_word=0x7facf79718c0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  4    Thread 0x7fac6f1ff700 (LWP 29293) "ZooKeeperRecv" 0x00007facf7dd2b2f in epoll_wait (epfd=3, events=0x7fac6f1fa630, maxevents=1, timeout=10000)
    at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  5    Thread 0x7fac1dbff700 (LWP 29294) "ZooKeeperSend" futex_abstimed_wait_cancelable (private=0, abstime=0x7fac1dbfa610, expected=0, futex_word=0x7fac1b83a2cc)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  6    Thread 0x7fac1cdff700 (LWP 29295) "DDLWorker" futex_wait_cancelable (private=0, expected=0, futex_word=0x7facf7809578) at ../sysdeps/unix/sysv/linux/futex-internal.h:88

P.S. this server is running clickhouse that compiled from https://github.com/azat-archive/ClickHouse/tree/thread-pool-join but AFAICS

• it should not create such issue (although fix is not strictly correct, refs detach thread before erasing it from the FreeThreadPool #4485)
• Fixed ThreadPool when there are too many inactive threads #4485 #4505 does not fix the issue, but maybe I missing something

[alexey-milovidov]

Fixed in recent releases.