Segmentation violation in Boost ASIO

[waTeim]

Please provide the following information

libtorrent version (or branch):

libtorrent 1.1.4
boost 1.64

platform/architecture:

lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 9.4 (stretch)
Release:	9.4
Codename:	stretch

compiler and compiler version:

gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.9.2-10' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --with-arch-32=i586 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.2 (Debian 4.9.2-10) 

please describe what symptom you see, what you would expect to see instead and
how to reproduce it.

This is possibly out of the scope of libtorrent, but specific guards might be included to avoid it, or some warning how it may occur.

This seems likely to the known boost asio problem 7611

Here's the stack trace.

(gdb) where
#0  0x000000000078f284 in boost::asio::detail::epoll_reactor::start_op (allow_speculative=<optimized out>, is_continuation=false, op=0x7f6358117760, descriptor_data=@0x7f635812b7d0: 0x0, descriptor=<optimized out>, op_type=1, this=0x19de170)
at /usr/local/include/boost/asio/detail/impl/epoll_reactor.ipp:230
#1  boost::asio::detail::reactive_socket_service_base::start_op (this=<optimized out>, impl=..., op_type=1, op=0x7f6358117760, is_continuation=<optimized out>, is_non_blocking=<optimized out>, noop=false)
at /usr/local/include/boost/asio/detail/impl/reactive_socket_service_base.ipp:214
#2  0x000000000079fe76 in boost::asio::detail::reactive_socket_service_base::async_send<std::vector<boost::asio::const_buffer>, libtorrent::aux::allocating_handler<boost::_bi::bind_t<void, boost::_mfi::mf2<void, libtorrent::peer_connection, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<libtorrent::peer_connection> >, boost::arg<1>, boost::arg<2> > >, 336ul> > (handler=..., flags=0, buffers=std::vector of length 1, capacity 8 = {...}, impl=..., this=0x19de298)
at /usr/local/include/boost/asio/detail/reactive_socket_service_base.hpp:216
#3  boost::asio::stream_socket_service<boost::asio::ip::tcp>::async_send<std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, libtorrent::aux::allocating_handler<boost::_bi::bind_t<void, boost::_mfi::mf2<void, libtorrent::peer_connection, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<libtorrent::peer_connection> >, boost::arg<1>, boost::arg<2> > >, 336ul> > (this=0x19de270, impl=..., buffers=std::vector of length 1, capacity 8 = {...},
flags=0, handler=...) at /usr/local/include/boost/asio/stream_socket_service.hpp:334
#4  0x00000000007b99d5 in boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >::async_write_some<std::vector<boost::asio::const_buffer>, libtorrent::aux::allocating_handler<boost::_bi::bind_t<void, boost::_mfi::mf2<void, libtorrent::peer_connection, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<libtorrent::peer_connection> >, boost::arg<1>, boost::arg<2> > >, 336ul> > (handler=..., buffers=...,
this=<optimized out>) at /usr/local/include/boost/asio/basic_stream_socket.hpp:732
#5  libtorrent::proxy_base::async_write_some<std::vector<boost::asio::const_buffer>, libtorrent::aux::allocating_handler<boost::_bi::bind_t<void, boost::_mfi::mf2<void, libtorrent::peer_connection, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<libtorrent::peer_connection> >, boost::arg<1>, boost::arg<2> > >, 336ul> > (handler=..., buffers=..., this=<optimized out>) at ../include/libtorrent/proxy_base.hpp:122
#6  libtorrent::socket_type::async_write_some<std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, libtorrent::aux::allocating_handler<boost::_bi::bind_t<void, boost::_mfi::mf2<void, libtorrent::peer_connection, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<libtorrent::peer_connection> >, boost::arg<1>, boost::arg<2> > >, 336ul> > (this=0x7f63580084c8, buffers=<error reading variable: Cannot access memory at address 0x88>, handler=...)
at ../include/libtorrent/socket_type.hpp:240
#7  0x0000000000787e82 in libtorrent::network_thread_pool::process_job (this=<optimized out>, j=..., post=<optimized out>) at session_impl.cpp:199
#8  0x000000000078d6cf in libtorrent::thread_pool<libtorrent::socket_job>::thread_fun (this=0x7f63580029d0, thread_id=0) at ../include/libtorrent/thread_pool.hpp:133
#9  0x00000000007295ff in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x7f6358002d10) at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:64
#10 0x00007f636b594494 in start_thread (arg=0x7f635da72700) at pthread_create.c:333
#11 0x00007f636aa39acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
(gdb) down
Bottom (innermost) frame selected; you cannot go down.
(gdb) print descriptor_data
$1 = (boost::asio::detail::epoll_reactor::per_descriptor_data &) @0x7f635812b7d0: 0x0

[waTeim]

Other (asio) threads

gdb) thread 17
[Switching to thread 17 (Thread 0x7f6360a78700 (LWP 10238))]
#0  0x00007f636aa3a0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
84      ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) where
#0  0x00007f636aa3a0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x000000000071da37 in boost::asio::detail::epoll_reactor::run (this=0x19de170, block=<optimized out>, ops=...) at /usr/local/include/boost/asio/detail/impl/epoll_reactor.ipp:416
#2  0x000000000074c4e8 in boost::asio::detail::task_io_service::do_run_one (ec=..., this_thread=..., lock=..., this=0x19d9d10) at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:356
#3  boost::asio::detail::task_io_service::run (ec=..., this=0x19d9d10) at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:149
#4  boost::asio::io_service::run (this=<optimized out>) at /usr/local/include/boost/asio/impl/io_service.ipp:59
#5  0x00000000007295ff in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x19e0620) at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:64
#6  0x00007f636b594494 in start_thread (arg=0x7f6360a78700) at pthread_create.c:333
#7  0x00007f636aa39acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97


(gdb) thread 18
[Switching to thread 18 (Thread 0x7f6360277700 (LWP 10239))]
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
225     ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S: No such file or directory.
(gdb) where
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00000000007de934 in libtorrent::condition_variable::wait_for (this=this@entry=0x19dad00, l=..., rel_time=rel_time@entry=...) at thread.cpp:92
#2  0x00000000008bf8d7 in libtorrent::alert_manager::wait_for_alert (this=0x19dacd8, max_wait=...) at alert_manager.cpp:67


(gdb) thread 19
[Switching to thread 19 (Thread 0x7f635fa76700 (LWP 10240))]
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185     ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
(gdb) where
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000071e2c8 in boost::asio::detail::posix_event::wait<boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex> > (lock=..., this=0x19ddfc8) at /usr/local/include/boost/asio/detail/posix_event.hpp:106
#2  boost::asio::detail::task_io_service::do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:380
#3  boost::asio::detail::task_io_service::run (this=0x19ddf70, ec=...) at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:149
#4  0x0000000000791346 in boost::asio::io_service::run (this=0x19ddf00) at /usr/local/include/boost/asio/impl/io_service.ipp:59
#5  boost::asio::detail::resolver_service_base::work_io_service_runner::operator() (this=<optimized out>) at /usr/local/include/boost/asio/detail/impl/resolver_service_base.ipp:32
#6  boost::asio::detail::posix_thread::func<boost::asio::detail::resolver_service_base::work_io_service_runner>::run (this=<optimized out>) at /usr/local/include/boost/asio/detail/posix_thread.hpp:82
#7  0x00000000007295ff in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x7f6358001120) at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:64
#8  0x00007f636b594494 in start_thread (arg=0x7f635fa76700) at pthread_create.c:333
#9  0x00007f636aa39acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Threads 20,21, and 22 have the same stack trace

(gdb) thread 20
[Switching to thread 20 (Thread 0x7f635f275700 (LWP 10241))]
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185     in ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
(gdb) where
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000078d72c in libtorrent::thread_pool<libtorrent::socket_job>::thread_fun (this=0x7f6358001bb0, thread_id=0) at ../include/libtorrent/thread_pool.hpp:120
#2  0x00000000007295ff in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x7f6358001500) at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:64
#3  0x00007f636b594494 in start_thread (arg=0x7f635f275700) at pthread_create.c:333
#4  0x00007f636aa39acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 23 is the one shown above

Threads 24 through 35 have the same stack trace

(gdb) thread 24
[Switching to thread 24 (Thread 0x7f635d271700 (LWP 10245))]
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185     ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such file or directory.
(gdb) where
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000078d72c in libtorrent::thread_pool<libtorrent::socket_job>::thread_fun (this=0x7f6358002ee0, thread_id=0) at ../include/libtorrent/thread_pool.hpp:120
#2  0x00000000007295ff in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x7f63580032b0) at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:64
#3  0x00007f636b594494 in start_thread (arg=0x7f635d271700) at pthread_create.c:333
#4  0x00007f636aa39acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97 

Threads 36 through 39 have the same

(gdb) thread 36
[Switching to thread 36 (Thread 0x7f634abd8700 (LWP 10259))]
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185     in ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
(gdb) where
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000728089 in libtorrent::disk_io_thread::thread_fun (this=0x19dae00, thread_id=0, type=libtorrent::disk_io_thread::generic_thread, w=...) at disk_io_thread.cpp:3264
#2  0x0000000000729e41 in boost::_mfi::mf3<void, libtorrent::disk_io_thread, int, libtorrent::disk_io_thread::thread_type_t, boost::shared_ptr<boost::asio::io_service::work> >::operator() (a3=..., a2=<optimized out>, a1=<optimized out>, p=<optimized out>, this=<optimized out>)
at /usr/local/include/boost/bind/mem_fn_template.hpp:393
#3  boost::_bi::list4<boost::_bi::value<libtorrent::disk_io_thread*>, boost::_bi::value<int>, boost::_bi::value<libtorrent::disk_io_thread::thread_type_t>, boost::_bi::value<boost::shared_ptr<boost::asio::io_service::work> > >::operator()<boost::_mfi::mf3<void, libtorrent::disk_io_thread, int, libtorrent::disk_io_thread::thread_type_t, boost::shared_ptr<boost::asio::io_service::work> >, boost::_bi::list0> (a=<synthetic pointer>..., f=..., this=<optimized out>) at /usr/local/include/boost/bind/bind.hpp:463
#4  boost::_bi::bind_t<void, boost::_mfi::mf3<void, libtorrent::disk_io_thread, int, libtorrent::disk_io_thread::thread_type_t, boost::shared_ptr<boost::asio::io_service::work> >, boost::_bi::list4<boost::_bi::value<libtorrent::disk_io_thread*>, boost::_bi::value<int>, boost::_bi::value<libtorrent::disk_io_thread::thread_type_t>, boost::_bi::value<boost::shared_ptr<boost::asio::io_service::work> > > >::operator() (this=<optimized out>) at /usr/local/include/boost/bind/bind_template.hpp:20
#5  boost::asio::detail::posix_thread::func<boost::_bi::bind_t<void, boost::_mfi::mf3<void, libtorrent::disk_io_thread, int, libtorrent::disk_io_thread::thread_type_t, boost::shared_ptr<boost::asio::io_service::work> >, boost::_bi::list4<boost::_bi::value<libtorrent::disk_io_thread*>, boost::_bi::value<int>, boost::_bi::value<libtorrent::disk_io_thread::thread_type_t>, boost::_bi::value<boost::shared_ptr<boost::asio::io_service::work> > > > >::run (this=<optimized out>) at /usr/local/include/boost/asio/detail/posix_thread.hpp:82
#6  0x00000000007295ff in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x7f6358007890) at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:64
#7  0x00007f636b594494 in start_thread (arg=0x7f634abd8700) at pthread_create.c:333
#8  0x00007f636aa39acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

[arvidn]

try setting settings_pack::network_threads to 0

[waTeim]

done, testing now. this takes from 1 to 5 days to occur so have 2 populations -- one with network_threads set to 0 and the other the previous value of 16. Will return with results.

[arvidn]

in fact, the network thread feature has been removed in master, and deliberately defaulted to 0 in 1.1.x, because it doesn't work.

[waTeim]

Haha, ok, well then as a followup, here are the results. After a week, I found that out of the population with network-threads set to a non zero value there were 5 crashes and in the population where it was set to 0, there were 0 crashes.