help: segfault in boost::asio::detail::epoll_reactor::start_op while upload_from_stream in multiple threads

[yxiang92128]

This issue is related to the following:
Azure/azure-storage-cpp#374

We hit a segfault while performing performance test with multiple threads (8 sessions) uploading 16 blobs simultaneously to the same bucket.
Again our environment is:
OS: SLES12SP3
Azure Storage C++ SDK: 7.0.0
REST C++ SDK: 2.10.14
libboost: 1.58

I wonder if you've seen the following trace before and any possible hint to work around it. Looks like the "descriptor_data" is 0x0 in start_op.

One of the Azure Storage C++ developer had pointed out a potential root cause due to the following:
It seems to be a known issue for boost, the linked issue has a similar trace: arvidn/libtorrent#3084
There isn't a good update in the origin post for the root cause: https://svn.boost.org/trac10/ticket/7611#no2
I believe this is a CPPRest SDK issue, there is nothing that can be done from this SDK's perspective. This conclusion is drawn due to the linked issue being resolved by disabling the network thread pool, which is purely controlled by CPPRest.

I wonder from your end, would there be a possible workaround or hardening we can do to prevent this situation (segfault) to occur?

Let me know if you find anything while I am also digging. Your immediate reponse would be highly appreciated.

Many thanks,

Yang

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f3ef4ed0565 in boost::asio::detail::epoll_reactor::start_op (this=0x7f3ce009b6f0, op_type=1, descriptor=10, descriptor_data=@0x7f3d940404d0: 0x0, op=0x7f3c2401fdf0,
is_continuation=true, allow_speculative=true) at /usr/local/include/boost/asio/detail/impl/epoll_reactor.ipp:219
219 /usr/local/include/boost/asio/detail/impl/epoll_reactor.ipp: No such file or directory.
(gdb) bt
#0 0x00007f3ef4ed0565 in boost::asio::detail::epoll_reactor::start_op (this=0x7f3ce009b6f0, op_type=1, descriptor=10, descriptor_data=@0x7f3d940404d0: 0x0, op=0x7f3c2401fdf0,
is_continuation=true, allow_speculative=true) at /usr/local/include/boost/asio/detail/impl/epoll_reactor.ipp:219
#1 0x00007f3ef4ed6a18 in boost::asio::detail::reactive_socket_service_base::start_op (this=0x7f3ce00bbf88, impl=..., op_type=1, op=0x7f3c2401fdf0, is_continuation=true,
is_non_blocking=true, noop=false) at /usr/local/include/boost/asio/detail/impl/reactive_socket_service_base.ipp:214
#2 0x00007f3ef4ff92a2 in boost::asio::detail::reactive_socket_service_base::async_send<boost::asio::mutable_buffers_1, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > > (this=0x7f3ce00bbf88, impl=..., buffers=..., flags=0, handler=...) at /usr/local/include/boost/asio/detail/reactive_socket_service_base.hpp:216
#3 0x00007f3ef4ff7210 in boost::asio::stream_socket_serviceboost::asio::ip::tcp::async_send<boost::asio::mutable_buffers_1, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > >(boost::asio::detail::reactive_socket_serviceboost::asio::ip::tcp::implementation_type&, boost::asio::mutable_buffers_1 const&, int, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >&&) (this=0x7f3ce00bbf60, impl=..., buffers=..., flags=0,
handler=<unknown type in /opt/teradata/tdobject/lib/libcpprest.so.2.10, CU 0xb62f0c, DIE 0xc5f8a4>) at /usr/local/include/boost/asio/stream_socket_service.hpp:330
#4 0x00007f3ef4ff5102 in boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >::async_write_some<boost::asio::mutable_buffers_1, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > >(boost::asio::mutable_buffers_1 const&, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >&&) (this=0x7f3d940404c8, buffers=..., handler=<unknown type in /opt/teradata/tdobject/lib/libcpprest.so.2.10, CU 0xb62f0c, DIE 0xc5c49b>)
at /usr/local/include/boost/asio/basic_stream_socket.hpp:732
#5 0x00007f3ef4ff273c in boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_strea---Type to continue, or q to quit---
mbuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >::operator() (this=0x7f3e893fc300, ec=..., bytes_transferred=0, start=1)
at /usr/local/include/boost/asio/impl/write.hpp:258
#6 0x00007f3ef4fefa18 in boost::asio::async_write<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >(boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&, boost::asio::mutable_buffers_1 const&, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > >&&) (s=..., buffers=...,
handler=<unknown type in /opt/teradata/tdobject/lib/libcpprest.so.2.10, CU 0xb62f0c, DIE 0xc53a5a>) at /usr/local/include/boost/asio/impl/write.hpp:621
#7 0x00007f3ef4fec1eb in boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > >::operator() (this=0x7f3e893fc550, ec=..., bytes_transferred=0, start=1)
at /usr/local/include/boost/asio/ssl/detail/io.hpp:189
#8 0x00007f3ef4fe8853 in boost::asio::ssl::detail::async_io<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > (next_layer=..., core=..., op=..., handler=...) at /usr/local/include/boost/asio/ssl/detail/io.hpp:337
#9 0x00007f3ef4fe494a in boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>::async_write_some<boost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > >(boost::asio::const_buffers_1 const&, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > >&&) (this=0x7f3d940f3e20, buffers=...,
handler=<unknown type in /opt/teradata/tdobject/lib/libcpprest.so.2.10, CU 0xb62f0c, DIE 0xc3fe4e>) at /usr/local/include/boost/asio/ssl/stream.hpp:646
#10 0x00007f3ef4fe0f06 in boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > >::operator() (this=0x7f3e893fcba8, ec=..., bytes_transferred=16384, start=0) at /usr/local/include/boost/asio/impl/write.hpp:334
#11 0x00007f3ef4fefabc in boost::asio::ssl::detail::write_opboost::asio::const_buffers_1::call_handler<boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > (this=0x7f3e893fcb78, handler=..., ec=...,
---Type to continue, or q to quit---
bytes_transferred=@0x7f3e893fc860: 16384) at /usr/local/include/boost/asio/ssl/detail/write_op.hpp:59
#12 0x00007f3ef4fec46f in boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > >::operator() (this=0x7f3e893fcb68, ec=..., bytes_transferred=16413, start=0)
at /usr/local/include/boost/asio/ssl/detail/io.hpp:266
#13 0x00007f3ef4ff27f0 in boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >::operator() (this=0x7f3e893fcb40, ec=..., bytes_transferred=16413,
start=0) at /usr/local/include/boost/asio/impl/write.hpp:269
#14 0x00007f3ef4ffeffb in boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>::operator() (this=0x7f3e893fcb40) at /usr/local/include/boost/asio/detail/bind_handler.hpp:127
#15 0x00007f3ef4ffef8b in boost::asio::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long> > (function=...) at /usr/local/include/boost/asio/handler_invoke_hook.hpp:69
#16 0x00007f3ef4ffee45 in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > (function=..., context=...)
at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#17 0x00007f3ef4ffeccb in boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boo---Type to continue, or q to quit---
st::system::error_code, unsigned long>, std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > (function=..., this_handler=0x7f3e893fcbd0)
at /usr/local/include/boost/asio/impl/write.hpp:699
#18 0x00007f3ef4ffeb67 in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > (
function=..., context=...) at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#19 0x00007f3ef4ffe915 in boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > (function=..., this_handler=0x7f3e893fcba8) at /usr/local/include/boost/asio/impl/write.hpp:565
#20 0x00007f3ef4ffe559 in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > (function=..., context=...) at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#21 0x00007f3ef4ffe147 in boost::asio::ssl::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > (function=..., this_handler=0x7f3e893fcb68) at /usr/local/include/boost/asio/ssl/detail/io.hpp:320
---Type to continue, or q to quit---
#22 0x00007f3ef4ffdd2a in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > (function=..., context=...)
at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#23 0x00007f3ef4ffd805 in boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > (function=..., this_handler=0x7f3e893fcb40)
at /usr/local/include/boost/asio/impl/write.hpp:565
#24 0x00007f3ef4ffd10d in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > > (function=..., context=...)
at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#25 0x00007f3ef4ffc371 in boost::asio::detail::reactive_socket_send_op<boost::asio::mutable_buffers_1, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::de---Type to continue, or q to quit---
tails::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> (*)()> > > > > > >::do_complete (owner=0x7f3ce00613e0, base=0x7f3c2401fdf0) at /usr/local/include/boost/asio/detail/reactive_socket_send_op.hpp:107
#26 0x00007f3ef4ecf550 in boost::asio::detail::task_io_service_operation::complete (this=0x7f3c2401fdf0, owner=..., ec=..., bytes_transferred=0)
at /usr/local/include/boost/asio/detail/task_io_service_operation.hpp:38
#27 0x00007f3ef4ed2058 in boost::asio::detail::task_io_service::do_run_one (this=0x7f3ce00613e0, lock=..., this_thread=..., ec=...)
at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:372
#28 0x00007f3ef4ed1a8c in boost::asio::detail::task_io_service::run (this=0x7f3ce00613e0, ec=...) at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:149
#29 0x00007f3ef4ed2345 in boost::asio::io_service::run (this=0x7f3ef53fd248 <(anonymous namespace)::initialize_shared_threadpool(unsigned long)::uninit_threadpool+8>)
at /usr/local/include/boost/asio/impl/io_service.ipp:59
#30 0x00007f3ef4fc3db9 in (anonymous namespace)::threadpool_impl::thread_start (
arg=0x7f3ef53fd240 <(anonymous namespace)::initialize_shared_threadpool(unsigned long)::uninit_threadpool>)
at /home/jenkins/cpprestsdk-2.10.14/Release/src/pplx/threadpool.cpp:79
#31 0x00007f3ef4fc3ced in (anonymous namespace)::threadpool_impl::<lambda()>::operator()(void) const (__closure=0x7f3ce0062bd8)
at /home/jenkins/cpprestsdk-2.10.14/Release/src/pplx/threadpool.cpp:64
#32 0x00007f3ef4fc47a2 in boost::asio::detail::posix_thread::func<(anonymous namespace)::threadpool_impl::add_thread()::<lambda()> >::run(void) (this=0x7f3ce0062bd0)
at /usr/local/include/boost/asio/detail/posix_thread.hpp:82
#33 0x00007f3ef4ed72c9 in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x7f3ce0062bd0) at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:64
#34 0x00007f41f94106b4 in start_thread () from /lib64/libpthread.so.0
#35 0x00007f41f2ed22dd in clone () from /lib64/libc.so.6
(gdb)
(gdb) print descriptor_data
$1 = (boost::asio::detail::epoll_reactor::per_descriptor_data &) @0x7f3d940404d0: 0x0
(gdb)

[bell-2]

Hi I have same problem....
How resolve this problem?
Thanks to give me help.

[AtsushiMorinaga]

https://stackoverflow.com/questions/21500944/is-iptcpsocket-close-thread-safe

This article says boost::asio::socket object is not thread safe.
I suppose that asio client and server should use strand to avoid race conditions.