Segmentation fault, when the client closes the session while the server is inside the callback

[omidnikta]

I have a server that a callback to a request takes a while. Meanwhile if the client closes the connection, I got a segmentation fault in the server. Here is the full backtrace

(gdb) bt
#0  0x000055555578bbd3 in asio::ssl::detail::engine::perform(int (asio::ssl::detail::engine::*)(void*, unsigned long), void*, unsigned long, std::error_code&, unsigned long*) ()
#1  0x000055555578ebde in asio::ssl::detail::io_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::ssl::detail::write_op<asio::const_buffers_1>, asio::detail::write_op<asio::ssl::stream<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> > >, asio::const_buffers_1, asio::detail::transfer_all_t, std::function<void (std::error_code const&, unsigned long)> > >::operator()(std::error_code, unsigned long, int) ()
#2  0x000055555578f464 in asio::detail::write_op<asio::ssl::stream<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> > >, asio::const_buffers_1, asio::detail::transfer_all_t, std::function<void (std::error_code const&, unsigned long)> >::operator()(std::error_code const&, unsigned long, int) ()
#3  0x000055555578b390 in restbed::detail::SocketImpl::write(std::vector<unsigned char, std::allocator<unsigned char> > const&, std::function<void (std::error_code const&, unsigned long)> const&) ()
#4  0x00005555557874dc in restbed::detail::SessionImpl::transmit(restbed::Response const&, std::function<void (std::error_code const&, unsigned long)> const&) const ()
#5  0x0000555555781f47 in restbed::Session::close(restbed::Response const&) ()
#6  0x0000555555783fe3 in restbed::Session::close(int, std::vector<unsigned char, std::allocator<unsigned char> > const&, std::multimap<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) ()
#7  0x0000555555784146 in restbed::Session::close(int, std::string const&, std::multimap<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) ()
#8  0x0000555555785a2d in restbed::detail::SessionImpl::failure(std::shared_ptr<restbed::Session>, int, std::exception const&) const ()
#9  0x0000555555780512 in std::_Function_handler<void (std::error_code const&, unsigned long), restbed::Session::close(restbed::Response const&)::{lambda(std::error_code const&, unsigned long)#1}>::_M_invoke(std::_Any_data const&, std::error_code const&, unsigned long&&) ()
#10 0x000055555578c2b7 in asio::detail::write_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::const_buffers_1, asio::detail::transfer_all_t, std::function<void (std::error_code const&, unsigned long)> >::operator()(std::error_code const&, unsigned long, int) ()
#11 0x000055555578c45e in asio::detail::reactive_socket_send_op<asio::const_buffers_1, asio::detail::write_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::const_buffers_1, asio::detail::transfer_all_t, std::function<void (std::error_code const&, unsigned long)> > >::do_complete(asio::detail::task_io_service*, asio::detail::task_io_service_operation*, std::error_code const&, unsigned long) ()
#12 0x0000555555752d0b in std::thread::_Impl<std::_Bind_simple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2} ()> >::_M_run()
    ()
#13 0x00007ffff5411350 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>)
    at /build/gcc-multilib/src/gcc-5.2.0/libstdc++-v3/src/c++11/thread.cc:84
#14 0x00007ffff5dd24a4 in start_thread () from /usr/lib/libpthread.so.0
#15 0x00007ffff4b8913d in clone () from /usr/lib/libc.so.6

[ben-crowhurst]

Thanks for highlighting this issue. Would it be possible to supply a working example and appropriate instructions to replicate this defect?

[omidnikta]

Here is a sample. Run it as "./main n" which "n" is a number. After running make a rest request by

curl -H "Content-Type: application/json" -X POST -d 'hi' http://localhost:1984/resource

and then kill this curl execution (before 5 seconds). In my system if "n" is 100000, I get a crash. But smaller numbers are ok (I don't know the exact threshold).

#include <memory>
#include <cstdlib>
#include <restbed>
#include <string>
#include <sstream>
#include <unistd.h>
#include <iostream>

using namespace std;
using namespace restbed;

int *num;

void post_method_handler( const shared_ptr< Session > session )
{
    const auto request = session->get_request( );

    int content_length = 0;
    request->get_header( "Content-Length", content_length );

    session->fetch( content_length, [ ]( const shared_ptr< Session > session, const Bytes & body )
    {
        fprintf( stdout, "%.*s\n", ( int ) body.size( ), body.data( ) );
        stringstream ss;
        for (int i = 0; i < *num; ++i)
            ss << "abcdef";
        sleep(5);
        session->close( OK, ss.str());
    } );
}

int main( const int argc, const char** argv)
{
    if (argc < 1) {
        cout << "Usage main <n>" << endl;
        return EXIT_SUCCESS;
    }
    num = new int(stoi(argv[1]));
    auto resource = make_shared< Resource >( );
    resource->set_path( "/resource" );
    resource->set_method_handler( "POST", post_method_handler );

    auto settings = make_shared< Settings >( );
    settings->set_port( 1984 );
    settings->set_default_header( "Connection", "close" );

    Service service;
    service.publish( resource );
    service.start( settings );

    return EXIT_SUCCESS;
}

[ben-crowhurst]

I've attempted to replicate this issue on the following environments with varying configurations of sleep (1,5,10,15,30) and loop iteration limits (10, 1000, 10000, 100000, 500000).

• FreeBSD
• Windows 7
• CentOS
• Ubuntu
• Solaris

As of yet I've failed to reproduce the defect.

I can see from your back trace you have SSL enabled and potentially a custom failure handler? Would you be willing to provide the exact code, configuration and environment details you are experiencing this issue in?

[omidnikta]

Interesting, I installed restbed on Archlinux from this AUR
https://aur.archlinux.org/packages/restbed/

and compiled the above sample file with

g++ main.cpp -o main -lcrypto -lssl -lrestbed -lpthread -std=c++11

The same problem happened on Ubuntu.

And here is the backtrace of the above sample

(gdb) bt
#0 0x0000000000450e0d in asio::ssl::detail::engine::perform(int (asio::ssl::detail::engine::)(void, unsigned long), void_, unsigned long, std::error_code&, unsigned long_) ()
#1 0x00000000004521ef in asio::ssl::detail::io_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_serviceasio::ip::tcp >, asio::ssl::detail::write_opasio::const_buffers_1, asio::detail::write_op<asio::ssl::stream<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_serviceasio::ip::tcp > >, asio::const_buffers_1, asio::detail::transfer_all_t, std::function<void (std::error_code const&, unsigned long)> > >::operator()(std::error_code, unsigned long, int) ()
#2 0x0000000000452d99 in asio::detail::write_op<asio::ssl::stream<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_serviceasio::ip::tcp > >, asio::const_buffers_1, asio::detail::transfer_all_t, std::function<void (std::error_code const&, unsigned long)> >::operator()(std::error_code const&, unsigned long, int) ()
#3 0x00000000004505db in restbed::detail::SocketImpl::write(std::vector<unsigned char, std::allocator > const&, std::function<void (std::error_code const&, unsigned long)> const&) ()
#4 0x000000000044cc1d in restbed::detail::SessionImpl::transmit(restbed::Response const&, std::function<void (std::error_code const&, unsigned long)> const&) const ()
#5 0x0000000000447cb8 in restbed::Session::close(restbed::Response const&) ()
#6 0x0000000000449b08 in restbed::Session::close(int, std::vector<unsigned char, std::allocator > const&, std::multimap<std::string, std::string, std::lessstd::string, std::allocator<std::pair<std::string const, std::string> > > const&) ()
#7 0x0000000000449c2b in restbed::Session::close(int, std::string const&, std::multimap<std::string, std::string, std::lessstd::string, std::allocator<std::pair<std::string const, std::string> > > const&) ()
#8 0x000000000044b2d8 in restbed::detail::SessionImpl::failure(std::shared_ptrrestbed::Session, int, std::exception const&) const ()
#9 0x0000000000446308 in std::_Function_handler<void (std::error_code const&, unsigned long), restbed::Session::close(restbed::Response const&)::{lambda(std::error_code const&, unsigned long)#1}>::M_invoke(std::Any_data const&, std::error_code const&, unsigned long&&) ()
#10 0x0000000000451324 in asio::detail::write_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_serviceasio::ip::tcp >, asio::const_buffers_1, asio::detail::transfer_all_t, std::function<void (std::error_code const&, unsigned long)> >::operator()(std::error_code const&, unsigned long, int) ()
#11 0x0000000000451516 in asio::detail::reactive_socket_send_op<asio::const_buffers_1, asio::detail::write_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_serviceasio::ip::tcp >, asio::const_buffers_1, asio::detail::transfer_all_t, std::function<void (std::error_code const&, unsigned long)> > >::do_complete(void, asio::detail::scheduler_operation, std::error_code const&, unsigned long) ()
#12 0x000000000041f66b in restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) ()
#13 0x000000000040909b in main ()

[ben-crowhurst]

Ok, could you attempt to pull and build the latest to see if this change fixes your issue?

Restbed Build

git clone --recursive https://github.com/corvusoft/restbed.git
mkdir restbed/build
cd restbed/build
cmake -DBUILD_TESTS=YES -DBUILD_EXAMPLES=YES ..
make install
make test

Example Build

g++ -o test test.cpp -lrestbed -lpthread -I restbed/distribution/include -L restbed/distribution/library -std=c++11

4.0 Release is scheduled for December so hopefully this will resolve the problem. I'll carry on testing from my end as well; Its just nice to have a second opinion.

[omidnikta]

Thanks, I admit that the master branch doesn't have this issue. Please close the issue.

[ben-crowhurst]

Great news @omidnikta, thanks for all your help.

The linked commit resolves this issue and will be available in the 4.0 release this December.