Service dies randomly after a few hours

[deanjenkins]

I've tried implementing your example code in an executable called m3 but it seems to die randomly after a few hours (when not being called upon over http). Here's the SIGSEV backtrace.

./m3(_Z7handleri+0x15)[0x430845]
/lib/x86_64-linux-gnu/libc.so.6(+0x35180)[0x7f5d2c65c180]
./m3(_ZN4asio3ssl6detail5io_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS5_EEEENS1_8write_opINS_15const_buffers_1EEENS_6detail8write_opINS0_6streamIS8_EESA_NSC_14transfer_all_tESt8functionIFvRKSt10error_codemEEEEEclESI_mi+0x138)[0x475ef8]
./m3(_ZN4asio6detail8write_opINS_3ssl6streamINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS6_EEEEEENS_15const_buffers_1ENS0_14transfer_all_tESt8functionIFvRKSt10error_codemEEEclESG_mi+0xea)[0x47656a]
./m3(_ZN7restbed6detail10SocketImpl5writeERKSt6vectorIhSaIhEERKSt8functionIFvRKSt10error_codemEE+0x13a)[0x472e5a]
./m3(_ZNK7restbed6detail11SessionImpl8transmitERKNS_8ResponseERKSt8functionIFvRKSt10error_codemEE+0x2aa)[0x46f7fa]
./m3(_ZN7restbed7Session5closeERKNS_8ResponseE+0x98)[0x46b4c8]
./m3(_ZN7restbed7Session5closeEiRKSt6vectorIhSaIhEERKSt8multimapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x48)[0x46c8e8]
./m3(_ZN7restbed7Session5closeEiRKSsRKSt8multimapISsSsSt4lessISsESaISt4pairIS1_SsEEE+0x2b)[0x46c9db]
./m3(_ZNK7restbed6detail11SessionImpl7failureESt10shared_ptrINS_7SessionEEiRKSt9exception+0x27b)[0x46e21b]

[ben-crowhurst]

Thanks for reporting this @deanjenkins. Please provide a code example and environmental details. In the meantime we'll attempt to replicate this issue.

[ben-crowhurst]

Source

HTTP Service example

Command

sudo time distribution/example/http_service 

Result
11+ Hours with no detrimental effect.

real    42698.358s
user         0.00s
sys          0.00s

[deanjenkins]

Thanks.

It's on Debian 8 64bit server doing nothing else.

My code does include accessing a log file locally and linking to MySQL.

However, the fault occurs not when it is being called from the web using a
GET.

I'd happily share my code but not publicly. Can I send to you direct?

Dean

On 5 November 2015 at 08:29, Corvusoft notifications@github.com wrote:

Source

HTTP Service example
https://github.com/Corvusoft/restbed/blob/master/example/http_service/source/example.cpp

Command

sudo time distribution/example/http_service

Result
11+ Hours with no detrimental effect.

real 42698.358s
user 0.00s
sys 0.00s

—
Reply to this email directly or view it on GitHub
#48 (comment).

[ben-crowhurst]

Private link is good, please send to support@corvusoft.co.uk

[ben-crowhurst]

From first impressions, nothing obvious about your code stands out to be causing the issue. I'll leave running with the MySQL logic disabled in the hope of replicating the problem.

The backtrace you've supplied clearly shows a Session::close being invoked. I can't for the life of me think how this could happen without receiving an incoming request first?

Can you confirm that the service is started and receives no requests at any time?

[ben-crowhurst]

I've pushed the following. Please attempt to replicate the issue.

[ben-crowhurst]

@deanjenkins Has this change altered your situation?

[deanjenkins]

Rebuilt and running it now. Will report back.

On 9 November 2015 at 10:58, Corvusoft notifications@github.com wrote:

@deanjenkins https://github.com/deanjenkins Has this change altered
your situation?

—
Reply to this email directly or view it on GitHub
#48 (comment).

[deanjenkins]

Stopped after a couple of hours ... with same trace. :-(

Error: signal 11:
./m3(_Z7handleri+0x15)[0x430845]
/lib/x86_64-linux-gnu/libc.so.6(+0x35180)[0x7fc6f3e16180]
./m3(_ZN4asio3ssl6detail5io_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS5_EEEENS1_8write_opINS_15const_buffers_1EEENS_6detail8write_opINS0_6streamIS8_EESA_NSC_14transfer_all_tESt8functionIFvRKSt10error_codemEEEEEclESI_mi+0x138)[0x475ef8]
./m3(_ZN4asio6detail8write_opINS_3ssl6streamINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS6_EEEEEENS_15const_buffers_1ENS0_14transfer_all_tESt8functionIFvRKSt10error_codemEEEclESG_mi+0xea)[0x47656a]
./m3(_ZN7restbed6detail10SocketImpl5writeERKSt6vectorIhSaIhEERKSt8functionIFvRKSt10error_codemEE+0x13a)[0x472e5a]
./m3(_ZNK7restbed6detail11SessionImpl8transmitERKNS_8ResponseERKSt8functionIFvRKSt10error_codemEE+0x2aa)[0x46f7fa]
./m3(_ZN7restbed7Session5closeERKNS_8ResponseE+0x98)[0x46b4c8]
./m3(_ZN7restbed7Session5closeEiRKSt6vectorIhSaIhEERKSt8multimapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x48)[0x46c8e8]
./m3(_ZN7restbed7Session5closeEiRKSsRKSt8multimapISsSsSt4lessISsESaISt4pairIS1_SsEEE+0x2b)[0x46c9db]
./m3(_ZNK7restbed6detail11SessionImpl7failureESt10shared_ptrINS_7SessionEEiRKSt9exception+0x27b)[0x46e21b]

Will comment out all the MySQL++ library calls, rebuild, and retry.

On 9 November 2015 at 16:51, Dean Jenkins dean@broadgrins.com wrote:

Rebuilt and running it now. Will report back.

On 9 November 2015 at 10:58, Corvusoft notifications@github.com wrote:

@deanjenkins https://github.com/deanjenkins Has this change altered
your situation?

—
Reply to this email directly or view it on GitHub
#48 (comment).

[deanjenkins]

Without the MySQL++ it seems to live 12 hours plus. Apologies it maybe the
database libraries that are causing the issue ... but still you'd expect
the restbed web service to survive.

I've added the MySQL calls back in and attached strace for more info.

On 9 November 2015 at 20:03, Dean Jenkins dean@broadgrins.com wrote:

Stopped after a couple of hours ... with same trace. :-(

Error: signal 11:
./m3(_Z7handleri+0x15)[0x430845]
/lib/x86_64-linux-gnu/libc.so.6(+0x35180)[0x7fc6f3e16180]

./m3(_ZN4asio3ssl6detail5io_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS5_EEEENS1_8write_opINS_15const_buffers_1EEENS_6detail8write_opINS0_6streamIS8_EESA_NSC_14transfer_all_tESt8functionIFvRKSt10error_codemEEEEEclESI_mi+0x138)[0x475ef8]

./m3(_ZN4asio6detail8write_opINS_3ssl6streamINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS6_EEEEEENS_15const_buffers_1ENS0_14transfer_all_tESt8functionIFvRKSt10error_codemEEEclESG_mi+0xea)[0x47656a]

./m3(_ZN7restbed6detail10SocketImpl5writeERKSt6vectorIhSaIhEERKSt8functionIFvRKSt10error_codemEE+0x13a)[0x472e5a]

./m3(_ZNK7restbed6detail11SessionImpl8transmitERKNS_8ResponseERKSt8functionIFvRKSt10error_codemEE+0x2aa)[0x46f7fa]
./m3(_ZN7restbed7Session5closeERKNS_8ResponseE+0x98)[0x46b4c8]

./m3(_ZN7restbed7Session5closeEiRKSt6vectorIhSaIhEERKSt8multimapISsSsSt4lessISsESaISt4pairIKSsSsEEE+0x48)[0x46c8e8]

./m3(_ZN7restbed7Session5closeEiRKSsRKSt8multimapISsSsSt4lessISsESaISt4pairIS1_SsEEE+0x2b)[0x46c9db]

./m3(_ZNK7restbed6detail11SessionImpl7failureESt10shared_ptrINS_7SessionEEiRKSt9exception+0x27b)[0x46e21b]

Will comment out all the MySQL++ library calls, rebuild, and retry.

On 9 November 2015 at 16:51, Dean Jenkins dean@broadgrins.com wrote:

Rebuilt and running it now. Will report back.

On 9 November 2015 at 10:58, Corvusoft notifications@github.com wrote:

@deanjenkins https://github.com/deanjenkins Has this change altered
your situation?

—
Reply to this email directly or view it on GitHub
#48 (comment).

[ben-crowhurst]

Restbed will not catch Segmentation Faults as they lead to undefined behaviour.

You can catch SIGSEGV using the sigaction API with the following snippet. However I'd recommend you wait until you have a better grasp of C/C++ programming and understand the side effects.

#include <cstdlib>
#include <csignal>

void handler( int signal )
{
    exit( EXIT_FAILURE );
}

int main()
{
    struct sigaction action;
    sigaction.sa_handler = handler;
    sigemptyset( &sigaction.sa_mask );
    sigaction.sa_flags = 0;

    sigaction( SIGSEGV, &action, 0 );

    return EXIT_SUCCESS;
}

It is not recommended that you handle these issues unless you can be certain you can recover. Its better to fix buggy code.