Assertion with post_transform retry

[scw00]

History

$2 = {{
    fileline = 0xab5c60 "HttpSM.cc:553", 
    function = 0xac16c0 <HttpSM::state_read_client_request_header(int, void*)::__FUNCTION__> "state_read_client_request_header", 
    event = 100, 
    reentrancy = 2
  }, {
    fileline = 0xabfc80 "HttpSM.cc:7214", 
    function = 0xac2da0 <HttpSM::set_next_state()::__FUNCTION__> "set_next_state", 
    event = 65535, 
    reentrancy = 2
  }, {
    fileline = 0xabbb80 "HttpSM.cc:4434", 
    function = 0xac2480 <HttpSM::do_cache_lookup_and_read()::__FUNCTION__> "do_cache_lookup_and_read", 
    event = 29352, 
    reentrancy = 2
  }, {
    fileline = 0xab6240 "HttpSM.cc:815", 
    function = 0xac1720 <HttpSM::state_watch_for_client_abort(int, void*)::__FUNCTION__> "state_watch_for_client_abort", 
    event = 100, 
    reentrancy = 1
  }, {
    fileline = 0xa9b2c0 "HttpCacheSM.cc:118", 
    function = 0xa9c3e0 <HttpCacheSM::state_cache_open_read(int, void*)::__FUNCTION__> "state_cache_open_read", 
    event = 1102, 
    reentrancy = -1
  }, {
    fileline = 0xab8900 "HttpSM.cc:2429", 
    function = 0xac1c80 <HttpSM::state_cache_open_read(int, void*)::__FUNCTION__> "state_cache_open_read", 
    event = 1102, 
    reentrancy = 1
  }, {
    fileline = 0xac0080 "HttpSM.cc:7329", 
    function = 0xac2da0 <HttpSM::set_next_state()::__FUNCTION__> "set_next_state", 
    event = 65535, 
    reentrancy = 1
  }, {
---Type <return> to continue, or q <return> to quit---
    fileline = 0xa9b620 "HttpCacheSM.cc:177", 
    function = 0xa9c420 <HttpCacheSM::state_cache_open_write(int, void*)::__FUNCTION__> "state_cache_open_write", 
    event = 1108, 
    reentrancy = -1
  }, {
    fileline = 0xab86c0 "HttpSM.cc:2310", 
    function = 0xac1c40 <HttpSM::state_cache_open_write(int, void*)::__FUNCTION__> "state_cache_open_write", 
    event = 1108, 
    reentrancy = 2
  }, {
    fileline = 0xabfd00 "HttpSM.cc:7227", 
    function = 0xac2da0 <HttpSM::set_next_state()::__FUNCTION__> "set_next_state", 
    event = 65535, 
    reentrancy = 2
  }, {
    fileline = 0xabd1e0 "HttpSM.cc:5545", 
    function = 0xac2760 <HttpSM::do_setup_post_tunnel(HttpVC_t)::__FUNCTION__> "do_setup_post_tunnel", 
    event = 65535, 
    reentrancy = 2
  }, {
    fileline = 0xaba160 "HttpSM.cc:3339", 
    function = 0xac2040 <HttpSM::tunnel_handler_post_ua(int, HttpTunnelProducer*)::__FUNCTION__> "tunnel_handler_post_ua", 
    event = 2302, 
    reentrancy = 2
  }, {
    fileline = 0xaba8e0 "HttpSM.cc:3685", 
    function = 0xac2180 <HttpSM::tunnel_handler_transform_write(int, HttpTunnelConsumer*)::__FUNCTION__> "tunnel_handler_transform_write", 
    event = 103, 
    reentrancy = 0
  }, {
    fileline = 0xab6860 "HttpSM.cc:1111", 
    function = 0xac17e0 <HttpSM::state_request_wait_for_transform_read(int, void*)::__FUNCTION__> "state_request_wait_for_transform_read", 
    event = 2301, 
    reentrancy = 1
---Type <return> to continue, or q <return> to quit---
  }, {
    fileline = 0xab6a80 "HttpSM.cc:1181", 
    function = 0xac18a0 <HttpSM::state_common_wait_for_transform_read(HttpTransformInfo*, int (HttpSM::*)(int, void*), int, void*)::__FUNCTION__> "state_common_wait_for_transform_read", 
    event = 2301, 
    reentrancy = 1
  }, {
    fileline = 0xab6860 "HttpSM.cc:1111", 
    function = 0xac17e0 <HttpSM::state_request_wait_for_transform_read(int, void*)::__FUNCTION__> "state_request_wait_for_transform_read", 
    event = 2000, 
    reentrancy = 1
  }, {
    fileline = 0xab6240 "HttpSM.cc:815", 
    function = 0xac1720 <HttpSM::state_watch_for_client_abort(int, void*)::__FUNCTION__> "state_watch_for_client_abort", 
    event = 104, 
    reentrancy = 1
  }, {
    fileline = 0xab7a80 "HttpSM.cc:1920", 
    function = 0xac1a60 <HttpSM::state_send_server_request_header(int, void*)::__FUNCTION__> "state_send_server_request_header", 
    event = 103, 
    reentrancy = 1
  }, {
    fileline = 0xabd060 "HttpSM.cc:5451", 
    function = 0xac2700 <HttpSM::setup_transform_to_server_transfer()::__FUNCTION__> "setup_transform_to_server_transfer", 
    event = 65535, 
    reentrancy = 1
  }, {
    fileline = 0xabaa40 "HttpSM.cc:3771", 
    function = 0xac21c0 <HttpSM::tunnel_handler_transform_read(int, HttpTunnelProducer*)::__FUNCTION__> "tunnel_handler_transform_read", 
    event = 102, 
    reentrancy = 0
  }, {
    fileline = 0xab7a80 "HttpSM.cc:1920", 
    function = 0xac1a60 <HttpSM::state_send_server_request_header(int, void*)::__FUNCTION__> "state_send_server_request_header", 
---Type <return> to continue, or q <return> to quit---
    event = 3, 
    reentrancy = 1
  }, {
    fileline = 0xabcd20 "HttpSM.cc:5326", 
    function = 0xac26c0 <HttpSM::handle_server_setup_error(int, void*)::__FUNCTION__> "handle_server_setup_error", 
    event = 3, 
    reentrancy = 1
  }, {
    fileline = 0xaba4a0 "HttpSM.cc:3463", 
    function = 0xac20c0 <HttpSM::tunnel_handler_post_server(int, HttpTunnelConsumer*)::__FUNCTION__> "tunnel_handler_post_server", 
    event = 3, 
    reentrancy = 1
  }, {
    fileline = 0xab9000 "HttpSM.cc:2599", 
    function = 0xac1d00 <HttpSM::tunnel_handler_post(int, void*)::__FUNCTION__> "tunnel_handler_post", 
    event = 2301, 
    reentrancy = 2
  }, {
    fileline = 0xabcba0 "HttpSM.cc:5217", 
    function = 0xac2680 <HttpSM::handle_post_failure()::__FUNCTION__> "handle_post_failure", 
    event = 0, 
    reentrancy = 2
  }, {
    fileline = 0xabfd00 "HttpSM.cc:7227", 
    function = 0xac2da0 <HttpSM::set_next_state()::__FUNCTION__> "set_next_state", 
    event = 65535, 
    reentrancy = 2
  }, {
    fileline = 0xab74e0 "HttpSM.cc:1656", 
    function = 0xac19c0 <HttpSM::state_http_server_open(int, void*)::__FUNCTION__> "state_http_server_open", 
    event = 200, 
    reentrancy = 3
  }, {
    fileline = 0x0, 
    function = 0x0, 
---Type <return> to continue, or q <return> to quit---
    event = 0, 
    reentrancy = 0
  } <repeats 37 times>}

#0  0x00007ffff2be1c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff2be5028 in __GI_abort () at abort.c:89
#2  0x00007ffff4bd5943 in ink_abort (message_format=0x7ffff4c1b140 "%s:%d: failed assertion `%s`") at ink_error.cc:99
#3  0x00007ffff4bd003d in _ink_assert (expression=0xabd700 "post_transform_info.vc == nullptr", file=0xab64c0 "HttpSM.cc", 
    line=5095) at ink_assert.cc:37
#4  0x00000000006becdb in HttpSM::do_post_transform_open (this=0x7fffee74e3b0) at HttpSM.cc:5095
#5  0x00000000006c12b0 in HttpSM::handle_http_server_open (this=0x7fffee74e3b0) at HttpSM.cc:5368
#6  0x00000000006bbd0e in HttpSM::do_http_server_open (this=0x7fffee74e3b0, raw=false) at HttpSM.cc:4795
#7  0x00000000006d23fd in HttpSM::set_next_state (this=0x7fffee74e3b0) at HttpSM.cc:7317
#8  0x00000000006d03e5 in HttpSM::call_transact_and_set_next_state (this=0x7fffee74e3b0, 
    f=0x714bd8 <HttpTransact::HandleResponse(HttpTransact::State*)>) at HttpSM.cc:7122
#9  0x00000000006c09c7 in HttpSM::handle_post_failure (this=0x7fffee74e3b0) at HttpSM.cc:5315
#10 0x00000000006a7ace in HttpSM::tunnel_handler_post (this=0x7fffee74e3b0, event=2301, data=0x7fffee74f640) at HttpSM.cc:2714
#11 0x00000000006a6a6a in HttpSM::main_handler (this=0x7fffee74e3b0, event=2301, data=0x7fffee74f640) at HttpSM.cc:2582
#12 0x0000000000520a9e in Continuation::handleEvent (this=0x7fffee74e3b0, event=2301, data=0x7fffee74f640)
    at /root/trafficserver/iocore/eventsystem/I_Continuation.h:153
#13 0x0000000000763a0f in HttpTunnel::main_handler (this=0x7fffee74f640, event=3, data=0x60aa00026ba0) at HttpTunnel.cc:1662
#14 0x0000000000520a9e in Continuation::handleEvent (this=0x7fffee74f640, event=3, data=0x60aa00026ba0)
    at /root/trafficserver/iocore/eventsystem/I_Continuation.h:153
#15 0x00000000006c1db2 in HttpSM::handle_server_setup_error (this=0x7fffee74e3b0, event=3, data=0x60aa00026b30) at HttpSM.cc:5425
#16 0x00000000006a111a in HttpSM::state_send_server_request_header (this=0x7fffee74e3b0, event=3, data=0x60aa00026b30)
    at HttpSM.cc:2060
#17 0x00000000006a6835 in HttpSM::main_handler (this=0x7fffee74e3b0, event=3, data=0x60aa00026b30) at HttpSM.cc:2579
#18 0x0000000000520a9e in Continuation::handleEvent (this=0x7fffee74e3b0, event=3, data=0x60aa00026b30)
    at /root/trafficserver/iocore/eventsystem/I_Continuation.h:153
#19 0x00000000009cf507 in read_signal_and_update (event=3, vc=0x60aa00026a00) at UnixNetVConnection.cc:145
#20 0x00000000009cfbfc in read_signal_done (event=3, nh=0x7ffff1ea3650, vc=0x60aa00026a00) at UnixNetVConnection.cc:206
#21 0x00000000009cfd2c in read_signal_error (nh=0x7ffff1ea3650, vc=0x60aa00026a00, lerrno=32) at UnixNetVConnection.cc:230
#22 0x00000000009d0193 in read_from_net (nh=0x7ffff1ea3650, vc=0x60aa00026a00, thread=0x7ffff1e9f800)
    at UnixNetVConnection.cc:276
#23 0x00000000009d5a15 in UnixNetVConnection::net_read_io (this=0x60aa00026a00, nh=0x7ffff1ea3650, lthread=0x7ffff1e9f800)
    at UnixNetVConnection.cc:1006
#24 0x00000000009bcecb in NetHandler::mainNetEvent (this=0x7ffff1ea3650, event=5, e=0x608e00006d60) at UnixNet.cc:509
#25 0x0000000000520a9e in Continuation::handleEvent (this=0x7ffff1ea3650, event=5, data=0x608e00006d60)
    at /root/trafficserver/iocore/eventsystem/I_Continuation.h:153
#26 0x0000000000a21aa7 in EThread::process_event (this=0x7ffff1e9f800, e=0x608e00006d60, calling_code=5) at UnixEThread.cc:129
---Type <return> to continue, or q <return> to quit---
#27 0x0000000000a2273c in EThread::execute (this=0x7ffff1e9f800) at UnixEThread.cc:256
#28 0x0000000000a200e5 in spawn_thread_internal (a=0x600800016d10) at Thread.cc:91
#29 0x00007ffff4e63b98 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.0
#30 0x00007ffff399f184 in start_thread (arg=0x7ffff1e9c700) at pthread_create.c:312
#31 0x00007ffff2ca8bed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[scw00]

The reason is that we did not clear the post_transform.vc pointer when the OS aborted. we will get a coredump if we delete this ink_assert because the TransformVConnection has already been closed.

[scw00]

That I think, we can use a fake server(fake vc) between the server and transform vc.

producer     consumer                 producer                 consumer             producer            consumer
ua ------>> transform ------>> transform ------>> fake vc ------>> fake vc ------>> server

That fakevc will collect all data and never consume unless post succeed or post failed entirely.

If we want to retry, we can just setup a new tunnel as follow.

producer      consumer             producer            consumer
ua ------>> fake vc ------>> fake vc ------>> server

[bryancall]

If this is still an issue please reopen.