Skip to content

Conversation

@shinrich
Copy link
Member

One of our groups was seeing the following stack track

[ 00 ] libc-2.12.so        __GI___waitpid                                                               
[ 01 ] traffic_server      crash_logger_invoke                                                          ( Crash.cc:165 )
[ 02 ] libc-2.12.so                                                                                     
[ 03 ] traffic_server      mime_hdr_field_find                                                          ( MIME.cc:1319 )
[ 04 ] traffic_server      mime_hdr_prepare_for_value_set                                               ( MIME.cc:1692 )
[ 05 ] traffic_server      MIMEHdr::value_set(char const*, int, char const*, int)                       ( MIME.h:1419 )
[ 06 ] traffic_server      HttpSM::do_drain_request_body()                                              ( HttpSM.cc:5755 )
[ 07 ] traffic_server      HttpSM::set_next_state()                                                     ( HttpSM.cc:7601 )
[ 08 ] traffic_server      HttpSM::call_transact_and_set_next_state(void (*)(HttpTransact::State*))     ( HttpSM.cc:7335 )
[ 09 ] traffic_server      HttpSM::handle_api_return()                                                  ( HttpSM.cc:1661 )
[ 10 ] traffic_server      HttpSM::state_api_callout(int, void*)                                        ( HttpSM.cc:1596 )
[ 11 ] traffic_server      HttpSM::state_api_callback(int, void*)                                       ( HttpSM.cc:1363 )
[ 12 ] traffic_server      TSHttpTxnReenable                                                            ( InkAPI.cc:5921 )
[ 13 ] esi.so              globalHookHandler                                                            ( esi.cc:1659 )
[ 14 ] traffic_server      INKContInternal::handle_event(int, void*)                                    ( InkAPI.cc:1064 )
[ 15 ] traffic_server      Continuation::handleEvent(int, void*)                                        ( I_Continuation.h:153 )
[ 16 ] traffic_server      APIHook::invoke(int, void*)                                                  ( InkAPI.cc:1284 )
[ 17 ] traffic_server      HttpSM::state_api_callout(int, void*)                                        ( HttpSM.cc:1523 )
[ 18 ] traffic_server      HttpSM::do_api_callout_internal()                                            ( HttpSM.cc:5270 )
[ 19 ] traffic_server      HttpSM::do_api_callout()                                                     ( HttpSM.cc:438 )
[ 20 ] traffic_server      HttpSM::set_next_state()                                                     ( HttpSM.cc:7368 )
[ 21 ] traffic_server      HttpSM::call_transact_and_set_next_state(void (*)(HttpTransact::State*))     ( HttpSM.cc:7335 )
[ 22 ] traffic_server      HttpSM::handle_api_return()                                                  ( HttpSM.cc:1661 )
[ 23 ] traffic_server      HttpSM::state_api_callout(int, void*)                                        ( HttpSM.cc:1596 )
[ 24 ] traffic_server      HttpSM::state_api_callback(int, void*)                                       ( HttpSM.cc:1363 )
[ 25 ] traffic_server      TSHttpTxnReenable                                                            ( InkAPI.cc:5921 )
[ 26 ] libatscppapi.so.7   (anonymous namespace)::handleTransactionEvents(tsapi_cont*, TSEvent, void*)  ( utils_internal.cc:93 )
[ 27 ] traffic_server      INKContInternal::handle_event(int, void*)                                    ( InkAPI.cc:1064 )
[ 28 ] traffic_server      Continuation::handleEvent(int, void*)                                        ( I_Continuation.h:153 )
[ 29 ] traffic_server      APIHook::invoke(int, void*)                                                  ( InkAPI.cc:1284 )
[ 30 ] traffic_server      HttpSM::state_api_callout(int, void*)                                        ( HttpSM.cc:1523 )
[ 31 ] traffic_server      HttpSM::do_api_callout_internal()                                            ( HttpSM.cc:5270 )
[ 32 ] traffic_server      HttpSM::do_api_callout()                                                     ( HttpSM.cc:438 )
[ 33 ] traffic_server      HttpSM::set_next_state()                                                     ( HttpSM.cc:7368 )
[ 34 ] traffic_server      HttpSM::call_transact_and_set_next_state(void (*)(HttpTransact::State*))     ( HttpSM.cc:7335 )
[ 35 ] traffic_server      HttpSM::state_cache_open_read(int, void*)                                    ( HttpSM.cc:2671 )
[ 36 ] traffic_server      HttpSM::main_handler(int, void*)                                             ( HttpSM.cc:2733 )
[ 37 ] traffic_server      Continuation::handleEvent(int, void*)                                        ( I_Continuation.h:153 )
[ 38 ] traffic_server      HttpCacheSM::state_cache_open_read(int, void*)                               ( HttpCacheSM.cc:132 )
[ 39 ] traffic_server      Continuation::handleEvent(int, void*)                                        ( I_Continuation.h:153 )
[ 40 ] traffic_server      CacheVC::callcont(int)                                                       ( P_CacheInternal.h:653 )
[ 41 ] traffic_server      CacheVC::openReadStartHead(int, Event*)                                      ( CacheRead.cc:1158 )
[ 42 ] traffic_server      Continuation::handleEvent(int, void*)                                        ( I_Continuation.h:153 )
[ 43 ] traffic_server      CacheVC::handleReadDone(int, Event*)                                         ( Cache.cc:2469 )
[ 44 ] traffic_server      Continuation::handleEvent(int, void*)                                        ( I_Continuation.h:153 )
[ 45 ] traffic_server      AIOCallbackInternal::io_complete(int, void*)                                 ( P_AIO.h:117 )
[ 46 ] traffic_server      Continuation::handleEvent(int, void*)                                        ( I_Continuation.h:153 )
[ 47 ] traffic_server      EThread::process_event(Event*, int)                                          ( UnixEThread.cc:131 )
[ 48 ] traffic_server      EThread::process_queue(Queue<Event, Event::Link_link>*, int*, int*)          ( UnixEThread.cc:170 )
[ 49 ] traffic_server      EThread::execute_regular()                                                   ( UnixEThread.cc:230 )
[ 50 ] traffic_server      EThread::execute()                                                           ( UnixEThread.cc:325 )
[ 51 ] traffic_server      spawn_thread_internal                                                        ( Thread.cc:85 )
[ 52 ] libpthread-2.12.so  start_thread                                                                 

In the stack trace, t_state.hdr_info.client_response is not yet initialized, so eventually things crash because of null heaps. Looking through the response logic, I think it is sufficient the just set t_state.client_info.keep_alive. The build_response logic will look at that flag and set the connection header appropriately. We have been running with that patch internally and it seems to have fixed the issue.

@shinrich shinrich added this to the 9.0.0 milestone Feb 11, 2019
@shinrich shinrich self-assigned this Feb 11, 2019
@shinrich shinrich force-pushed the remove-crashing-header-manipulation branch from 086b2ed to 67a959c Compare February 12, 2019 17:06
@shinrich
Copy link
Member Author

Updated the PR based on autest failures. We do need to set the Connection header, but there are two cases for transform and non-transform transactions. The response headers differ. So moved the do_drain_request_header into the if-else clauses and pass in the appropriate header to update.

@shinrich shinrich merged commit 985894d into apache:master Feb 13, 2019
@bryancall bryancall modified the milestones: 9.0.0, 8.1.0 Mar 27, 2019
@bryancall
Copy link
Contributor

Cherry picked to 8.1.0

@zwoop zwoop modified the milestones: 8.1.0, 8.1.0-nogo Mar 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants