Crash possibly related to PR 5952 and 6686. #6932
Description
After a week of working through and resolving more frequent crashes in prod, we ended up with hopefully one of the last remaining slow burn crashes. This crash would happen once or twice a day out-of-nowhere after seemingly running fine until then indicating that this is related to some sort of unique request or some other asynchronous/background activity. Digging deeper, I've isolated that to a spike in the metric proxy.process.hostdb.cache.total_inserts - Each of those odd crashes (we've ats9 running in a box in each of our DCs) always seem to follow a noticeable spike in hostdb inserts a minute before the crash.
That led me to look through all the recent HostDB changes which may be causing the issue and to isolate further, I've reverted PR #5952 (and the follow-up PR #6686 ) which seems to have fixed the issue. Will need to run this through the weekend to confirm but with those two PRs reverted, haven't seen any crash since yesterday.
Below are some example stack traces that were seen. In most cases, the stack traces don't seem to give much to go on, as they are only symptoms of what appears to be some sort of memory corruption.
(gdb) bt
#0 0x00002b1c1476106f in _Unwind_IteratePhdrCallback (info=<optimized out>, size=<optimized out>, ptr=0x2b1c21901980) at ../.././libgcc/unwind-dw2-fde-dip.c:398
#1 0x00002b1c14aa242c in dl_iterate_phdr () from /lib64/libc.so.6
#2 0x00002b1c14761501 in _Unwind_Find_FDE (pc=0x2b1c16d38e81 <lixer::LixPlugin::handleReadRequestHeadersPreRemap(atscppapi::Transaction&)+845>, bases=bases@entry=0x2b1c21901b08) at ../.././libgcc/unwind-dw2-fde-dip.c:469
#3 0x00002b1c1475da43 in uw_frame_state_for (context=context@entry=0x2b1c21901a60, fs=fs@entry=0x2b1c21901b50) at ../.././libgcc/unwind-dw2.c:1249
#4 0x00002b1c1475f988 in _Unwind_Backtrace (trace=0x2b1c14a79da0 <backtrace_helper>, trace_argument=0x2b1c21901d10) at ../.././libgcc/unwind.inc:290
#5 0x00002b1c14a79f16 in backtrace () from /lib64/libc.so.6
#6 0x00002b1c12064a43 in ink_stack_trace_dump () at ink_stack_trace.cc:63
#7 0x00002b1c120798b3 in signal_crash_handler (signo=signo@entry=11) at signals.cc:180
#8 0x00000000004c42de in crash_logger_invoke (signo=11, info=0x2b1c219022b0, ctx=0x2b1c21902180) at traffic_server/Crash.cc:173
#9 <signal handler called>
#10 0x00000000004cf935 in ~Ptr (this=0x2b1c21902e70, __in_chrg=<optimized out>) at ../include/tscore/Ptr.h:215
#11 MutexTryLock::~MutexTryLock (this=0x2b1c21902e70, __in_chrg=<optimized out>) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Lock.h:550
#12 0x00000000004ecd02 in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6097
#13 0x00002b1c2f4df5ad in UreGenerator::handleReadRequestHeadersPreRemap (this=0x2b1c15fffa80, transaction=...) at /export/content/data/multiproduct-post-commit/i001/workspace/ats-plugin-userrequestevent_78f261cbcb2e5016edb9902f1accbb4e89878e6f/ats-plugin-userrequestevent/main/src/UreGenerator.cc:311
#14 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b1ccb9c0490) at GlobalPlugin.cc:65
#15 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c17f53ba0, event=60016, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
#16 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60016, this=0x2b1c17f53ba0) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#17 APIHook::invoke (this=this@entry=0x2b1c15d75ec0, event=60016, edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
#18 0x000000000054ef77 in HttpSM::state_api_callout (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1500
#19 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#20 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#21 0x00002b1c16d38e82 in lixer::LixPlugin::handleReadRequestHeadersPreRemap (this=0x2b1c17bcafc0, transaction=...) at lixer.cc:127
#22 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b1ccb9c0490) at GlobalPlugin.cc:65
#23 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c17f53e20, event=60016, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
#24 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60016, this=0x2b1c17f53e20) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#25 APIHook::invoke (this=this@entry=0x2b1c15d75ee0, event=60016, edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
#26 0x000000000054ef77 in HttpSM::state_api_callout (this=0x2b1ccb9c0490, event=event@entry=0, data=data@entry=0x0) at HttpSM.cc:1500
#27 0x000000000054f8df in HttpSM::do_api_callout_internal (this=<optimized out>) at HttpSM.cc:5335
#28 0x000000000055e4b5 in HttpSM::do_api_callout (this=this@entry=0x2b1ccb9c0490) at HttpSM.cc:365
#29 0x00000000005588b3 in HttpSM::set_next_state (this=0x2b1ccb9c0490) at HttpSM.cc:7408
#30 0x00000000005433f2 in HttpSM::call_transact_and_set_next_state (this=this@entry=0x2b1ccb9c0490, f=f@entry=0x0) at HttpSM.cc:7374
#31 0x000000000055392a in HttpSM::handle_api_return (this=0x2b1ccb9c0490) at HttpSM.cc:1635
#32 0x000000000054f4ae in HttpSM::state_api_callout (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1567
#33 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#34 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#35 0x00002b1c417f72b6 in setSceLixValue (contp=0x2b1c56111560, event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b1ccb9c0490) at HttpDebugger.cc:109
#36 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c56111560, event=60002, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
#37 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60002, this=0x2b1c56111560) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#38 APIHook::invoke (this=this@entry=0x2b1c15d75ac0, event=60002, edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
#39 0x000000000054ef77 in HttpSM::state_api_callout (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1500
#40 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#41 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#42 0x00002b1c3ac2d3b2 in cont_rewrite_headers (contp=0x2b1c17f52f20, event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b1ccb9c0490) at header_rewrite.cc:332
#43 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c17f52f20, event=60002, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
#44 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60002, this=0x2b1c17f52f20) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#45 APIHook::invoke (this=this@entry=0x2b1c15d75ce0, event=60002, edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
#46 0x000000000054ef77 in HttpSM::state_api_callout (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1500
#47 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#48 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#49 0x00002b1c3a977ff1 in dc_selection::EntityRoutingPlugin::handleReadRequestHeaders (this=0x2b1c15c210c0, transaction=...) at EntityRoutingPlugin.cc:342
#50 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b1ccb9c0490) at GlobalPlugin.cc:65
#51 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c17f53060, event=60002, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
#52 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60002, this=0x2b1c17f53060) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#53 APIHook::invoke (this=this@entry=0x2b1c15d75d00, event=60002, edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
#54 0x000000000054ef77 in HttpSM::state_api_callout (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1500
#55 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#56 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#57 0x00002b1c340a9bf9 in TraceHeadersPlugin::handleReadRequestHeaders (this=0x2b1c15c06bc0, transaction=...) at trace-headers.cc:875
#58 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b1ccb9c0490) at GlobalPlugin.cc:65
#59 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c17f53a60, event=60002, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
---Type <return> to continue, or q <return> to quit---q
#0 ssl_callback_session_ticket (ssl=0x2b588f91e000, keyname=0x2b57d2f043f0 "\260D\360\322W+", iv=0x2b57d2f043e0 "", cipher_ctx=0x2b585887cd40, hctx=0x2b582398e700, enc=1) at SSLSessionTicket.cc:67
67 SSLSessionTicket.cc: No such file or directory.
(gdb) bt
#0 ssl_callback_session_ticket (ssl=0x2b588f91e000, keyname=0x2b57d2f043f0 "\260D\360\322W+", iv=0x2b57d2f043e0 "", cipher_ctx=0x2b585887cd40, hctx=0x2b582398e700, enc=1) at SSLSessionTicket.cc:67
#1 0x00002b57c96a1718 in tls_construct_new_session_ticket () from /lib/libssl.so.1.1
#2 0x00002b57c96933f7 in state_machine () from /lib/libssl.so.1.1
#3 0x00002b57c9668dea in ssl3_read_bytes () from /lib/libssl.so.1.1
#4 0x00002b57c96708fa in ssl3_read () from /lib/libssl.so.1.1
#5 0x00002b57c967c07d in ssl_read_internal () from /lib/libssl.so.1.1
#6 0x00002b57c967c253 in SSL_read () from /lib/libssl.so.1.1
#7 0x0000000000754315 in SSLReadBuffer (ssl=0x2b588f91e000, buf=0x2b589ce95000, nbytes=nbytes@entry=4096, nread=@0x2b57d2f04968: 0) at SSLUtils.cc:1851
#8 0x00000000007439fa in ssl_read_from_net (ret=<synthetic pointer>, lthread=<optimized out>, sslvc=0x2b58dcf86330) at SSLNetVConnection.cc:275
#9 SSLNetVConnection::net_read_io (this=0x2b58dcf86330, nh=0x2b57ce223d80, lthread=<optimized out>) at SSLNetVConnection.cc:665
#10 0x0000000000762498 in NetHandler::process_ready_list (this=this@entry=0x2b57ce223d80) at UnixNet.cc:412
#11 0x000000000076278d in NetHandler::waitForActivity (this=0x2b57ce223d80, timeout=<optimized out>) at UnixNet.cc:547
#12 0x00000000007c6a9a in EThread::execute_regular (this=this@entry=0x2b57ce220000) at UnixEThread.cc:266
#13 0x00000000007c6d62 in EThread::execute (this=0x2b57ce220000) at UnixEThread.cc:327
#14 0x00000000007c5109 in spawn_thread_internal (a=0x2b57cc56de80) at Thread.cc:92
#15 0x00002b57ca315dd5 in start_thread () from /lib64/libpthread.so.0
#16 0x00002b57cb0c6ead in clone () from /lib64/libc.so.6
(gdb) info thr
Id Target Id Frame
#9 <signal handler called>
#10 std::_Rb_tree_insert_and_rebalance (__insert_left=<optimized out>, __x=0x2b4c234e0fa0, __p=<optimized out>, __header=...) at ../../../.././libstdc++-v3/src/c++98/tree.cc:237
#11 0x00002b4c2c3528aa in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, AtsPluginUtils::CaseInsensitiveStringComparator, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >::_M_insert_<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, AtsPluginUtils::CaseInsensitiveStringComparator, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >::_Alloc_node>(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&&, std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, AtsPluginUtils::CaseInsensitiveStringComparator, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >::_Alloc_node&) (this=0x2b4c2350bb20, __x=0x0, __p=0x2b4c224e0d00, __v=<unknown type in /lib64/libtrafficserver-plugin-utils.so, CU 0x3ec0e, DIE 0x730ae>, __node_gen=...) at /usr/include/c++/7.3.0/bits/stl_tree.h:1755
#12 0x00002b4c2c350df2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, AtsPluginUtils::CaseInsensitiveStringComparator, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >::_M_insert_unique<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&&) (this=0x2b4c2350bb20, __v=<unknown type in /lib64/libtrafficserver-plugin-utils.so, CU 0x3ec0e, DIE 0x756e9>) at /usr/include/c++/7.3.0/bits/stl_tree.h:2096
#13 0x00002b4c2c34f124 in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, AtsPluginUtils::CaseInsensitiveStringComparator, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >::insert(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&&) (this=0x2b4c2350bb20, __x=<unknown type in /lib64/libtrafficserver-plugin-utils.so, CU 0x3ec0e, DIE 0x77e21>) at /usr/include/c++/7.3.0/bits/stl_map.h:803
#14 0x00002b4c2c34b442 in AtsPluginUtils::ClientRequestImpl::appendHeaderField (this=0x2b4c2350ba00, name=..., value=...) at ClientRequestImpl.cc:565
#15 0x00002b4c2c34b310 in AtsPluginUtils::ClientRequestImpl::setHeaderField (this=0x2b4c2350ba00, name=..., value=...) at ClientRequestImpl.cc:555
#16 0x00002b4c2c3474f9 in AtsPluginUtils::ClientRequest::setHeaderField (this=0x2b4c235fe8e0, name=..., value=...) at ClientRequest.cc:171
#17 0x00002b4c2f84c5a3 in PluginControlInternals::addPluginToSignalHeader (pluginName=..., clientRequest=..., pLogger=0x2b4c2422c100) at PluginHeaderSignal.cc:137
#18 0x00002b4c2f83f7f6 in PluginControls::canIrun (this=0x2b4c20503350) at PluginControls.cc:564
#19 0x00002b4c2f30f2ee in UreGenerator::handleReadRequestHeadersPreRemap (this=0x2b4c15fffb80, transaction=...) at /export/content/data/multiproduct-post-commit/i001/workspace/ats-plugin-userrequestevent_78f261cbcb2e5016edb9902f1accbb4e89878e6f/ats-plugin-userrequestevent/main/src/UreGenerator.cc:280
#20 0x00002b4c2ba44c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b4c25107950) at GlobalPlugin.cc:65
#21 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b4c17b7bba0, event=60016, edata=0x2b4c25107950) at traffic_server/InkAPI.cc:1096
#22 0x00000000004ea60b in handleEvent (data=0x2b4c25107950, event=60016, this=0x2b4c17b7bba0) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#23 APIHook::invoke (this=this@entry=0x2b4c15d75ec0, event=60016, edata=edata@entry=0x2b4c25107950) at traffic_server/InkAPI.cc:1333
#24 0x000000000054ef77 in HttpSM::state_api_callout (this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1500
#25 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#26 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b4c25107950, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#27 0x00002b4c2b81de82 in lixer::LixPlugin::handleReadRequestHeadersPreRemap (this=0x2b4c16ff2fc0, transaction=...) at lixer.cc:127
#28 0x00002b4c2ba44c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b4c25107950) at GlobalPlugin.cc:65
#29 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b4c17b7be20, event=60016, edata=0x2b4c25107950) at traffic_server/InkAPI.cc:1096
#30 0x00000000004ea60b in handleEvent (data=0x2b4c25107950, event=60016, this=0x2b4c17b7be20) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#31 APIHook::invoke (this=this@entry=0x2b4c15d75ee0, event=60016, edata=edata@entry=0x2b4c25107950) at traffic_server/InkAPI.cc:1333
#32 0x000000000054ef77 in HttpSM::state_api_callout (this=0x2b4c25107950, event=event@entry=0, data=data@entry=0x0) at HttpSM.cc:1500
#33 0x000000000054f8df in HttpSM::do_api_callout_internal (this=<optimized out>) at HttpSM.cc:5335
#34 0x000000000055e4b5 in HttpSM::do_api_callout (this=this@entry=0x2b4c25107950) at HttpSM.cc:365
#35 0x00000000005588b3 in HttpSM::set_next_state (this=0x2b4c25107950) at HttpSM.cc:7408
#36 0x00000000005433f2 in HttpSM::call_transact_and_set_next_state (this=this@entry=0x2b4c25107950, f=f@entry=0x0) at HttpSM.cc:7374
#37 0x000000000055392a in HttpSM::handle_api_return (this=0x2b4c25107950) at HttpSM.cc:1635
38 0x000000000054f4ae in HttpSM::state_api_callout (this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1567
#39 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#40 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b4c25107950, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#41 0x00002b4c41bf72b6 in setSceLixValue (contp=0x2b4c54d3a560, event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b4c25107950) at HttpDebugger.cc:109
#42 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b4c54d3a560, event=60002, edata=0x2b4c25107950) at traffic_server/InkAPI.cc:1096
#43 0x00000000004ea60b in handleEvent (data=0x2b4c25107950, event=60002, this=0x2b4c54d3a560) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
---Type <return> to continue, or q <return> to quit---q