-
Notifications
You must be signed in to change notification settings - Fork 844
Closed
Labels
Description
For some time, we have seen cores from secondary threads due to static data structures associated with libraries disappearing when traffic_server process shuts down. So configuration failures log in diags.log but get obscured by unrelated cores with stacks like the following which occurred from a Fatal() due to a bad disk on cache initialization.
(gdb) bt
#0 0x00002b5a1da98387 in raise () from /lib64/libc.so.6
#1 0x00002b5a1da99a78 in abort () from /lib64/libc.so.6
#2 0x00002b5a1d2a3765 in __gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6
#3 0x00002b5a1d2a1746 in ?? () from /lib64/libstdc++.so.6
#4 0x00002b5a1d2a1773 in std::terminate() () from /lib64/libstdc++.so.6
#5 0x00002b5a1d2a22df in __cxa_pure_virtual () from /lib64/libstdc++.so.6
#6 0x00000000007973c4 in UnixNetProcessor::connect_re_internal(Continuation*, sockaddr const*, NetVCOptions*) () at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetProcessor.cc:187
#7 0x000000000058403b in connect_re (opts=0x2b5a2048f330, addr=<optimized out>, cont=0x2b5a27962800, this=<optimized out>)
at /sd/workspace/src/git.vzbuilders.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/net/P_UnixNetProcessor.h:62
#8 HttpSM::do_http_server_open(bool) () at ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:5194
#9 0x0000000000586af8 in HttpSM::set_next_state() () at ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7518
#10 0x0000000000578f86 in HttpSM::state_cache_open_write(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2502
#11 0x000000000057d0eb in HttpSM::main_handler(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2615
#12 0x00000000005e0e4d in handleEvent (data=0xffffffffffffb04f, event=1109, this=0x2b5a27962800)
at /sd/workspace/src/git.vzbuilders.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#13 handleEvent (data=0xffffffffffffb04f, event=1109, this=0x2b5a27962800)
at /sd/workspace/src/git.vzbuilders.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#14 HttpCacheSM::open_write (this=this@entry=0x2b5a279645c8, key=key@entry=0x2b5a27964688, url=<optimized out>, request=<optimized out>, old_info=<optimized out>, pin_in_cache=<optimized out>, retry=true,
allow_multiple=false) at ../../../../../../_vcs/trafficserver9/proxy/http/HttpCacheSM.cc:358
#15 0x00000000005e1024 in HttpCacheSM::state_cache_open_write(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http/HttpCacheSM.cc:231
#16 0x00000000007ded43 in handleEvent (data=0x2b5ab9203900, event=2, this=0x2b5a279645c8) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#17 handleEvent (data=0x2b5ab9203900, event=2, this=0x2b5a279645c8) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#18 EThread::process_event(Event*, int) () at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:132
#19 0x00000000007dfc94 in EThread::execute_regular (this=this@entry=0x2b5a2069ad40) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:241
#20 0x00000000007e00a6 in execute (this=0x2b5a2069ad40) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
#21 EThread::execute (this=0x2b5a2069ad40) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
#22 0x00000000007de289 in spawn_thread_internal (a=0x2b5a1eadb840) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
#23 0x00002b5a1ce2aea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00002b5a1db608dd in clone () from /lib64/libc.so.6
We should be able to call _exit() instead on fatal and emergency shutdowns to avoid the exit cleanup handlers which surprise the extra threads on these unplanned shutdowns.