Skip to content

Cleanup crashes due to configuration start up failures #7394

@shinrich

Description

@shinrich

For some time, we have seen cores from secondary threads due to static data structures associated with libraries disappearing when traffic_server process shuts down. So configuration failures log in diags.log but get obscured by unrelated cores with stacks like the following which occurred from a Fatal() due to a bad disk on cache initialization.

(gdb) bt
#0  0x00002b5a1da98387 in raise () from /lib64/libc.so.6
#1  0x00002b5a1da99a78 in abort () from /lib64/libc.so.6
#2  0x00002b5a1d2a3765 in __gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6
#3  0x00002b5a1d2a1746 in ?? () from /lib64/libstdc++.so.6
#4  0x00002b5a1d2a1773 in std::terminate() () from /lib64/libstdc++.so.6
#5  0x00002b5a1d2a22df in __cxa_pure_virtual () from /lib64/libstdc++.so.6
#6  0x00000000007973c4 in UnixNetProcessor::connect_re_internal(Continuation*, sockaddr const*, NetVCOptions*) () at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetProcessor.cc:187
#7  0x000000000058403b in connect_re (opts=0x2b5a2048f330, addr=<optimized out>, cont=0x2b5a27962800, this=<optimized out>)
    at /sd/workspace/src/git.vzbuilders.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/net/P_UnixNetProcessor.h:62
#8  HttpSM::do_http_server_open(bool) () at ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:5194
#9  0x0000000000586af8 in HttpSM::set_next_state() () at ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7518
#10 0x0000000000578f86 in HttpSM::state_cache_open_write(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2502
#11 0x000000000057d0eb in HttpSM::main_handler(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2615
#12 0x00000000005e0e4d in handleEvent (data=0xffffffffffffb04f, event=1109, this=0x2b5a27962800)
    at /sd/workspace/src/git.vzbuilders.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#13 handleEvent (data=0xffffffffffffb04f, event=1109, this=0x2b5a27962800)
    at /sd/workspace/src/git.vzbuilders.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#14 HttpCacheSM::open_write (this=this@entry=0x2b5a279645c8, key=key@entry=0x2b5a27964688, url=<optimized out>, request=<optimized out>, old_info=<optimized out>, pin_in_cache=<optimized out>, retry=true, 
    allow_multiple=false) at ../../../../../../_vcs/trafficserver9/proxy/http/HttpCacheSM.cc:358
#15 0x00000000005e1024 in HttpCacheSM::state_cache_open_write(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http/HttpCacheSM.cc:231
#16 0x00000000007ded43 in handleEvent (data=0x2b5ab9203900, event=2, this=0x2b5a279645c8) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#17 handleEvent (data=0x2b5ab9203900, event=2, this=0x2b5a279645c8) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#18 EThread::process_event(Event*, int) () at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:132
#19 0x00000000007dfc94 in EThread::execute_regular (this=this@entry=0x2b5a2069ad40) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:241
#20 0x00000000007e00a6 in execute (this=0x2b5a2069ad40) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
#21 EThread::execute (this=0x2b5a2069ad40) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
#22 0x00000000007de289 in spawn_thread_internal (a=0x2b5a1eadb840) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
#23 0x00002b5a1ce2aea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00002b5a1db608dd in clone () from /lib64/libc.so.6

We should be able to call _exit() instead on fatal and emergency shutdowns to avoid the exit cleanup handlers which surprise the extra threads on these unplanned shutdowns.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions