ats92/master crash

[traeak]

stack trace ats92 (9df7f8d)
live linear cache ram drives only, testing session mutex congestion. Turned on http_ss debug but also for fun made a custom debug tag with session lifetime information which also causes the same crash.

Note CacheVC::openReadFromWriterFailure, failure to attach read while writer.

#0  0x00007f1ecdf44aff in raise () from /lib64/libc.so.6
#1  0x00007f1ecdf17ea5 in abort () from /lib64/libc.so.6
#2  0x00007f1ed040cace in ink_abort (
    message_format=0x7f1ed0479fc8 "%s:%d: failed assertion `%s`")
    at ink_error.cc:99
#3  0x00007f1ed04083cb in _ink_assert (
    expression=0xb0e2f5 "!new_hdr->valid()", 
    file=0xb0e268 "HttpTransactHeaders.cc", line=209) at ink_assert.cc:37
#4  0x000000000083437f in HttpTransactHeaders::copy_header_fields (
    src_hdr=0x7f1e593a55a0, new_hdr=0x7f1e593a5620, 
    retain_proxy_auth_hdrs=false, date=0) at HttpTransactHeaders.cc:209
#5  0x0000000000825147 in HttpTransact::build_request (s=0x7f1e593a4ef8, 
    base_request=0x7f1e593a55a0, outgoing_request=0x7f1e593a5620, 
    outgoing_version=...) at HttpTransact.cc:7767
#6  0x000000000080c601 in HttpTransact::HandleCacheOpenReadMiss (
    s=0x7f1e593a4ef8) at HttpTransact.cc:3367
#7  0x0000000000805479 in HttpTransact::HandleCacheOpenRead (s=0x7f1e593a4ef8)
    at HttpTransact.cc:2440
#8  0x00000000007dad12 in HttpSM::call_transact_and_set_next_state (
    this=0x7f1e593a4df0, f=0x0) at HttpSM.cc:7589
#9  0x00000000007bbc15 in HttpSM::handle_api_return (this=0x7f1e593a4df0)
    at HttpSM.cc:1706
#10 0x00000000007bb90b in HttpSM::state_api_callout (this=0x7f1e593a4df0, 
    event=0, data=0x0) at HttpSM.cc:1638
#11 0x00000000007d10b1 in HttpSM::do_api_callout_internal (this=0x7f1e593a4df0)
    at HttpSM.cc:5555
#12 0x00000000007e4fc3 in HttpSM::do_api_callout (this=0x7f1e593a4df0)
    at HttpSM.cc:421
#13 0x00000000007e500a in HttpSM::setup_cache_lookup_complete_api (
    this=0x7f1e593a4df0) at HttpSM.cc:2648
#14 0x00000000007c1cf8 in HttpSM::state_cache_open_read (this=0x7f1e593a4df0, 
    event=1103, data=0xffffffffffffb04f) at HttpSM.cc:2709
#15 0x00000000007c2190 in HttpSM::main_handler (this=0x7f1e593a4df0, 
    event=1103, data=0xffffffffffffb04f) at HttpSM.cc:2753
#16 0x00000000007186f1 in Continuation::handleEvent (this=0x7f1e593a4df0, 
    event=1103, data=0xffffffffffffb04f)
    at /home/bolsen149/git/tsb-ats92/src/ats/iocore/eventsystem/I_Continuation.h:227
#17 0x00000000008574b5 in HttpCacheSM::state_cache_open_read (
    this=0x7f1e593a6c18, event=1103, data=0xffffffffffffb04f)
    at HttpCacheSM.cc:128
#18 0x00000000007186f1 in Continuation::handleEvent (this=0x7f1e593a6c18, 
    event=1103, data=0xffffffffffffb04f)
    at /home/bolsen149/git/tsb-ats92/src/ats/iocore/eventsystem/I_Continuation.h:227
#19 0x00000000009ab64d in CacheVC::openReadFromWriterFailure (this=
    0x7f1e80caf200, event=1103, e=0xffffffffffffb04f) at CacheRead.cc:189
#20 0x00000000009ac521 in CacheVC::openReadFromWriter (this=0x7f1e80caf200, 
    event=2, e=0x7f1d145e39c0) at CacheRead.cc:348
#21 0x00000000007186f1 in Continuation::handleEvent (this=0x7f1e80caf200, 
    event=2, data=0x7f1d145e39c0)
    at /home/bolsen149/git/tsb-ats92/src/ats/iocore/eventsystem/I_Continuation.h:227
#22 0x0000000000acd599 in EThread::process_event (this=0x7f1ecc5da010, 
    e=0x7f1d145e39c0, calling_code=2) at UnixEThread.cc:164
#23 0x0000000000acdb04 in EThread::execute_regular (this=0x7f1ecc5da010)
    at UnixEThread.cc:273
#24 0x0000000000acde6b in EThread::execute (this=0x7f1ecc5da010)
    at UnixEThread.cc:364
#25 0x0000000000acc628 in spawn_thread_internal (a=0x1455350) at Thread.cc:91
#26 0x00007f1ecedf71cf in start_thread () from /lib64/libpthread.so.0
#27 0x00007f1ecdf2fe73 in clone () from /lib64/libc.so.6

[traeak]

The only way I can make this crash happen is with a vastly under resourced box being heavily cpu congested (all cores slammed). Turning on debug pushes it over the edge, even using an invalid tag (like XXX). Crash seems to still be valid but is some nasty race condition.