ATS crash during ssl handshake

[dnj12345]

Hi,
I am running into an ATS crash during ssl handshake. I am using ATS v7.1.6. Other details:

OS: Ubuntu 16.04.
Openssl: OpenSSL 1.0.2g

I have a custom plugin (using atscppapi). Previously, I was using it with 6.2.3. I am upgrading ATS to 7.1.6. With no changes to the plugin and upgrade to 7.1.6 seems to result in this crash. Release 6.2.1 or 6.2.3 doesn't exhibit this behavior. I am able to reproduce this crash very easily by sending a few thousand requests a minute. The same testbed doesn't cause any issues with 6.2.3. SSL is only enabled on the client-to-ats. The origin-server connection is over plain HTTP. Let me know if I can provide any more info. Thanks.

Dk.

PS: Trace below...

Thread 3 "[ET_NET 0]" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x2b9100907700 (LWP 6624)]
0x00002b90f6acde10 in lh_insert () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(gdb) bt full
#0  0x00002b90f6acde10 in lh_insert () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
No symbol table info available.
#1  0x00002b90f6a17314 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
No symbol table info available.
#2  0x00002b90f6a17ccb in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
No symbol table info available.
#3  0x00002b90f6ab9f47 in DH_new_method () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
No symbol table info available.
#4  0x00002b90f6abb4ee in DHparams_dup () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
No symbol table info available.
#5  0x00002b90f678af15 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
No symbol table info available.
#6  0x00002b90f678a1b3 in SSL_new () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
No symbol table info available.
#7  0x0000000000774ad6 in make_ssl_connection (ctx=<optimized out>,
    netvc=netvc@entry=0x2b911427bb50) at SSLNetVConnection.cc:143
        ssl = <optimized out>
#8  0x000000000077d594 in SSLNetVConnection::sslStartHandShake (this=0x2b911427bb50,
    event=<optimized out>, err=@0x2b9100906b60: 0) at SSLNetVConnection.cc:985
        lookup = {ptr = 0x31514e0}
        dst = {sa = {sa_family = 2,
            sa_data = "\035\035\254\027\000\002\000\000\000\000\000\000\000"}, sin = {
            sin_family = 2, sin_port = 7453, sin_addr = {s_addr = 33560492},
            sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 7453,
            sin6_flowinfo = 33560492, sin6_addr = {__in6_u = {
                __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 134364816}}
        namelen = 16
        cc = 0x0
        params = {ptr = 0x313d650}
        __FUNCTION__ = "sslStartHandShake"
#9  0x000000000077829d in SSLNetVConnection::net_read_io (this=0x2b911427bb50, nh=0x2b9100204eb0,
    lthread=0x2b9100201010) at SSLNetVConnection.cc:488
        err = 0
        ret = <optimized out>
        r = 0
        bytes = 0
---Type <return> to continue, or q <return> to quit---
        s = 0x2b911427bc78
        lock = {m = {m_ptr = 0x2b91141cd810}, lock_acquired = true}
        __FUNCTION__ = "net_read_io"
        buf = @0x2b911427bca0: {mbuf = 0x3169e90, entry = 0x0}
#10 0x000000000078d6be in NetHandler::waitForActivity (this=0x2b9100204eb0,
    timeout=<optimized out>) at UnixNet.cc:497
        epd = <optimized out>
        poll_timeout = <optimized out>
        lock = {m = {m_ptr = 0x2b90fc0038a0}, locked_p = true}
        vc = <optimized out>
        __FUNCTION__ = "waitForActivity"
#11 0x00000000007d6c2b in EThread::execute_regular (this=0x2b9100201010) at UnixEThread.cc:248
        done_one = <optimized out>
        sleep_time = 60000000
        e = <optimized out>
        NegativeQueue = {<DLL<Event, Event::Link_link>> = {head = 0x0}, tail = 0x0}
        next_time = <optimized out>
#12 0x00000000007d50c5 in spawn_thread_internal (a=0x2f31750) at Thread.cc:84
        p = 0x2f31750
#13 0x00002b90f6dff6ba in start_thread (arg=0x2b9100907700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x2b9100907700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {47901779719936, 4559030032966393076, 0,
                140729372655279, 140729372655696, 0, 7522960304432258292, 7522418851103284468},
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0,
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#14 0x00002b90f7bc141d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
(gdb)

[masaori335]

On frame 6, it's creating SSL object. But the crash is happened inside of OpenSSL.
Could you get back trace with OpenSSL debug symbols?

[dnj12345]

Agree, but the same doesn't happen for ATS 6.2.3. I am using standard openssl libraries, don't have a debug version of the ssl library. Is it possible, a wrong option was being passed to the ssl library?

…

On Mon, May 27, 2019 at 6:49 PM Masaori Koshiba ***@***.***> wrote: On frame 6, it's creating SSL object. But the crash is happened inside of OpenSSL. Could you get back trace with OpenSSL debug symbols? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5572?email_source=notifications&email_token=ACUH7IOTICBBTSFJQGMXXRDPXSFUBA5CNFSM4HPLK7ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWKXZIQ#issuecomment-496336034>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACUH7ILIRQCLLRMZ2B2G2HLPXSFUBANCNFSM4HPLK7ZA> .

[masaori335]

I am using standard openssl libraries, don't have a debug version of the ssl library.

We don't need debug built OpenSSL. I thought we can install debug symbol packages for OpenSSL (openssl-dbgsym?) and resolve some symbols.

Is it possible, a wrong option was being passed to the ssl library?

I doubt ATS 7.1.x made something wrong with SSL_CTX which is used to create SSL object.

[dnj12345]

Ok. I’ve not loaded debugsym before for OpenSSL. I’ll figure out how to do it and get back to you with the necessary info. Dk.

…

On May 27, 2019, at 10:17 PM, Masaori Koshiba ***@***.***> wrote: I am using standard openssl libraries, don't have a debug version of the ssl library. We don't need debug built OpenSSL. I thought we can install debug symbol packages for OpenSSL (openssl-dbgsym?) and resolve some symbols. Is it possible, a wrong option was being passed to the ssl library? I doubt ATS 7.1.x made something wrong with SSL_CTX which is used to create SSL object. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[bryancall]

Please reopen if you are still having issues with SSL and ATS.