Crash on shutdown when compiled against openssl 1.1.1

[shinrich]

Since building with openssl 1.1.1, we frequently see a crash on shutdown from one of our plugin threads. It looks like one of openssl's static table gets freed before the plugin threads get shutdown.

{code}
(gdb) bt
#0 0x00002ba48896e00b in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
#1 0x00002ba4881b0b69 in CRYPTO_THREAD_write_lock (lock=) at crypto/threads_pthread.c:73
#2 0x00002ba48816f425 in RAND_get_rand_method () at crypto/rand/rand_lib.c:684
#3 0x00002ba48816f680 in RAND_seed (buf=0x2ba4a55fbcc8, num=8) at crypto/rand/rand_lib.c:735
#4 0x00002ba49fb5aabe in ycrEVPgenerateEnvironment (context=0x2ba4a5a06500, info=0x2ba49fb4ba60 <channel_key> "tuxPczXpHfdmPMWxnZ9Hg.JPQ7T45yroXLyb4sDypKI-", infoLength=44) at _vcs/ysecure/ycr/evpCypher.c:111
#5 0x00002ba49fb5e300 in ycrCreateSymmetricContext (context=context@entry=0x2ba4a55fbd38, secret=0x2ba49fb4ba60 <channel_key> "tuxPczXpHfdmPMWxnZ9Hg.JPQ7T45yroXLyb4sDypKI-", secretLen=secretLen@entry=44)
at _vcs/ysecure/ycr/ycr_sign.c:263
#6 0x00002ba49f92fd52 in decrypt_session (
encrypted_data="cpaHBaRqbuAWuXnR7AGBQ3KY1utf0Ys0pV9p8zTyF4RVJFkHjm4JjVM3uu1SOZZZF29zX1oXSjkm7hRpT_0E1ma2soNaOxaVyr1JFp07pPyKH4xtve58R1PcxppL_N7PxgrGblBYRlAbr2uxAp1E.U4tr1PIy8JLyzX6PwJNBmPS5vEJXHdmbDoLRZQwgfZBlbFyRElM"..., key=, key_length=key_length@entry=44,
session_data=session_data@entry=0x2ba4a55fbdc0 "0\201\263\002\001\001\002\002\003\003\004\002\300/\004 \227[\016\205\254\065\311G\363\240\261of\362=\327\023v\376i\302\t\253-\275\340\250G\233\254l\232\004\060\323\064\255\244(\243yN\001\201вQ\365\334v\350a\317g\206\356\344\272\310T/\352N\266\326,z\024\200\360\004\251\245rG"\334\026\265\070\066r\241\006\002\004[\230\261j\242\004\002\002\034 \244"\004 !S\343?&\226"ۻ8\026凇\035z>\001\373e\212[ٻq;\221zGT3\a\246\033\004\031caldav.calendar.yahoo.com\255\003\002\001\001\001\001", session_data_len=@0x2ba4a55fbd84: 10240) at _vcs/ssl_session_reuse-7/ssl_session_reuse/ats_plugin/session_process.cc:96
#7 0x00002ba49f9300a8 in add_session (session_id=session_id@entry=0x2ba4a55fe640 "$;\311\352i\266\350۬\033iۚx\300\255MqWt\003\065ԾD\357\207\005\246A"q", session_id_len=32,
encrypted_session="cpaHBaRqbuAWuXnR7AGBQ3KY1utf0Ys0pV9p8zTyF4RVJFkHjm4JjVM3uu1SOZZZF29zX1oXSjkm7hRpT_0E1ma2soNaOxaVyr1JFp07pPyKH4xtve58R1PcxppL_N7PxgrGblBYRlAbr2uxAp1E.U4tr1PIy8JLyzX6PwJNBmPS5vEJXHdmbDoLRZQwgfZBlbFyRElM"...) at _vcs/ssl_session_reuse-7/ssl_session_reuse/ats_plugin/session_process.cc:178
#8 0x00002ba49f93d42e in RedisSubscriber::run (this=0x2ba48c70a6e0) at _vcs/ssl_session_reuse-7/ssl_session_reuse/ats_plugin/subscriber.cc:190
#9 0x00002ba49f93d669 in setup_subscriber (arg=) at _vcs/ssl_session_reuse-7/ssl_session_reuse/ats_plugin/subscriber.cc:19
#10 0x00000000004f2f3f in ink_thread_trampoline (data=0x2ba4a2c83bc0) at InkIOCoreAPI.cc:128
#11 0x00002ba48896adc5 in start_thread () from /lib64/libpthread.so.0
#12 0x00002ba48969976d in clone () from /lib64/libc.so.6
{code}

[shinrich]

So rand_cleaning_up is 1 in the core, which means rand_cleanup_int has been called. Which means OPENSSL_cleanup has been called.

One other observation. An openssl change in August nulls the rand mutex after deleting the mutex. The previous code would only delete the mutex. So perhaps in our tests with the prerelease openssl 1.1.1 we got lucky with the stale mutex reference as the system was shutting down.

This harsh shutdown much also explain some of the other intermittent shutdown crashes we see in the accept and connect loops.

[shinrich]

Closing this issue. @duke8253 fixed our plugin's issue by adding code to run when the shutdown lifecycle hook goes off. That code sets a flag that the plugin specific threads watch. When the flag is set, the threads exit themselves, so they are not around to make SSL calls as the openssl library is freeing it's data structures.