[s3_auth] Crash on reloading

[masaori335]

[ 00 ] libpthread-2.17.so    __libc_waitpid                                                                                                                                                                     
[ 01 ] traffic_server        crash_logger_invoke                                                                                                                                                                ( Crash.cc:168 )
[ 02 ] libpthread-2.17.so                                                                                                                                                                                       
[ 03 ] <anonymous>                                                                                                                                                                                              
[ 04 ] s3_auth.so (deleted)                                                                                                                                                                                     
[ 05 ] traffic_server                                                                                                                                                                                           ( RemapPluginInfo.cc:214 )
[ 06 ] traffic_server        done                                                                                                                                                                               ( PluginFactory.cc:66 )
[ 07 ] traffic_server        operator()                                                                                                                                                                         ( PluginFactory.cc:281 )
[ 08 ] traffic_server        operator()                                                                                                                                                                         ( IntrusiveDList.h:865 )
[ 09 ] traffic_server        Intrusive_DList_Apply<ts::IntrusiveLinkage<RemapPluginInst, &RemapPluginInst::_next, &RemapPluginInst::_prev>, (lambda at ../../../include/tscpp/util/IntrusiveDList.h:865:23) &>  ( IntrusiveDList.h:876 )
[ 10 ] traffic_server        apply<(lambda at ../../../include/tscpp/util/IntrusiveDList.h:865:23)>                                                                                                             ( IntrusiveDList.h:887 )
[ 11 ] traffic_server        Intrusive_DList_Apply<ts::IntrusiveLinkage<RemapPluginInst, &RemapPluginInst::_next, &RemapPluginInst::_prev>, (lambda at PluginFactory.cc:281:19) &>                              ( IntrusiveDList.h:865 )
[ 12 ] traffic_server        apply<(lambda at PluginFactory.cc:281:19)>                                                                                                                                         ( IntrusiveDList.h:887 )
[ 13 ] traffic_server        PluginFactory::deactivate()                                                                                                                                                        ( PluginFactory.cc:281 )
[ 14 ] traffic_server        UrlRewrite::~UrlRewrite()                                                                                                                                                          ( UrlRewrite.cc:116 )
[ 15 ] traffic_server        UrlRewrite::~UrlRewrite()                                                                                                                                                          ( UrlRewrite.cc:104 )
[ 16 ] traffic_server        DeleterContinuation<UrlRewrite>::dieEvent(int, void*)                                                                                                                              ( iocore/eventsystem/P_Freer.h:42 )
[ 17 ] traffic_server        handleEvent                                                                                                                                                                        ( I_Continuation.h:227 )
[ 18 ] traffic_server        EThread::process_event(Event*, int)                                                                                                                                                ( UnixEThread.cc:153 )
[ 19 ] traffic_server        EThread::process_queue(Queue<Event, Event::Link_link>*, int*, int*)                                                                                                                ( UnixEThread.cc:188 )
[ 20 ] traffic_server        EThread::execute_regular()                                                                                                                                                         ( UnixEThread.cc:248 )
[ 21 ] traffic_server        EThread::execute()                                                                                                                                                                 ( UnixEThread.cc:353 )
[ 22 ] traffic_server        spawn_thread_internal                                                                                                                                                              ( Thread.cc:79 )
[ 23 ] libpthread-2.17.so    start_thread                                                                                                                                                                       

[masaori335]

The dynamic reloading plugin is one of the factors. It appears that this is not related.

proxy.config.plugin.dynamic_reload_mode: 1

[masaori335]

I got repro. The steps are below.

1. load the s3_auth plugin with the expiration setting in the past - .e.g, expiration=1660768600
2. change the expiration to 0 and wait 1 min for the s3_auth plugin to reload the config automatically
3. touch remap.config
4. run traffic_ctl config reload

traffic_server: received signal 10 (Bus error: 10)
traffic_server - STACK TRACE:
0   traffic_server                      0x000000000d88cd80 _Z19crash_logger_invokeiP9__siginfoPv + 112
1   libsystem_platform.dylib            0x00007ff817055c1d _sigtramp + 29
2   ???                                 0x0000000000000048 0x0 + 72
3   s3_auth.so                          0x000000000e468bf3 TSRemapDeleteInstance + 19
4   traffic_server                      0x000000000d9a8833 _ZN15RemapPluginInfo12doneInstanceEPv + 35
5   traffic_server                      0x000000000d9ad231 _ZN13PluginFactory10deactivateEv + 177
6   traffic_server                      0x000000000d9b4149 _ZN10UrlRewriteD2Ev + 153
7   traffic_server                      0x000000000d9b44be _ZN10UrlRewriteD0Ev + 14
8   traffic_server                      0x000000000dae6748 _ZN19DeleterContinuationI10UrlRewriteE8dieEventEiPv + 24
9   traffic_server                      0x000000000db6fe0f _ZN7EThread13process_eventEP5Eventi + 607
10  traffic_server                      0x000000000db7020a _ZN7EThread13process_queueEP5QueueI5EventNS1_9Link_linkEEPiS5_ + 458
11  traffic_server                      0x000000000db7060a _ZN7EThread15execute_regularEv + 442
12  traffic_server                      0x000000000db70be5 _ZN7EThread7executeEv + 373
13  traffic_server                      0x000000000db6f2eb _ZL21spawn_thread_internalPv + 75
14  libsystem_pthread.dylib             0x00007ff81702b259 _pthread_start + 125
15  libsystem_pthread.dylib             0x00007ff817026c7b thread_start + 15

The problem is the s3_auth plugin's auto-reloading config feature scheduled an event that is tracked by S3Config:: _conf_rld_act. However, it's not cleared when the event is handled. So when ATS reloads the remap config, the destructor of S3Config tries to cancel the already done event.

This is fixed by #9329.