Skip to content

Conversation

@masaori335
Copy link
Contributor

@masaori335 masaori335 commented Jan 23, 2023

I faced below assertion failure with old expiration config of s3_auth plugin.

[Jan 23 16:34:50.071] [ET_NET 9] DIAG: (s3_auth) config expiration time is in the past, re-checking in 1 minute
Fatal: traffic_server/InkAPI.cc:1113: failed assertion `!"not reached"`
2023-01-23 16:34:50.071619+0900 traffic_server[79864:35081143] Fatal: traffic_server/InkAPI.cc:1113: failed assertion `!"not reached"`
Process 79864 stopped
* thread #11, name = '[ET_NET 9]', stop reason = signal SIGABRT
    frame #0: 0x00007ff818bcd30e libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`:
->  0x7ff818bcd30e <+10>: jae    0x7ff818bcd318            ; <+20>
    0x7ff818bcd310 <+12>: movq   %rax, %rdi
    0x7ff818bcd313 <+15>: jmp    0x7ff818bc6dc2            ; cerror_nocancel
    0x7ff818bcd318 <+20>: retq
(lldb) bt
* thread #11, name = '[ET_NET 9]', stop reason = signal SIGABRT
  * frame #0: 0x00007ff818bcd30e libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007ff818c04f7b libsystem_pthread.dylib`pthread_kill + 263
    frame #2: 0x00007ff818b4eca5 libsystem_c.dylib`abort + 123
    frame #3: 0x00000000019b1faa libtscore.10.dylib`ink_abort(message_format="%s:%d: failed assertion `%s`") at ink_error.cc:99:3
    frame #4: 0x00000000019adebc libtscore.10.dylib`::_ink_assert(expression="!\"not reached\"", file="traffic_server/InkAPI.cc", line=1113) at ink_assert.cc:37:3
    frame #5: 0x0000000000033532 traffic_server`INKContInternal::handle_event_count(this=0x000000000303dcd0, event=1) at InkAPI.cc:1113:7
    frame #6: 0x000000000004a08f traffic_server`::TSActionCancel(actionp=0x00000000030344c1) at InkAPI.cc:7138:10
    frame #7: 0x0000000006f0b76f s3_auth.so`S3Config::schedule_conf_reload(this=0x0000000001e71b00, delay=60) at s3_auth.cc:489:7
    frame #8: 0x0000000006f0b11d s3_auth.so`config_reloader(cont=0x000000000303dcd0, event=TS_EVENT_TIMEOUT, edata=0x00000000030344c0) at s3_auth.cc:1070:11
    frame #9: 0x0000000000032f42 traffic_server`INKContInternal::handle_event(this=0x000000000303dcd0, event=2, edata=0x00000000030344c0) at InkAPI.cc:1137:29
    frame #10: 0x00000000000162ce traffic_server`Continuation::handleEvent(this=0x000000000303dcd0, event=2, data=0x00000000030344c0) at I_Continuation.h:227:12
    frame #11: 0x0000000000533564 traffic_server`EThread::process_event(this=0x0000000018228000, e=0x00000000030344c0, calling_code=2) at UnixEThread.cc:152:22
    frame #12: 0x0000000000533e4b traffic_server`EThread::execute_regular(this=0x0000000018228000) at UnixEThread.cc:258:11
    frame #13: 0x00000000005344e5 traffic_server`EThread::execute(this=0x0000000018228000) at UnixEThread.cc:337:11
    frame #14: 0x00000000005327b0 traffic_server`spawn_thread_internal(a=0x000060000211c000) at Thread.cc:79:12
    frame #15: 0x00007ff818c05259 libsystem_pthread.dylib`_pthread_start + 125
    frame #16: 0x00007ff818c00c7b libsystem_pthread.dylib`thread_start + 15

It looks like the S3Config::_conf_rld_act is tracking the reloading action. However, if the reloading is scheduled again, it tries to cancel the ongoing event. The event should be clear when the event handler starts handling the event.

void
schedule_conf_reload(long delay)
{
if (_conf_rld_act != nullptr && !TSActionDone(_conf_rld_act)) {
TSActionCancel(_conf_rld_act);
}
_conf_rld_act = TSContScheduleOnPool(_conf_rld, delay * 1000, TS_THREAD_POOL_NET);
}

@masaori335 masaori335 added this to the 10.0.0 milestone Jan 23, 2023
@masaori335 masaori335 requested a review from duke8253 January 23, 2023 08:04
@masaori335 masaori335 self-assigned this Jan 23, 2023
@bryancall bryancall requested a review from ezelkow1 January 24, 2023 00:06
@masaori335
Copy link
Contributor Author

It turned out this fixes #9331. We need this for 9.2.1.

@masaori335 masaori335 merged commit 6ec45d5 into apache:master Jan 25, 2023
@masaori335 masaori335 linked an issue Jan 25, 2023 that may be closed by this pull request
zwoop pushed a commit that referenced this pull request Jan 25, 2023
@zwoop
Copy link
Contributor

zwoop commented Jan 25, 2023

Cherry-picked to v9.2.x

@zwoop zwoop modified the milestones: 10.0.0, 9.2.1 Jan 25, 2023
masaori335 pushed a commit to masaori335/trafficserver that referenced this pull request Feb 21, 2023
* asf/9.2.x:
  Updated ChangeLog
  Make 204 cacheable again (apache#9333)
  s3_auth: Fix assertion failure of TSActionCancel (apache#9329)
cmcfarlen pushed a commit to cmcfarlen/trafficserver that referenced this pull request Jun 3, 2024
* asf/master: (623 commits)
  records.config to records.yaml (apache#9264)
  Updates the release roadmap, adjusting for delays (apache#9360)
  Upgrades master branch to use clang-format v15.0.7 (apache#9355)
  Disable merging on GitHub (apache#9354)
  Clang-format 15.0.7 is finicky, and does not like these old school array inits (apache#9356)
  Enable merging for 10-Dev merge (apache#9353)
  Fix an error on SSL config reload (plus some cleanup). (apache#9334)
  Cleanup of legacy, makes newer clang-format crash right now (apache#9350)
  Update the roadmap / branch management doc page (apache#9340)
  Proxy Protocol out fixes (apache#9341)
  Memory leaks with storing configuration filenames (apache#9324)
  s3_auth autest: convert from gold file to file contains (apache#9337)
  Make 204 cacheable again (apache#9333)
  Add param to forward headers from the auth server to the origin (apache#9271)
  s3_auth: Fix assertion failure of TSActionCancel (apache#9329)
  Added http connect Autest with proxy verifier (apache#9315)
  s3_auth: Schedule reloading config event on TASK thread (apache#9328)
  Register ET_UDP thread type even if no UDP threads are requested (apache#9314)
  Don't send response body on status 204 No Content (apache#9330)
  Limit the serching range of static table by the first letter of header name (apache#9298)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[s3_auth] Crash on reloading

3 participants