Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

100% CPU usage and keep posting event 'reset_instances' to go plugin server #12364

Closed
1 task done
mong0520 opened this issue Jan 18, 2024 · 11 comments · Fixed by #12582
Closed
1 task done

100% CPU usage and keep posting event 'reset_instances' to go plugin server #12364

mong0520 opened this issue Jan 18, 2024 · 11 comments · Fixed by #12582
Labels
task/bug

Comments

@mong0520
Copy link

mong0520 commented Jan 18, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Kong version ($ kong version)

Kong 3.5 / go-pdk 0.6.0

Current Behavior

There is a self-development plugin written by Go with go-pdk v0.6.0.
It is running well and stable more than one year.
Recently we've upgraded kong from 3.4.2 to 3.5.0 and then found sometimes 100% CPU usage happens and Kong keep calling reset_instances repeatedly but always failed.

My key configuration to enable the Go plugin server:

KONG_PLUGINSERVER_NAMES=my_plugin
KONG_PLUGINSERVER_MY_PLUGIN_QUERY_CMD="/usr/local/bin/my_plugin -dump"

When this issue happens, I noticed that:

  • 100% CPU usage (see the figure below)
  • Cannot find any plugin crash logs, exception, nil pointer, errors during that period.
  • Not reproducible on my local environment but it happens more than one times on production environment.
  • Log shows:
2024/01/17 10:27:46 [warn][kong] mp_rpc.lua:340 [myPlugin] no plugin instance 31
(repeat)
// ....after few moments...
2024/01/17 10:28:10 [error][kong] mp_rpc.lua:336 [myPlugin] failed to post plugin_server reset_instances event: failed to 
publish event: queue overflow
(repeat)
image

After some quick tracing, it seems like this function https://github.com/Kong/kong/blob/3.5.0/kong/runloop/plugin_servers/mp_rpc.lua#L328-L345 is a recursive function and keep posting event reset_instance and call itself again if the error is still no plugin instance and then run out of event queue size.

The most weirdest thing is that there is NO crash log related to the plugin but kong tries to reset it and then failed.

Also I found some similar issues:
#7148
#8293
#9301

cc @samugi @StarlightIbuki (not sure you guys are familiar with this issue)

Expected Behavior

Not making CPU 100% and restart the plugin server.

Steps To Reproduce

Not reproducible

Anything else?

No response
@ms2008
Copy link
Contributor

ms2008 commented Jan 18, 2024

It is highly recommended to use the latest go-pdk version, currently 0.10.0.

We did some refactoring for this in 3.5. For details, please see:

@ms2008 ms2008 mentioned this issue Jan 18, 2024
Closed
1 task
@StarlightIbuki
Copy link
Contributor

It's a fixed bug of 3.5. This line was missing in a refactoring PR and it's then added back. Please use the nightly build or wait for 3.6 release for this fix.

@mong0520
Copy link
Author

mong0520 commented Jan 18, 2024 •
edited
Loading

@StarlightIbuki

Thank you for your prompt reply.

The file you indicate is pb_rpc.lua but the issue I faced is located in mp_rpc.lua. Could they be the same thing?

updated:
It seems this line for mp_rpc.lua is also needed but not added back yet. It exists in 3.4.2 but not 3.5.0 or master branch

@StarlightIbuki
Copy link
Contributor

@StarlightIbuki

Thank you for your prompt reply.

The file you indicate is pb_rpc.lua but the issue I faced is located in mp_rpc.lua. Could they be the same thing?

updated: It seems this line for mp_rpc.lua is also needed but not added back yet. It exists in 3.4.2 but not 3.5.0 or master branch

You're right. They are missing in the same PR.

@chronolaw chronolaw added the pending author feedback Waiting for the issue author to get back to a maintainer with findings, more details, etc... label Jan 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

This issue is marked as stale because it has been open for 14 days with no activity.

@github-actions github-actions bot added the stale label Feb 5, 2024
@mong0520
Copy link
Author

mong0520 commented Feb 5, 2024

May I know the expected release date of next version?

@github-actions github-actions bot removed the stale label Feb 6, 2024
@mong0520
Copy link
Author

@StarlightIbuki
It seems like the fix for "mp_rpc.lua" was not included in 3.6.0

@StarlightIbuki StarlightIbuki added task/bug and removed pending author feedback Waiting for the issue author to get back to a maintainer with findings, more details, etc... labels Feb 19, 2024
@StarlightIbuki StarlightIbuki mentioned this issue Feb 19, 2024
Merged
1 task
@StarlightIbuki
Copy link
Contributor

@mong0520 Sorry for replying late. The fix failed to catch up 3.6: #12582.

And Go plugin uses pb_rpc.lua as its RPC implementation thus it should not be affected by the issue. Do you see the issue occurring with the latest Kong with go plugin?

@mong0520
Copy link
Author

@StarlightIbuki
Thank you!
I understand the fix did applied to pb_rpc.lua. I just want to remind that the fix did not applied to mp_rpc.lua which means older plugin server(use messagePack but not protocol buf) might encounter this issue.

I'll try to upgrade kong from 3.5 to 3.6 along with go-pdk 0.10 to see if this issue is fixed.
Because I can still see this errors present in kong 3.5 + manual patch + go-pdk 0.10 in my environment.
image

StarlightIbuki added a commit that referenced this issue Feb 20, 2024
@StarlightIbuki @gszr
fix(core): fix an issue where Kong do not handle properly when Python…
3fa6c98 
… and Javascript plugin server crashes

The bug was introduced when refactoring/cherry-picking.

Fix #12364

Co-authored-by: Guilherme Salazar <gsz@acm.org>
@mong0520
Copy link
Author

mong0520 commented Feb 20, 2024 •
edited
Loading

@StarlightIbuki
Some feedbacks here -- Is there any memory limitation for the running plugin server?
I'm pretty sure my server has enough memory, regarding to the monitor, it has more than 50% (500MB) left.
But I got this error one time, you can scroll to the bottom and see fatal error: runtime: out of memory

Sorry about the long call stack, I have try my best to reduce it.

@timestamp @message
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc00006acb0 sp=0xc00006ac68 pc=0x545de9, context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] net.(*conn).Read(0xc000014328, {0xc0000436ac?, 0xc0000b9d70?, 0x7fc17299d5b8?}), context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/net/net.go:183 +0x45 fp=0xc00006acf8 sp=0xc00006acb0 pc=0x554185, context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] net.(*UnixConn).Read(0x7?, {0xc0000436ac?, 0xc0000b9d60?, 0x44f192?}), context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] :1 +0x29 fp=0xc00006ad28 sp=0xc00006acf8 pc=0x5676e9, context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] io.ReadAtLeast({0xc64120, 0xc000014328}, {0xc0000436ac, 0x4, 0x4}, 0x4), context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/io/io.go:332 +0x9a fp=0xc00006ad70 sp=0xc00006ad28 pc=0x47595a, context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] io.ReadFull(...), context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/io/io.go:351, context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] encoding/binary.Read({0xc64120, 0xc000014328}, {0xc6aec0, 0x111ee30}, {0xa7e720?, 0xc0000436a8}), context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/encoding/binary/binary.go:233 +0xc8 fp=0xc00006aef0 sp=0xc00006ad70 pc=0x4afba8, context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/server.readPbFrame({0xc6b630?, 0xc000014328}), context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:43 +0x6d fp=0xc00006af50 sp=0xc00006aef0 pc=0x88888d, context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/server.servePb({0xc6b630, 0xc000014328}, 0x0?), context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:19 +0x38 fp=0xc00006afb8 sp=0xc00006af50 pc=0x888738, context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/server.StartServer.func2(), context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:203 +0x2e fp=0xc00006afe0 sp=0xc00006afb8 pc=0x88982e, context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goexit(), context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00006afe8 sp=0xc00006afe0 pc=0x46a301, context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] created by github.com/Kong/go-pdk/server.StartServer, context: ngx.timer
2024-02-20 08:23:45.737 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:203 +0x13a, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000587e8 sp=0xc0000587e0 pc=0x46a301, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] created by runtime.gcBgMarkStartWorkers, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mgc.go:1199 +0x25, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] goroutine 10 [GC worker (idle)]:, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gopark(0x1705f1170de?, 0x0?, 0x0?, 0x0?, 0x0?), context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000058f50 sp=0xc000058f30 pc=0x43aef6, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gcBgMarkWorker(), context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000058fe0 sp=0xc000058f50 pc=0x41d071, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goexit(), context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000058fe8 sp=0xc000058fe0 pc=0x46a301, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] created by runtime.gcBgMarkStartWorkers, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mgc.go:1199 +0x25, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] goroutine 34 [IO wait, 2 minutes]:, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gopark(0x200000003?, 0xb?, 0x0?, 0x0?, 0x7?), context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00006ab50 sp=0xc00006ab30 pc=0x43aef6, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.netpollblock(0x4c51c5?, 0x40538f?, 0x0?), context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc00006ab88 sp=0xc00006ab50 pc=0x433517, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] internal/poll.runtime_pollWait(0x7fc14980dbe8, 0x72), context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc00006aba8 sp=0xc00006ab88 pc=0x464789, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] internal/poll.(*pollDesc).wait(0xc000296080?, 0xc0000436ac?, 0x0), context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00006abd0 sp=0xc00006aba8 pc=0x4e0712, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] internal/poll.(*pollDesc).waitRead(...), context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/internal/poll/fd_poll_runtime.go:89, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] internal/poll.(*FD).Read(0xc000296080, {0xc0000436ac, 0x4, 0x4}), context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc00006ac68 sp=0xc00006abd0 pc=0x4e1af9, context: ngx.timer
2024-02-20 08:23:45.736 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] net.(*netFD).Read(0xc000296080, {0xc0000436ac?, 0x4c7167?, 0xc0000b9ce0?}), context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000057f70 sp=0xc000057f50 pc=0x43aef6, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goparkunlock(...), context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:387, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.(*scavengerState).park(0x10ef500), context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000057fa0 sp=0xc000057f70 pc=0x423f73, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.bgscavenge(0x0?), context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000057fc8 sp=0xc000057fa0 pc=0x424565, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gcenable.func2(), context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000057fe0 sp=0xc000057fc8 pc=0x41b2a6, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goexit(), context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000057fe8 sp=0xc000057fe0 pc=0x46a301, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] created by runtime.gcenable, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mgc.go:179 +0xaa, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] goroutine 18 [finalizer wait, 16 minutes]:, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gopark(0x0?, 0xbc4d60?, 0xc0?, 0x40?, 0x2000000020?), context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000056628 sp=0xc000056608 pc=0x43aef6, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.runfinq(), context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000567e0 sp=0xc000056628 pc=0x41a347, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goexit(), context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000567e8 sp=0xc0000567e0 pc=0x46a301, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] created by runtime.createfing, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mfinal.go:163 +0x45, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] goroutine 12 [running]:, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] goroutine running on other thread; stack unavailable, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] created by github.com/Kong/go-pdk/server.StartServer, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:203 +0x13a, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] goroutine 9 [GC worker (idle)]:, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gopark(0x1705f12fbe7?, 0xc0000149e0?, 0x0?, 0x0?, 0x0?), context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000058750 sp=0xc000058730 pc=0x43aef6, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gcBgMarkWorker(), context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000587e0 sp=0xc000058750 pc=0x41d071, context: ngx.timer
2024-02-20 08:23:45.735 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goexit(), context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000064fe8 sp=0xc000064fe0 pc=0x46a301, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] goroutine 2 [force gc (idle), 2 minutes]:, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gopark(0x1705f05effb?, 0x0?, 0x0?, 0x0?, 0x0?), context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000056fb0 sp=0xc000056f90 pc=0x43aef6, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goparkunlock(...), context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:387, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.forcegchelper(), context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000056fe0 sp=0xc000056fb0 pc=0x43ad30, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goexit(), context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000056fe8 sp=0xc000056fe0 pc=0x46a301, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] created by runtime.init.6, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:293 +0x25, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] goroutine 3 [GC sweep wait]:, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?), context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000057780 sp=0xc000057760 pc=0x43aef6, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goparkunlock(...), context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:387, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.bgsweep(0x0?), context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000577c8 sp=0xc000057780 pc=0x4260be, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gcenable.func1(), context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000577e0 sp=0xc0000577c8 pc=0x41b306, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goexit(), context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000577e8 sp=0xc0000577e0 pc=0x46a301, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] created by runtime.gcenable, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mgc.go:178 +0x6b, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] goroutine 4 [GC scavenge wait]:, context: ngx.timer
2024-02-20 08:23:45.734 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gopark(0x10ef500?, 0x18a673?, 0x0?, 0x0?, 0x0?), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] goroutine 1 [IO wait]:, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000064c68 sp=0xc000064c48 pc=0x43aef6, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.netpollblock(0x7fc149816cb8?, 0x40538f?, 0x0?), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000064ca0 sp=0xc000064c68 pc=0x433517, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] internal/poll.runtime_pollWait(0x7fc14980deb8, 0x72), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000064cc0 sp=0xc000064ca0 pc=0x464789, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] internal/poll.(*pollDesc).wait(0xc000185f80?, 0x20?, 0x0), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000064ce8 sp=0xc000064cc0 pc=0x4e0712, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] internal/poll.(*pollDesc).waitRead(...), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/internal/poll/fd_poll_runtime.go:89, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] internal/poll.(*FD).Accept(0xc000185f80), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000064d90 sp=0xc000064ce8 pc=0x4e601d, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] net.(*netFD).accept(0xc000185f80), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000064e48 sp=0xc000064d90 pc=0x547f75, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] net.(*UnixListener).accept(0x443440?), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/net/unixsock_posix.go:172 +0x1c fp=0xc000064e70 sp=0xc000064e48 pc=0x56357c, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] net.(*UnixListener).Accept(0xc0002802a0), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/net/unixsock.go:260 +0x3d fp=0xc000064ea0 sp=0xc000064e70 pc=0x561edd, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/server.StartServer(0x0?, {0xc5f828, 0x7}, 0xc0000061a0?), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:198 +0x14d fp=0xc000064f50 sp=0xc000064ea0 pc=0x88970d, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] main.main(), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/src/myPlugin/main.go:175 +0x35 fp=0xc000064f80 sp=0xc000064f50 pc=0xa3a6b5, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.main(), context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc000064fe0 sp=0xc000064f80 pc=0x43aac7, context: ngx.timer
2024-02-20 08:23:45.733 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goexit(), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/bridge.readPbFrame({0xc6b630?, 0xc0000140a0}), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/bridge/bridge.go:40 +0x8c fp=0xc0000b77c8 sp=0xc0000b7768 pc=0x66aa6c, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/bridge.PdkBridge.Ask({{0xc6b630?, 0xc0000140a0?}}, {0xb864aa?, 0xc0002f02e0?}, {0xc63e40, 0xc0002f4bc0}, {0x0?, 0x0}), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/bridge/bridge.go:134 +0xcb fp=0xc0000b7818 sp=0xc0000b77c8 pc=0x66b40b, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/log.Log.doLog({{{0xc6b630?, 0xc0000140a0?}}}, {0xb864aa, 0xe}, {0xc0000b78d8?, 0x0?, 0x0?}), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/log/log.go:22 +0x8a fp=0xc0000b7878 sp=0xc0000b7818 pc=0x66faea, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/log.Log.Debug(...), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/log/log.go:50, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] main.(*myPluginService).IsReady(0xc000012a20, {0xc00013e420, 0x12}), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/src/myPlugin/myPluginService.go:168 +0x288 fp=0xc0000b7978 sp=0xc0000b7878 pc=0xa3be48, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] main.Config.Access({0x0, {0xc0002bfb80, 0xa}, 0xea60, 0x12c, 0xea60, 0x3f4ccccd}, 0xc0000026c0), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/src/myPlugin/main.go:128 +0x12e fp=0xc0000b7c30 sp=0xc0000b7978 pc=0xa39c0e, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] main.(*Config).Access(0xb544a0?, 0xc0002eb201?), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] :1 +0x85 fp=0xc0000b7cb8 sp=0xc0000b7c30 pc=0xa3c425, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/server.accesser.Access-fm(0xabf500?), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] :1 +0x31 fp=0xc0000b7cd8 sp=0xc0000b7cb8 pc=0x88abf1, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/server.handlePbEvent(0xc000262380, {0xc6b630?, 0xc000014110}, 0xc0002f4ac0), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:174 +0x2a3 fp=0xc0000b7d40 sp=0xc0000b7cd8 pc=0x8894a3, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/server.handlePbCmd(0xc00013e3a8?, {0xc6b630?, 0xc000014110?}, 0xc0002f4a80), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:147 +0x571 fp=0xc0000b7f08 sp=0xc0000b7d40 pc=0x889131, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/server.codecPb(0xc6b630?, {0xc6b630, 0xc000014110}, {0xc00013e3a8, 0x11, 0x11}), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:77 +0x8a fp=0xc0000b7f50 sp=0xc0000b7f08 pc=0x888aea, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/server.servePb({0xc6b630, 0xc000014110}, 0x0?), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:24 +0x5a fp=0xc0000b7fb8 sp=0xc0000b7f50 pc=0x88875a, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] github.com/Kong/go-pdk/server.StartServer.func2(), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:203 +0x2e fp=0xc0000b7fe0 sp=0xc0000b7fb8 pc=0x88982e, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.goexit(), context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000b7fe8 sp=0xc0000b7fe0 pc=0x46a301, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] created by github.com/Kong/go-pdk/server.StartServer, context: ngx.timer
2024-02-20 08:23:45.732 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /go/pkg/mod/github.com/!kong/go-pdk@v0.10.0/server/pbserver.go:203 +0x13a, context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mem.go:142 +0x35 fp=0x7ffc7e8f7290 sp=0x7ffc7e8f7260 pc=0x4198b5, context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.(*mheap).grow(0x1108080, 0x70281?), context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mheap.go:1522 +0x252 fp=0x7ffc7e8f7308 sp=0x7ffc7e8f7290 pc=0x42a072, context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.(*mheap).allocSpan(0x1108080, 0x70281, 0x0, 0x78?), context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mheap.go:1243 +0x1b7 fp=0x7ffc7e8f73a0 sp=0x7ffc7e8f7308 pc=0x4297b7, context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.(*mheap).alloc.func1(), context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mheap.go:961 +0x65 fp=0x7ffc7e8f73e8 sp=0x7ffc7e8f73a0 pc=0x429265, context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.systemstack(), context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x7ffc7e8f73f0 sp=0x7ffc7e8f73e8 pc=0x468149, context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] goroutine 33 [running]:, context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.systemstack_switch(), context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/asm_amd64.s:463 fp=0xc0000b7648 sp=0xc0000b7640 pc=0x4680e0, context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.(*mheap).alloc(0xe0502000?, 0x70281?, 0x90?), context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mheap.go:955 +0x65 fp=0xc0000b7690 sp=0xc0000b7648 pc=0x4291a5, context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.(*mcache).allocLarge(0xc6eaf0?, 0xe0500528, 0x1), context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mcache.go:234 +0x85 fp=0xc0000b76d8 sp=0xc0000b7690 pc=0x418945, context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.mallocgc(0xe0500528, 0xa91b20, 0x1), context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/malloc.go:1053 +0x4fe fp=0xc0000b7740 sp=0xc0000b76d8 pc=0x40f31e, context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.makeslice(0xc64120?, 0xc0000140a0?, 0xc6aec0?), context: ngx.timer
2024-02-20 08:23:45.731 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/slice.go:103 +0x52 fp=0xc0000b7768 sp=0xc0000b7740 pc=0x44f192, context: ngx.timer
2024-02-20 08:23:45.730 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime stack:, context: ngx.timer
2024-02-20 08:23:45.730 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.throw({0xb8b1a9?, 0x0?}), context: ngx.timer
2024-02-20 08:23:45.730 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7ffc7e8f7218 sp=0x7ffc7e8f71e8 pc=0x43811d, context: ngx.timer
2024-02-20 08:23:45.730 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.sysMapOS(0xc000800000, 0xe0800000?), context: ngx.timer
2024-02-20 08:23:45.730 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] /usr/local/go/src/runtime/mem_linux.go:187 +0x11b fp=0x7ffc7e8f7260 sp=0x7ffc7e8f7218 pc=0x419edb, context: ngx.timer
2024-02-20 08:23:45.730 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] runtime.sysMap(0x1108080?, 0xc3ffffffff?, 0x1118218?), context: ngx.timer
2024-02-20 08:23:45.723 2024/02/20 08:23:45 [info] 275#0: *2575 [myPlugin:278] fatal error: runtime: out of memory, context: ngx.timer

@StarlightIbuki
Copy link
Contributor

StarlightIbuki commented Feb 20, 2024 •
edited
Loading

The order of the traceback confuses me.
@mong0520 Many possible reasons can cause OOM, even if there is available memory. You could check if you've set any ulimit to the process, and see what dmesg recorded about the reason for OOM.

gszr added a commit that referenced this issue Feb 20, 2024
@StarlightIbuki @gszr
fix(pluginserver): properly restart messagepack-based instances
069da05 
The bug was introduced when refactoring/cherry-picking.

Fix #12364

Co-authored-by: Guilherme Salazar <gsz@acm.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
task/bug
Projects
None yet
Milestone
No milestone
4 participants
@chronolaw @ms2008 @mong0520 @StarlightIbuki