Possible Memory Leak

[jtannas]

Hello,
We recently had some OOM errors in our system.
Looking back on our metrics, it seems like the GoodJob runners have a significant memory leak.
Here's a graph of the memory usage over the last few months, with each of the dropoffs corresponding to deploying a new version of our application.

The leak rate appears to be ~150MB per day.
Thoughts?

[bensheldon]

That looks bad! 😬 ...but I don't imagine that it's specifically GoodJob; I haven't experienced that or seen it reported before (though someone does have to be first).

I'm imagining that the cause is one or more of your application's jobs that are allocating/retaining objects, and not specifically the GoodJob framework.

My recommendation would be to do some memory profiling around jobs. There's a lot of resources on using tools like MemoryProfiler or DerailedBenchmarks to profile them. Here's an example: https://blog.appsignal.com/2022/08/10/a-deep-dive-into-memory-leaks-in-ruby.html

[jtannas]

Will do :)
I assumed it was Goodjob due to not seeing the same pattern in our API, but I'll work on narrowing it down to something specific since no else is reporting the issue.

[mengqing]

Hi @jtannas, just wondering if you have found anything that might have caused the memory leak? I, too is having memory leak issue and is having difficulty tracking it down

[jtannas]

@mengqing
There was some smaller stuff from us performing very large queries and the objects not being deallocated, but unfortunately we're still looking for the primary cause.