Memory leak in CKMSQuantiles.java?

[mdedetrich]

At work we have noticed that one of our applications have started leaking memory. After diagnosing with VisualVM over a period of time, I have noticed that the heap size of io.prometheus.client.CKMSQuantiles$Item is gradually increasing over time without it ever lowering.

Using VisualVM, here is a screenshot of the first snaphot that we did

And here is the second, around an hour later

As you can see, the amounts of int's have also increased which seem to correspond to the members of the Quantiles Item class.

Although its expected that the number of items in the quantile increase, in our case the memory is never freed and our applications actually restart due to OOM roughly once a day. Our bucket size by default is 10 minutes, so it should be freeing up the memory shortly after that point.

We are btw using https://developer.lightbend.com/docs/cinnamon/current/plugins/prometheus/prometheus.html which uses client_java under the hood, you can also see the default configuration settings.

[brian-brazil]

Hmm, that's not good. Did you wait at least 20m before taking the first snapshot? Does the growth continue similarly at about 3MB per hour?

[mdedetrich]

I waited one hour before doing the first snapshot, and yes it does seem to continuously grow at a consistent rate.

I will double check this now as I am starting up some new nodes, will give an update tomorrow (it also doesn't appear to be entirely deterministic so I am clarifying this)

[mdedetrich]

@brian-brazil I am doing another run now, unfortunately it didn't work last night because K8s terminated our pod halfway through.

BTW, the leak is more than 3mb per hour. 3mb is just the CKMSQuantiles$Item class, but the int's are also increasing (since they are contained within the class), in the above screenshot this is roughly 19 megs (for a total of ~22 megs per hour). 22 * 24 = 528 and our pods have 1.5 gigs of memory allocated which does roughly coincide with how often they seem to be shutting down due to OOM

[mdedetrich]

Okay so last update, we don't think its due to client_java. There were issues with our JVM pods when it came to GC (within the context of Docker) that was causing the JVM to go OOM. Will close the issue and reopen if I find out its actually due to client_java

[kakaroto2]

Okay so last update, we don't think its due to client_java. There were issues with our JVM pods when it came to GC (within the context of Docker) that was causing the JVM to go OOM. Will close the issue and reopen if I find out its actually due to client_java

@mdedetrich can u describle the issues with your JVM pods? I've been meeting the same problem.

[mdedetrich]

See https://dzone.com/articles/why-my-java-application-is-oomkilled for more info