Direct buffer memory leak in HttpClient

[veita]

Jetty version
Jetty 10.0.21

Java version/vendor

openjdk 21.0.3 2024-04-16 LTS
OpenJDK Runtime Environment Temurin-21.0.3+9 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (build 21.0.3+9-LTS, mixed mode, sharing)

OS type/version
Windows Server 2019

Description
The latest Solr 9.6.1 Java client which uses the org.eclipse.jetty.client.HttpClient internally suffers from direct ByteBuffer memory leaks which probably come from Jetty's internal ByteBuffer pools.

A detailed description of the problem is provided by https://issues.apache.org/jira/browse/SOLR-17376 in Solr's bug tracker.

Keywords: HttpClient, DirectByteBuffer, RetainableByteBuffer, ArrayRetainableByteBufferPool

[lorban]

By any chance, have you collected a heap dump of the JVM that encountered that leak that you could share with us?

Thanks!

[alexanderveit]

I have a heap dump. Unfortunately I cannot share it because it contains user data and security credentials.

[lorban]

Here is a short list of things from this heap dump that could help us:

• the capacity of the leaked buffers
• if they're direct or not
• the length of the array holding onto them

Could you please also confirm that you're using the default buffer pool? If not, please share how exactly you set up the pool that you configure on the client.

As a workaround, you could set a tuned byte buffer pool onto your HttpClient instance before starting it by calling setByteBufferPool(new LogarithmicArrayByteBufferPool(-1, -1, -1, -1, -1, -2, -2)) See https://javadoc.jetty.org/jetty-10/org/eclipse/jetty/io/LogarithmicArrayByteBufferPool.html#%3Cinit%3E(int,int,int,long,long,long,long) for the details about each of those parameters.

The thing the above workaround would do is to avoid using a retainable pool (thanks to the last two -2 parameters). Non-retainable pools are slightly slower than retainable ones, but they are leak-resistant: when a buffer isn't explicitly re-pooled, the GC can collect it so leaked buffers do no accumulate around.

Let us know if this helps.

[veita]

See https://issues.apache.org/jira/browse/SOLR-17376

The buffers are direct. The java.nio direct buffer pool graph shows how the memory leaks evolve over time. The server gets unresponsive when about 7 to 8 GB of direct buffers are created.

Random samples from the 458485 DirectByteBuffers suggest that their size is 16384. The DirectByteBuffer shown is a typical example.

Here's another example with inbound references shown, starting from ArrayRetainableByteBufferPool.

I'm not sure that we're using the default buffer pool because we're only using the Jetty client indirectly through the Solr client. The code in https://issues.apache.org/jira/browse/SOLR-17376 shows how it is created. The Jetty client is encapsulated by the Solr client, so we don't have any influence on how the Jetty client is configured.

[dsmiley]

You can create a Jetty HttpClient, customize it as you wish, and provide it to Solr's Builder.

[olamy]

I have a heap dump. Unfortunately I cannot share it because it contains user data and security credentials.

In case it helps. Webtide (https://webtide.com/what-we-do/) is providing commercial support for Jetty and so can sign a NDA as part of the commercial support.

[lorban]

An alternative to using a leak-resistant pool is to wait for the soon-to-be-released Jetty 10.0.23 and use the new leak tracking pool so the pool can collect data about the leaked buffers; but you would still need to collect a heap dump and either share it with us or analyze it on your own.