Number of pending connections not being limited

[schreibaer]

In our productive system we seem to be having the problem that, during high traffic periods, thousands of pending connections are being opened even though a limit has been configured.

ConnectionProvider connectionProvider = builder
    .maxConnections(50)
    .pendingAcquireTimeout(Duration.ofMillis(50))
    .pendingAcquireMaxCount(50)
    .maxIdleTime(Duration.ofMinutes(5))
    .evictInBackground(Duration.ofMinutes(2))
    .disposeTimeout(Duration.ofSeconds(5))
    .build();

Expected Behavior

The number of connections should be limited according to the configuration

Actual Behavior

Thousands of pending connections are opened.

In the graph above, the scale on the right shows the number of pending connections, while the scale on the left is for the number of active connections. As can be seen, the number of active connections stays below the configured threshold of 50, while the pending connections reaches 35K.

The number of connections is only reduced once the application instances have been terminated due to OutOfMemory exceptions

Steps to Reproduce

Since the problem has only been observed during high load periods, providing steps for the reproduction is not simple. In essence, a service providing slow responses is required to ensure a buildup in the connections. Then starting a load test using the configuration described above to over

Question on StackOverflow

Your Environment

• Reactor version(s) used: reactor-core:3.4.16, reactor-netty-core:1.0.17
• JVM version (java -version): Amazon Coretto 11
• OS and version (eg. uname -a): Amazon Linux 2

[violetagg]

@HannoSchreiber1652 What is the meaning of the colours?

[schreibaer]

The colours indicate various services to which connections are being opened. Due to sensitivity concerns, I can unfortunately not include the names of the services, however no irregularities can be seen on the client side.

[violetagg]

@HannoSchreiber1652 OK Thanks

Is the protocol HTTP/1.1 or it is HTTP/2?

[schreibaer]

We don't seem to be explicitly setting the protocol to 2, so I assume we are using the default which is probably still HTTP/1.1. I will keep looking to confirm though

[violetagg]

We don't seem to be explicitly setting the protocol to 2, so I assume we are using the default which is probably still HTTP/1.1. I will keep looking to confirm though

yes by default it is HTTP/1.1

[violetagg]

@HannoSchreiber1652 Closing this in favour of reactor/reactor-pool#123

[NeilOnasch0402]

@violetagg @HannoSchreiber1652
I'm a colleague of Hanno.

We do not use reactor-pool as a dependency (not even transitive, as far as I can see, since the reactor-pool dep is in the optional "addons" section of the Reactor project), so we don't understand how these Bugs are connected.

Also, if reactor-pool should be somehow included within reactor-core or reactor-netty, we can confirm that the commit did not fix the issue described above.

[violetagg]

@NeilOnasch0402 Reactor Netty packages Reactor Pool and distributes it with its binaries. You can see that in reactor-netty-core you have reactor.netty.internal.shaded.reactor.pool package.

[NeilOnasch0402]

Ah yes, I did see the internal.shaded package but assumed it's something different. Thanks for the clarification. However, to quote myself

we can confirm that the commit did not fix the issue described above.

Since this is a reactor-pool issue, this one can stay closed the discussion will continue in reactor/reactor-pool#123