Pending Connections exceeding Max Pending Limit

[daanishsarguru]

We have a Spring Boot project which uses reactor-netty for WebClient which is dependent on reactor-pool.
We're using WebClient to make API calls and facing an issue wherein Webclient is exceeding the max acquire queue size.
We have set the pendingAcquireMaxCount to 10000, however from the graphs, it can be seen that it has increased much beyond that

Expected Behavior

The extra requests should've failed with PoolAcquirePendingLimitException exception

Actual Behavior

Pending Requests keep on increasing without check, and none of the request actually gets processed, all of them keep timing out. I have tested this with a service we call by giving it a very large timeout (1 minute)

Steps to Reproduce

Unfortunately, I haven't been able to reproduce this on my local machine. I've only been able to witness this on the production environment randomly on some of the running instances

Possible Solution

I'm suspecting this has something to do with the following method

boolean pendingOffer(Borrower<POOLABLE> pending)

If somehow the PENDING_COUNT value were to exceed maxPending, this issue is set into motion

Environment

Reactor version(s) used:
reactor-netty: 0.9.16.RELEASE
reactor-pool: 0.1.8.RELEASE
Spring Boot: 2.2.10.RELEASE
JVM version: openjdk version "1.8.0_211-b12"
OS and version: alpine:3.6

[simonbasle]

@daanishsarguru I have an improvement in #122, but it still isn't 100% perfect. BUT it seems in your test you have unbounded growth, and with the fix I expect there will be an upper bound now, much closer to the configured maxPending. Would you be able to test that PR, building reactor-pool and reactor-netty locally and see how the test behaves?

[simonbasle]

I'll close this through #122 @daanishsarguru
Nonetheless, please report results of that change back here and ping me 🙏.

I've also created #123 as a follow-up to keep track of the fact that maxPending isn't currently 100% accurate, but is now capped within a margin of error rather than plain ignoring the limit.

[daanishsarguru]

Thanks @simonbasle 👌
will test this and report the results