Improve connection pool.

[mikkokar]

Description

Fixes: #333.

This is a complete rewrite of the Styx connection pool.

It addresses all known problems:

• Leaking connection
• Returned connections not given back fairly
• A borrow triggering a TCP connection establishment must wait for the connection establishment, instead of given a returned connection.

Pool Behaviour Overview

The improved pool has two queues:

1. activeConnections contains all TCP connections that are immediately available at the pool. Any connections in this pool are already connected and ready to serve traffic.
2. waitingSubscribers is a FIFO queue for storing, well, waiting subscribers (aka pending connections).

Borrow Connection

When a connection is borrowed, the pool checks if there are any immediately available connections in the activeConnections queue.

If yes:

• Remove the connection from activeConnection and emit via the Publisher interface.

If no:

• Trigger a new TCP connection establishment (subject to pool constraints), and queue the subscriber in the waitingSubscribers queue.

Return Connection

When a connection is returned, offer it to the longest waiting subscriber in the waitingSubscribers queue (assuming the connection is still alive). If nobody is waiting (the watingSubscribers queue is empty) then put the connection back to the activeConnections queue.

The pool disposes any connection that is returned as disconnected.

TCP Connection Establishment completed

Behaves much like connection return. The established connection is first offered to a waiting subscriber. The connection is queued in the activeConnections when there are no waiting subscribers.

TCP Connection Termination

Eagerly trigger a new connection establishment.

Subscriber Cancels

Remove a cancelled subscriber from the waitingSubscribers queue.