Better handling extended protocol messages in the event of busy pool

[drdrsh]

Let's imagine a Pgcat instance with single pool containing a single instance. The pool is fully occupied and all available connections are fully booked.

A client comes in, it connects and is now sitting idle in the external loop https://github.com/levkk/pgcat/blob/main/src/client.rs#L573

The client attempts to perform an extended protocol query. So first thing it does is send a P message. We try to grab a connection from the pool here but we fail. This failure sends us to this branch that ends with continue. The P message never got buffered because we buffer message in the inner loop here. Subsequent extended protocol messages meet similar fate and now the client sends a sync S message, This one manages to break through, so we are able to grab a connection and we enter the inner loop (The transaction loop) but we arrived here without any of the previous messages being buffered so we end up with errors like ERROR: unnamed prepared statement does not exist or ERROR: portal "" does not exist. This does not affect clients that are already in the inner loop and have a server connection allocated.

This change allows us to buffer the extended protocol message early without even checking out a connection from the pool. The actual checkout will be performed when the S message arrives, at this point, we are guaranteed to have all messages buffered and have a client on the other end expecting a response. We can respond with SystemError: Failed to get a connection from the Pool or execute the query.

[levkk]

Cool!