CassandraSinkCluster: connection failure handling

[rukai]

Goal

Currently if we fail to create a connection to a destination node we just terminate the connection to the client.
This PR instead handles failure to connect by:

1. attempting to connect to all other possible valid nodes
2. failing that, returns an error message to the client and keeps the connection to the client open

Implementation

To achieve this the logic from CassandraSinkCluster::create_control_connection is moved into node_pool::get_accessible_node and reused for every connection type not just control connections.
Everywhere a connection is requested then has to handle failure by calling send_error_in_response_to_message/send_error_in_response_to_metadata when a failure to create a connection occurs.

I converted get_round_robin_node_in_dc_rack into a get_random_node_in_dc_rack.
A round robin solution would be a little faster then a randomizing solution as its cheaper to increment an index than to shuffle a list.
However they should both give equivalent load distributions.
I made this sacrifice as I know that we will be replacing it with a "power of two random choices" implementation eventually and implementing a round robin that was compatible with get_accessible_node would be possible but difficult to implement and prove correct.

It would be nicer to separate retry concerns out of NodePool.
However I know that when we implement "power of two random choices" routing we will want to be able to skip a random choice that cannot form a connection.
As such I have put the connection testing logic within the routing functions, this will allow our future "power of two random choices" implementation to take advantage of that.

[conorbros]

I'd usually say be wary of putting too much retry handling into Shotover and instead rely on the drivers to handle retry logic I think this change is simple enough and makes sense.

[rukai]

Your right in that it makes sense to leave it to the client when we can because it reduces our code complexity.

In this case we need to at least:

• issue a node.report_issue() to the dead node otherwise the client might hit the same dead node again
• report an error back to the user

And since we got this far we may as well retry another node for the client.
If we can completely hide that a node went down to the user that sounds like a win to me.
I guess a fair chunk of the complexity comes from this final bit, so if we needed to cut down on complexity it could be removed.