load balancing: slow coordinator avoidance

[kbr-]

It would be nice to have a real load balancing strategy which chooses coordinators based on actual metrics such as how fast it can process requests.

See scylladb/scylladb#5715 for motivation. I explain in the comments what the issue is (it is a driver-side issue).

The 4.x versions of the Java driver AFAIK do have such strategies - at least one.
From https://docs.datastax.com/en/developer/java-driver/4.11/manual/core/load_balancing/#built-in-policies:

DefaultLoadBalancingPolicy should almost always be used; it requires a local datacenter to be specified either programmatically when creating the session, or via the configuration (see below). It can also use a highly efficient slow replica avoidance mechanism, which is by default enabled.

[martin-sucha]

FYI, we have something like this in our fork of gocql https://github.com/kiwicom/gocql/blob/master/tokenawarelatency.go (but it is a bit different to the java policy linked above).

[sylwiaszunejko]

Done in #168