dispatcher gRPC balancer doesn't account for pod IP changes

[jgraettinger]

Pod IP changes can happen on pod restart (I see this occasionally with MicroK8s when my laptop sleeps, but have never seen it on a cloud provider).

Here's an example network trace from the primary broker's /debug/requests:

2019/10/03 01:30:40.843305 | 0.000039 | /protocol.Journal/Replicate
-- | -- | --
01:30:40.843308 | .     3 | ... RPC: to <nil> deadline:none
01:30:40.843320 | .    11 | ... Pick(Route: members:<zone:"zone-A" suffix:"gazette-77b6856ccf-lw7ml" > members:<zone:"zone-A" suffix:"gazette-77b6856ccf-vbtr7" > members:<zone:"zone-B" suffix:"gazette-77b6856ccf-nnc8b" > primary:1 endpoints:"http://10.1.1.58:8080" endpoints:"http://10.1.1.48:8080" endpoints:"http://10.1.1.55:8080" , ID: zone:"zone-A" suffix:"gazette-77b6856ccf-lw7ml" ) => zone:"zone-A" suffix:"gazette-77b6856ccf-lw7ml" (TRANSIENT_FAILURE)
01:30:40.843343 | .    23 | ... RPC: [rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 10.1.1.17:8080: connect: no route to host"]

The Pick() route shows the correct endpoint for gazette-77b6856ccf-lw7ml of http://10.1.1.58:8080, however the balancer is still holding on to a SubConn which is using it's former address of 10.1.1.17:8080. I don't believe the balancer considers that IPs of ProcessSpec_ID's can change (it should).