kvserver: TestDecommission times out

[jordanlewis]

See run here: https://teamcity.cockroachdb.com/viewLog.html?buildId=1989798&buildTypeId=Cockroach_UnitTests

W200605 02:52:13.117255 344814 kv/kvserver/raft_transport.go:637  [n5] while processing outgoing Raft queue to node 1: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.117286 349115 kv/kvserver/raft_transport.go:637  [n5] while processing outgoing Raft queue to node 3: rpc error: code = Canceled desc = grpc: the client connection is closing:
I200605 02:52:13.117343 354891 util/stop/stopper.go:539  quiescing
I200605 02:52:13.117742 354892 util/stop/stopper.go:539  quiescing
W200605 02:52:13.117997 345225 kv/kvserver/raft_transport.go:637  [n5] while processing outgoing Raft queue to node 2: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.118270 344774 kv/kvserver/raft_transport.go:637  [n5] while processing outgoing Raft queue to node 1: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.119105 344917 kv/kvserver/raft_transport.go:637  [n1] while processing outgoing Raft queue to node 5: EOF:
W200605 02:52:13.119303 349264 kv/kvserver/raft_transport.go:637  [n3] while processing outgoing Raft queue to node 5: EOF:
W200605 02:52:13.119378 348193 kv/kvserver/raft_transport.go:637  [n4] while processing outgoing Raft queue to node 5: EOF:
W200605 02:52:13.119380 343896 kv/kvserver/raft_transport.go:637  [n3] while processing outgoing Raft queue to node 1: rpc error: code = Canceled desc = grpc: the client connection is closing:
I200605 02:52:13.117547 354893 util/stop/stopper.go:539  quiescing
W200605 02:52:13.119429 349171 kv/kvserver/raft_transport.go:637  [n3] while processing outgoing Raft queue to node 5: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.119509 353254 kv/kvserver/raft_transport.go:637  [n3] while processing outgoing Raft queue to node 2: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.119565 343822 kv/kvserver/raft_transport.go:637  [n3] while processing outgoing Raft queue to node 1: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.119657 344355 kv/kvserver/raft_transport.go:637  [n3] while processing outgoing Raft queue to node 4: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.119760 353207 kv/kvserver/raft_transport.go:637  [n3] while processing outgoing Raft queue to node 2: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.120032 352088 kv/kvserver/raft_transport.go:637  [n2] while processing outgoing Raft queue to node 5: EOF:
W200605 02:52:13.120108 347679 kv/kvserver/raft_transport.go:637  [n4] while processing outgoing Raft queue to node 5: EOF:
W200605 02:52:13.120444 347780 kv/kvserver/raft_transport.go:637  [n5] while processing outgoing Raft queue to node 4: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.120863 345421 kv/kvserver/raft_transport.go:637  [n5] while processing outgoing Raft queue to node 2: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.121142 344788 kv/kvserver/raft_transport.go:637  [n1] while processing outgoing Raft queue to node 5: EOF:
W200605 02:52:13.123743 344293 gossip/gossip.go:1525  [n5] no incoming or outgoing connections
W200605 02:52:13.124421 343706 gossip/gossip.go:1525  [n4] no incoming or outgoing connections
W200605 02:52:13.124741 343739 kv/kvserver/raft_transport.go:637  [n1] while processing outgoing Raft queue to node 3: EOF:
W200605 02:52:13.124772 344109 kv/kvserver/raft_transport.go:637  [n4] while processing outgoing Raft queue to node 3: EOF:
W200605 02:52:13.124811 344186 kv/kvserver/raft_transport.go:637  [n1] while processing outgoing Raft queue to node 3: EOF:
W200605 02:52:13.124861 353127 kv/kvserver/raft_transport.go:637  [n2] while processing outgoing Raft queue to node 3: EOF:
W200605 02:52:13.124870 344870 kv/kvserver/raft_transport.go:637  [n4] while processing outgoing Raft queue to node 3: EOF:
W200605 02:52:13.124886 353202 kv/kvserver/raft_transport.go:637  [n2] while processing outgoing Raft queue to node 3: EOF:
W200605 02:52:13.125196 344840 kv/kvserver/raft_transport.go:637  [n3] while processing outgoing Raft queue to node 4: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.117389 349597 kv/kvserver/raft_transport.go:637  [n5] while processing outgoing Raft queue to node 3: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.125799 343842 kv/kvserver/raft_transport.go:637  [n2] while processing outgoing Raft queue to node 1: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.125895 347633 kv/kvserver/raft_transport.go:637  [n2] while processing outgoing Raft queue to node 4: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.125941 347621 kv/kvserver/raft_transport.go:637  [n2] while processing outgoing Raft queue to node 4: rpc error: code = Canceled desc = grpc: the client connection is closing:
W200605 02:52:13.126683 343633 kv/kvserver/raft_transport.go:637  [n1] while processing outgoing Raft queue to node 2: EOF:
W200605 02:52:13.126735 344185 kv/kvserver/raft_transport.go:637  [n1] while processing outgoing Raft queue to node 2: EOF:
W200605 02:52:13.126777 347504 kv/kvserver/raft_transport.go:637  [n4] while processing outgoing Raft queue to node 2: EOF:
W200605 02:52:13.126823 348258 kv/kvserver/raft_transport.go:637  [n4] while processing outgoing Raft queue to node 2: EOF:
W200605 02:52:13.127491 343800 kv/kvserver/raft_transport.go:637  [n2] while processing outgoing Raft queue to node 1: EOF:
W200605 02:52:13.127546 344236 kv/kvserver/raft_transport.go:637  [n4] while processing outgoing Raft queue to node 1: EOF:
W200605 02:52:13.127595 344253 kv/kvserver/raft_transport.go:637  [n4] while processing outgoing Raft queue to node 1: EOF:
W200605 02:52:13.127657 345227 kv/kvserver/raft_transport.go:637  [n2] while processing outgoing Raft queue to node 5: EOF:
W200605 02:52:13.127935 343090 gossip/gossip.go:1525  [n2] no incoming or outgoing connections
W200605 02:52:13.128275 344095 kv/kvserver/raft_transport.go:637  [n1] while processing outgoing Raft queue to node 4: EOF:
W200605 02:52:13.128390 344918 kv/kvserver/raft_transport.go:637  [n1] while processing outgoing Raft queue to node 4: EOF:
W200605 02:52:13.128475 347759 kv/kvserver/raft_transport.go:637  [n5] while processing outgoing Raft queue to node 4: EOF:
--- FAIL: TestDecommission (52.15s)
    client_raft_test.go:3033: condition failed to evaluate within 45s: still a replica on s1: r35:/{Table/Max-Max} [(n1,s1):1, (n2,s2):2, (n4,s4):5, next=6, gen=15, sticky=9223372036.854775807,2147483647]
        goroutine 342000 [running]:
        runtime/debug.Stack(0xc00804fa20, 0x4dfa520, 0xc00898afa0)
        	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x4f23ca0, 0xc000b27400, 0xc00804fa20)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:36 +0x6b
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestDecommission.func1(0x1, 0x3)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_raft_test.go:3033 +0xa2
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestDecommission(0xc000b27400)
        	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_raft_test.go:3056 +0x428
        testing.tRunner(0xc000b27400, 0x4657398)
        	/usr/local/go/src/testing/testing.go:909 +0xc9
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:960 +0x350