storage: allow quiescing in the presence of dead nodes

[petermattis]

Per a TODO comment:

// TODO(peter): When a node goes down, any range which has a replica on the
// down node will not quiesce. This could be a significant performance
// impact. Additionally, when the node comes back up we want to bring any
// replicas it contains back up to date. Right now this will be handled because
// those ranges never quiesce. One thought for handling both these scenarios is
// to hook into the StorePool and its notion of "down" nodes. But that might
// not be sensitive enough.

Cc @cockroachdb/stability

[bdarnell]

The StorePool currently takes 5 minutes to declare a node "down". We could lower this, although making it too low would result in more frequent repair operations for temporary blips. Maybe the StorePool should recognize different levels of "down":

1. After any disruption at all, remove the node from the pool of rebalance targets
2. After some fairly short time, allow ranges on this node to quiesce even though it isn't responding
3. After enough time has passed that we're sure the node isn't coming back soon, schedule its ranges for repair.

If the downed node comes back while in state 2 (and we're assuming this will be relatively common; otherwise we should go straight to state 3), it will try to start elections, which will be disruptive.

No matter how quickly we declare a node down, though, there will be some time during which we are unable to quiesce and heartbeats will be flowing at full volume. The risk of these load spikes remains a major concern with quiescence as an alternative to coalescing heartbeats.

[petermattis]

Yeah, using StorePool for this purpose feels awkward.

When a node goes down, all traffic to that node should start failing fast. It feels like there is something we can notice there, perhaps at the RaftTransport level. Something I haven't worked out is how to determine when the node has come back up in the absence of other traffic to the node.

[bdarnell]

Networks can fail in many mysterious ways; not all of them result in something that will promptly trigger a fail-fast in the raft transport.

If the node was down for at least the StoreGossipInterval, it will tell you via gossip when it comes back up. If it's down for a shorter period of time, it will tell you that it's back up by calling raft elections, although that's disruptive and we want to avoid that outcome. We could have it open all its raft streams to all peer stores at startup instead of waiting to do them lazily; we could then un-quiesce in RaftMessage before we go into the Recv loop.

[spencerkimball]

Raft transport employs its own circuit breaker for failing fast. We could
export a method to check that for whether or not to exclude a node from the
healthy set in order to quiesce. The problem is that the signal only works
if it's periodically primed by trying to send traffic to the node. But
circuit breakers do have an events callback mechanism, so it would be
pretty simple to give the raft transport the ability to accept
registrations and invoke a callback when it receives a BreakerReady event.

On Sun, Sep 18, 2016 at 12:08 PM Ben Darnell notifications@github.com
wrote:

Networks can fail in many mysterious ways; not all of them result in
something that will promptly trigger a fail-fast in the raft transport.

If the node was down for at least the StoreGossipInterval, it will tell
you via gossip when it comes back up. If it's down for a shorter period of
time, it will tell you that it's back up by calling raft elections,
although that's disruptive and we want to avoid that outcome. We could have
it open all its raft streams to all peer stores at startup instead of
waiting to do them lazily; we could then un-quiesce in RaftMessage before
we go into the Recv loop.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#9446 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF3MTZznAZFDATtOpI8Ywln7swdlUqw5ks5qrWHggaJpZM4J_q49
.

[petermattis]

True about fail-fast not being foolproof. If a node is partitioned away, we should stop receiving any traffic from it. Feels like there must be some reliable signal there.

Lazy Raft loading prevents a node from calling Raft elections when it comes back up.

[bdarnell]

Lazy Raft loading prevents a node from calling Raft elections when it comes back up.

If the process restarted. If it was a network-level problem, the process will still be there with its unquiesced raft groups calling for elections.