Skip to content

Commit

Permalink
Don't send meta snapshot when becoming metaleader
Browse files Browse the repository at this point in the history
Antithesis testing has found that late or out-of-order delivery of these
snapshots, likely due to latency or thread pauses, can cause stream
assignments to be reverted which results in assets being deleted and
recreated. There may also be a race condition where the metalayer comes
up before network connectivity is fully established so we may end up
generating snapshots that don't include assets we don't know about yet.

We will want to audit all uses of `SendSnapshot` as it somewhat breaks
the consistency model, especially now that we have fixed a significant
number of Raft bugs that `SendSnapshot` usage may have been papering over.

Further Antithesis runs without this code run fine and have eliminated
a number of unexpected calls to `processStreamRemoval`.

Signed-off-by: Neil Twigg <neil@nats.io>
  • Loading branch information
neilalexander committed Jul 30, 2024
1 parent aa37839 commit c20906a
Showing 1 changed file with 0 additions and 4 deletions.
4 changes: 0 additions & 4 deletions server/jetstream_cluster.go
Original file line number Diff line number Diff line change
Expand Up @@ -1391,10 +1391,6 @@ func (js *jetStream) monitorCluster() {
aq.recycle(&ces)

case isLeader = <-lch:
// For meta layer synchronize everyone to our state on becoming leader.
if isLeader && n.ApplyQ().len() == 0 {
n.SendSnapshot(js.metaSnapshot())
}
// Process the change.
js.processLeaderChange(isLeader)
if isLeader {
Expand Down

0 comments on commit c20906a

Please sign in to comment.