Memberlist ignore old tombstones #4420
Conversation
There was a problem hiding this comment.
I'm not as knowledgeable as @pstibrany in memberlist ring propagation but this change as well as #4419 are consistent with my understanding of how this works.
Additionally, I can confirm that deploying these changes to an environment with "unforgettable" ring members prevents the propagation of messages that were keeping these members around.
|
@joe-elliott these two PRs wouldn't be possible without your huge help with troubleshooting this and insights about (I still think there may be another bug lurking as I cannot yet explain why did nodes keep gossiping about those old tombstones, and why didn't such messages die during |
…d will not gossip about them. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
pstibrany
force-pushed
the
memberlist-ignore-old-tombstones
branch
from
e908472 to
6573606
Compare
| @@ -21,6 +21,7 @@ | |||
| * [CHANGE] Some files and directories created by Mimir components on local disk now have stricter permissions, and are only readable by owner, but not group or others. #4394 | |||
| * [CHANGE] Compactor: compactor will no longer try to compact blocks that are already marked for deletion. Previously compactor would consider blocks marked for deletion within `-compactor.deletion-delay / 2` period as eligible for compaction. #4328 | |||
| * [CHANGE] Memberlist: forward only changes, not entire original message. #4419 | |||
| * [CHANGE] Memberlist: don't accept old tombstones as incoming change, and don't forward such messages to other gossip members. #4420 | |||
There was a problem hiding this comment.
Why is this listed as a breaking change?
There was a problem hiding this comment.
Good shout. Should be bugfix as well.
* Memberlist KV will no longer consider old tombstones as a "change" and will not gossip about them. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> * CHANGELOG.md Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com> Signed-off-by: Alvin Lin <alvinlin@amazon.com>
What this PR does: This PR modifies memberlist KV store to avoid "accepting" old tombstones as a change, and avoid forwarding them to other gossip members. Before this PR, if there was a message with old tombstone in a gossip network, such message would basically never "die".
This PR builds on top of #4419, and will be rebased once #4419 is merged. Update: Rebased.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]