[FIXED] Ghost consumers during meta recovery #6092
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MauriceVanVeen
force-pushed
the
maurice/fix-ghost-consumers
branch
from
6faedb6 to
99cf6c8
Compare
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
…ude in snapshot Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
…d don't include in snapshot Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
MauriceVanVeen
force-pushed
the
maurice/fix-ghost-consumers
branch
from
f0b9900 to
857ada4
Compare
neilalexander
pushed a commit
that referenced
this pull request
Nov 12, 2024
During meta recovery `ru.updateConsumers` and `ru.removeConsumers` would not be properly cleared since the move from `map[string]*consumerAssignment` to `map[string]map[string]*consumerAssignment`. Which meant that consumers that needed to be removed were both in `ru.removeConsumers` and left in `ru.updateConsumers`. Resulting in a ghost consumer. Also don't clear recovering state while we still have items to process as part of recovery. De-flakes `TestJetStreamClusterLostConsumers`, and makes `TestJetStreamClusterConsumerLeak` more reliable by re-introducing the `ca.pending` flag. Since the consumer leader responds for consumer creation, but meta leader responds for consumer deletion, so need to have the consumer assignment available so meta leader can respond successfully. Signed-off-by: Maurice van Veen <github@mauricevanveen.com> --------- Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
neilalexander
pushed a commit
that referenced
this pull request
Nov 12, 2024
During meta recovery `ru.updateConsumers` and `ru.removeConsumers` would not be properly cleared since the move from `map[string]*consumerAssignment` to `map[string]map[string]*consumerAssignment`. Which meant that consumers that needed to be removed were both in `ru.removeConsumers` and left in `ru.updateConsumers`. Resulting in a ghost consumer. Also don't clear recovering state while we still have items to process as part of recovery. De-flakes `TestJetStreamClusterLostConsumers`, and makes `TestJetStreamClusterConsumerLeak` more reliable by re-introducing the `ca.pending` flag. Since the consumer leader responds for consumer creation, but meta leader responds for consumer deletion, so need to have the consumer assignment available so meta leader can respond successfully. Signed-off-by: Maurice van Veen <github@mauricevanveen.com> --------- Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
neilalexander
added a commit
that referenced
this pull request
Nov 12, 2024
Includes the following: * Some tweaks to the NRG test helpers * #6055 * #6061 * #6065 * #6041 (but with `math/rand` instead of `math/rand/v2` due to an older Go version in CI for 2.10.x) * #6066 * #6067 * #6069 * #6075 * #6082 * #6087 * #6086 * #6088 * #6089 * #6092 * #6096 * #6098 * #6097 * #6105 * #6104 * #6106 * #6109 * #6111 * #6112 Signed-off-by: Neil Twigg <neil@nats.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
During meta recovery
ru.updateConsumersandru.removeConsumerswould not be properly cleared since the move frommap[string]*consumerAssignmenttomap[string]map[string]*consumerAssignment. Which meant that consumers that needed to be removed were both inru.removeConsumersand left inru.updateConsumers. Resulting in a ghost consumer.Also don't clear recovering state while we still have items to process as part of recovery.
De-flakes
TestJetStreamClusterLostConsumers, and makesTestJetStreamClusterConsumerLeakmore reliable by re-introducing theca.pendingflag. Since the consumer leader responds for consumer creation, but meta leader responds for consumer deletion, so need to have the consumer assignment available so meta leader can respond successfully.Signed-off-by: Maurice van Veen github@mauricevanveen.com