Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SG handling for mutations deduplicated by Couchbase feed #775

Closed
adamcfraser opened this issue Apr 8, 2015 · 1 comment
Closed

SG handling for mutations deduplicated by Couchbase feed #775

adamcfraser opened this issue Apr 8, 2015 · 1 comment
Assignees
Milestone

Comments

@adamcfraser
Copy link
Collaborator

Both TAP and DCP include the ability to deduplicate mutations by DocID. The result is that Sync Gateway may not see a sequence for an interim mutation.

This won't result in any data loss from Sync Gateway's perspective, but it can result in delays in the Sync Gateway _changes feed, while it waits for the sequence from the deduped mutation.

In SG 1.0.4 and earlier, rapid mutations to a single doc have the potential for introducing significant latency into the _changes feed. Post-1.0.4 this is mitigated by the the fix for #525, which makes missing sequences non-blocking.

However, there is still processing and performance overhead in Sync Gateway for the deduped sequences - they keep the 'low' sequence number unnecessarily low until they are eventually abandoned, and there's overhead when we eventually prune the sequence.

A possible immediate-term approach would be to store the sequence number history in the _sync/history along with the revision number history, and use that to identify any sequences that didn't appear on the TAP feed. This increases the size of the _sync metadata, but is relatively small compared to the other revision-level metadata we already store.

Longer term, this would overlap with the discussion about using vbucket-striped sequences for change tracking.

@adamcfraser
Copy link
Collaborator Author

Potentially "enhance" Walrus to reproduce this issue for unit test purposes.

@zgramana zgramana added this to the 1.1.0 milestone Apr 17, 2015
@adamcfraser adamcfraser added review and removed ready labels Apr 23, 2015
tleyden added a commit that referenced this issue Apr 27, 2015
Issue #775 - handle sequences deduplicated by server feed
tleyden pushed a commit that referenced this issue Apr 28, 2015
The tap de-dupe will be useful for creating an enhanced unit test for Issue #775.  See #798
tleyden pushed a commit that referenced this issue Apr 28, 2015
The tap de-dupe will be useful for creating an enhanced unit test for Issue #775.  See #798
@adamcfraser adamcfraser removed the review label May 1, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants