Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal Pagination: Adds Merge Proofing #2084

Merged
merged 11 commits into from
Jan 11, 2021

Conversation

bchong95
Copy link
Contributor

@bchong95 bchong95 commented Dec 18, 2020

Internal Pagination: Adds Merge Proofing

Description

This PR merge proofs the pagination library which covers the pull models for ReadFeed, ChangeFeed, and Query.

For ReadFeed and ChangeFeed, we don't do anything special to the continuation token on top of what the pagination library handles. These operations were merge proof just from epk range handling.

For Query, we only emit the state of the partition that we resumed from. This makes it difficult to resume the query on a merge, since we have no context on what partitions were there in the past. The algorithm is documented in PartitionMapper.cs, but I will run through an example here:

Suppose we have the following partitions: [(A, B), (B, C), (E, F), (H, I), (I, L)]
And the following continuation token: [{"range": {"min": "I", "max": "K"}, "token": "blah"}]
Then we can resume the cross partition state like so:

  1. Merge the ranges as much as possible:
[(A, B), (B, C), (E, F), (H, I), (I, L)] 
    => [(A, C), (E, F), (H, L)]
  1. Split the Ranges Based on the Continuation Token
[(A, C), (E, F), (H, L)] 
    => [(A, C), (E, F), (H, I), (I, K), (K, L)]
  1. Match the Range to the Continuation Token:
[(A, C), (E, F), (H, I), (I, K), (K, L)] 
    => [((A, C), null), ((E, F), null), ((H, I), null), ((I, K), blah), ((K, L), null)]
  1. Let the Pagination Library Resume with SplitProofing:
[((A, C), null), ((E, F), null), ((H, I), null), ((I, K), blah), ((K, L), null)] 
    => [((A, B), null), ((B, C), null), ((E, F), null), ((H, I), null), ((I, K), blah), ((K, L), null)]

Extensive testing has been added that tests the cross product of: {UseContinuationTokens, SplitPartitions, MergePartitions}. This should cover all the scenarios for splits, merges, and continuation tokens.

InMemoryContainer needed to be updated to support merge proofing the token it generates for the SDK.

The follow up work item is adding integration tests for splits and merge to validate that there is no soft contract violations with the NetworkAttachedDocumentContainer class.

sboshra
sboshra previously approved these changes Dec 23, 2020
@sboshra sboshra merged commit bd3f1f1 into master Jan 11, 2021
@sboshra sboshra deleted the users/brchon/Pagination/MergeProofTesting branch January 11, 2021 23:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants