Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial commit for multi-part lists #3105

Merged
merged 82 commits into from
Apr 19, 2019
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
e8b96e6
Initial commit for multi-part lists
martinmr Feb 27, 2019
8aff4a7
Decide if list needs to be split based on approxLen.
martinmr Mar 6, 2019
dd630c4
Use size instead of length to do split decisions.
martinmr Mar 6, 2019
f6125ba
Several changes.
martinmr Mar 7, 2019
b572a39
Increase test size and remove print statements
martinmr Mar 7, 2019
e9d3f05
Combine partial and complete iterations into the same function.
martinmr Mar 7, 2019
9f8d54f
Remove unused function and optimize rollup.
martinmr Mar 7, 2019
4dc9397
Add extra test.
martinmr Mar 7, 2019
fc4c183
Add extra tests
martinmr Mar 7, 2019
75db0f5
Remove first_part field from proto.
martinmr Mar 7, 2019
e29a15d
Fix bug in Uids function.
martinmr Mar 7, 2019
ebd8303
Removed unused field
martinmr Mar 7, 2019
0250636
Simplify rollup logic.
martinmr Mar 7, 2019
b0eddc4
Change order of if-statements.
martinmr Mar 7, 2019
aad3f20
Remove unused field.
martinmr Mar 7, 2019
6029630
Load list parts on demand. Also simpler split logic.
martinmr Mar 7, 2019
78d35fb
Remove partialIteration field.
martinmr Mar 8, 2019
6847378
Remove unused proto fields.
martinmr Mar 8, 2019
29df604
Remove unnecessary parameter from method.
martinmr Mar 8, 2019
d1d6bce
Variable rename
martinmr Mar 8, 2019
a9eca0f
Clean map of uncommitted lists.
martinmr Mar 8, 2019
7670457
Move logic to clean uncommitted list map at the beginning of rollup.
martinmr Mar 8, 2019
8cef9e0
Loop until finding a part with valid entries.
martinmr Mar 8, 2019
a709f8e
Simplify rollup logic.
martinmr Mar 8, 2019
e3de2e1
Rename proto field and simplify split logic.
martinmr Mar 8, 2019
74b3889
Change tests to read and write posting lists to disk.
martinmr Mar 11, 2019
4c14401
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Mar 11, 2019
db750d6
remove readListPartFromDisk method.
martinmr Mar 11, 2019
b21038c
Check error
martinmr Mar 11, 2019
86c5951
Store splits in emptylists and fix logic to decide the list metadata.
martinmr Mar 12, 2019
d6c5dfb
Check error status of init.
martinmr Mar 12, 2019
f05f3ed
Fix marshalPostingList
martinmr Mar 12, 2019
240e898
Add flag to control max posting list size.
martinmr Mar 12, 2019
ccb820f
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Mar 12, 2019
08e7601
Parse startuid in keys.go and add tests.
martinmr Mar 12, 2019
f328c18
Update debug tool to understand multi-part lists.
martinmr Mar 12, 2019
3a52162
Restore comment.
martinmr Mar 13, 2019
e0d13ac
Fix posting tests
martinmr Mar 13, 2019
9236018
Add test files.
martinmr Mar 13, 2019
becab2d
Delete unused testing code.
martinmr Mar 13, 2019
294b8d4
Remove max_posting_list_size flag
martinmr Mar 13, 2019
38f7e7e
Add extra tests for multi-posting lists.
martinmr Mar 13, 2019
af6d2ac
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Mar 14, 2019
92c8a44
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Mar 19, 2019
dd9b968
Manish's review. Renamed some funcs, readPartList does not use cache,…
manishrjain Mar 21, 2019
b114232
Fix keys.go tests
martinmr Mar 21, 2019
d907e51
Rename deleteBelow and add sanity check for it.
martinmr Mar 21, 2019
2033a41
Move iteration logic to Next()
martinmr Mar 21, 2019
a8bb694
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Mar 21, 2019
b33444b
Add documentation for pickPostings
martinmr Mar 21, 2019
7dc357e
Make rollup read-only, remove cache.
martinmr Mar 21, 2019
92f8724
Remove err from plsAreEmpty
martinmr Mar 21, 2019
3dc54ad
Fix marshalling and deletion logic.
martinmr Mar 21, 2019
9cf7130
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Mar 21, 2019
65538a0
Remove start_uid field from proto.
martinmr Mar 21, 2019
b30d1ed
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Apr 2, 2019
58704c4
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Apr 3, 2019
e197a27
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Apr 3, 2019
ee37481
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Apr 9, 2019
184fc54
Some small nitpick changes.
manishrjain Apr 11, 2019
dfdf290
Remove unnecessary variables to simplify code reading. Also rename so…
manishrjain Apr 11, 2019
f4244c0
Addresssed review comments.
martinmr Apr 11, 2019
514e5e5
Remove unnecessary check.
martinmr Apr 12, 2019
0f13be3
Manish's review. Remove delete splits var in output. Rename some fiel…
manishrjain Apr 12, 2019
2165095
Fix marshal test.
martinmr Apr 12, 2019
f7debac
Do not use out.plist.Splits for iteration.
martinmr Apr 12, 2019
4c1ded9
Sort splits after removing empty list parts.
martinmr Apr 12, 2019
ba74bce
First split should start at 1 instead of zero.
martinmr Apr 12, 2019
b5b6a7f
Remove HasStartUid field from ParsedKey.
martinmr Apr 12, 2019
fb04677
Remove completed todo.
martinmr Apr 12, 2019
30c1799
First split should not be removed unless entire list is empty.
martinmr Apr 12, 2019
c13b4cb
Add comment to Rollup
martinmr Apr 13, 2019
8a18e67
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Apr 13, 2019
99cc5cb
Fix debug tool.
martinmr Apr 13, 2019
2adc138
go fmt.
martinmr Apr 15, 2019
d818bbe
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Apr 15, 2019
bca3c05
Copied comment.
martinmr Apr 15, 2019
4fb78a6
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Apr 18, 2019
51786ad
Fix comment length.
martinmr Apr 18, 2019
ad39e3b
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Apr 18, 2019
81f58f0
Merge remote-tracking branch 'origin/master' into martinmr/multi-part…
martinmr Apr 19, 2019
76e1a54
Remove extra file
martinmr Apr 19, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions posting/index.go
Original file line number Diff line number Diff line change
Expand Up @@ -550,15 +550,18 @@ func (r *rebuild) Run(ctx context.Context) error {
if le == 0 {
continue
}
kv, err := pl.MarshalToKv()
kvs, err := pl.MarshalToKv()
if err != nil {
return err
}
// We choose to write the PL at r.startTs, so it won't be read by txns,
// which occurred before this schema mutation. Typically, we use
// kv.Version as the timestamp.
if err = writer.SetAt(kv.Key, kv.Value, kv.UserMeta[0], r.startTs); err != nil {
return err

for _, kv := range kvs {
// We choose to write the PL at r.startTs, so it won't be read by txns,
// which occurred before this schema mutation. Typically, we use
// kv.Version as the timestamp.
if err = writer.SetAt(kv.Key, kv.Value, kv.UserMeta[0], r.startTs); err != nil {
return err
}
}
// This locking is just to catch any future issues. We shouldn't need
// to release this lock, because each posting list must only be accessed
Expand Down
Loading