Split lists in bulk loader #4875
Conversation
martinmr
force-pushed
the
martinmr/bulk-loader-splits
branch
from
5116631
to
2db6baa
Compare
martinmr
requested review from
danielmai,
MichaelJCompton and
pawanrawal
as code owners
There was a problem hiding this comment.
You could have a stream per split key per predicate. So, each pred has its own stream id. You need another stream id per such stream id to represent splits. You could achieve that by using the MSB bit to represent split stream for that predicate.
Reviewable status: 0 of 2 files reviewed, 1 unresolved discussion (waiting on @danielmai, @manishrjain, @martinmr, @MichaelJCompton, and @pawanrawal)
martinmr
force-pushed
the
martinmr/bulk-loader-splits
branch
from
51c2d8f
to
1426660
Compare
There was a problem hiding this comment.
Done using a diff stream id for the split keys.
Reviewable status: 0 of 2 files reviewed, 1 unresolved discussion (waiting on @danielmai, @golangcibot, @MichaelJCompton, and @pawanrawal)
dgraph/cmd/bulk/reduce.go, line 123 at r1 (raw file):
Previously, golangcibot (Bot from GolangCI) wrote…
Error return value of
wb.SetEntryis not checked (fromerrcheck)
Done.
There was a problem hiding this comment.
Do ensure that bulk loader runs and outputs a bunch of split keys, before merging.
@balajijinnah Can you look at this carefully? Defer to you for approval.
Reviewable status: 0 of 2 files reviewed, 1 unresolved discussion (waiting on @danielmai, @golangcibot, @MichaelJCompton, and @pawanrawal)
|
Verified keys show up in the debug tool. |
The equivalent of PR #4875 for the 1.2 branch. The original change cannot be cherry-picked because there are other changes to the bulk loader that are not included in this branch.
The equivalent of PR #4875 for the 1.2 branch. The original change cannot be cherry-picked because there are other changes to the bulk loader that are not included in this branch.
This change is
Docs Preview: