Benchmark and Optimize Raft #2975
Labels
area/performance
Performance related issues.
kind/maintenance
Maintenance tasks, such as refactoring, with no impact in features.
priority/P1
Serious issue that requires eventual attention (can wait a bit)
status/accepted
We accept to investigate/work on it.
Comments
manishrjain
added
optimization
priority/P1
|
One of the things might be that |
manishrjain
added a commit
that referenced
this issue
Feb 11, 2019
I've spent the last few days looking at how to optimize the live mutation path in Dgraph server. While trying many things in the server (past commits included), I realized my optimizations in the server are not improving things much, the throughput saturating at 20-30K NQuads/sec. Turns out, it was the live loader which was causing the saturation. The XID to UID assigner was the bottleneck causing the throughput to stagnate, despite the server being underutilized. This PR fixes that by optimizing the assigner. In particular, I've removed the slow LRU cache. Added buffer to `newRanges` channel to ensure we always have a range handy when we run out. Made passing badger DB instance optional, so we can avoid doing disk writes if not required. And made other optimizations around how we lock, etc. I also added benchmarks for the assigner, which shows each allocation (tested via parallel benchmark) takes 350 ns/op on my desktop. With these changes, the live loader throughput jumps to 100K-120K NQuads/sec on my desktop. In particular, pre-assigning UIDs to the RDF/JSON file yields maximum throughput. I can load 140M friend graph RDFs in 25 mins. Helps with #2975 . Changes: * Work on optimizing XidToUid map. * Add the test and benchmark for xid to uid map * Working code with decreased memory usage. Includes a new BumpUp API. * Working live loader, which can optionally just keep all the mapping in memory. * Adding shards back to XidMap speed up operations by a huge factor. Benchmark shows each allocation is 300ns. * Make BumpTo much faster by calling Zero directly, instead of looping through the newRanges channel. * Improve how BumpTo() happens by using a maxSeenUid variable.
campoy
added
area/performance
dna2github
pushed a commit
to dna2fork/dgraph
that referenced
this issue
Jul 19, 2019
I've spent the last few days looking at how to optimize the live mutation path in Dgraph server. While trying many things in the server (past commits included), I realized my optimizations in the server are not improving things much, the throughput saturating at 20-30K NQuads/sec. Turns out, it was the live loader which was causing the saturation. The XID to UID assigner was the bottleneck causing the throughput to stagnate, despite the server being underutilized. This PR fixes that by optimizing the assigner. In particular, I've removed the slow LRU cache. Added buffer to `newRanges` channel to ensure we always have a range handy when we run out. Made passing badger DB instance optional, so we can avoid doing disk writes if not required. And made other optimizations around how we lock, etc. I also added benchmarks for the assigner, which shows each allocation (tested via parallel benchmark) takes 350 ns/op on my desktop. With these changes, the live loader throughput jumps to 100K-120K NQuads/sec on my desktop. In particular, pre-assigning UIDs to the RDF/JSON file yields maximum throughput. I can load 140M friend graph RDFs in 25 mins. Helps with dgraph-io#2975 . Changes: * Work on optimizing XidToUid map. * Add the test and benchmark for xid to uid map * Working code with decreased memory usage. Includes a new BumpUp API. * Working live loader, which can optionally just keep all the mapping in memory. * Adding shards back to XidMap speed up operations by a huge factor. Benchmark shows each allocation is 300ns. * Make BumpTo much faster by calling Zero directly, instead of looping through the newRanges channel. * Improve how BumpTo() happens by using a maxSeenUid variable.
MichelDiz
added
the
kind/maintenance
|
Github issues have been deprecated.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm seeing that after proposing a mutation, it is taking over 130ms just to see it back in
CommitedEntries. We should try to figure out why it takes this long to see the mutation back via Raft. What is the slowdown here?SaveToStorage, the comms, or something else?The text was updated successfully, but these errors were encountered: