-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
opt(txn commits): Optimize txns by passing Skiplists to Badger (#7777)
In this PR, we use concurrent mutations system, introduced in #7694 , to generate Skiplists directly from a transaction, even before it is committed. When a transaction does get committed, we replace the key timestamps with the commit timestamp, in-place within the Skiplist. And in case of multiple transactions being committed, merge them together into one bigger Skiplist during the serial commit. This Skiplist is then handed over to Badger, bypassing its value log and WAL system. Instead, when Badger persists the skiplist into L0 tables, Dgraph gets a callback, which it uses to decide when it is safe to take a Raft snapshot / checkpoint. So, we no longer need Badger's WAL. Furthermore, live loader is also optimized to better deal with the conflicting transactions, and retrying them as frequently as possible, instead of only towards the end. The updated output from live loader shows we're able to deal with stragglers much better, with them causing almost no delays with this PR. There's also a bug fix in this PR. The Raft checkpointing code was not getting run as often as it should. This PR fixes that. ``` Master: [18:48:34-0700] Elapsed: 08m45s Txns: 21222 N-Quads: 21,221,870 N-Quads/s: 9,033 Aborts: 0 Number of TXs run : 21240 Number of N-Quads processed : 21239870 Time spent : 8m49.039284385s N-Quads processed per second : 40150 This PR: Number of TXs run : 21240 Number of N-Quads processed : 21239870 Time spent : 6m56.641399434s N-Quads processed per second : 51057 ``` Notable Changes: * Have each mutation create its own skiplist. Merge them during commit. * Use skiplist Builder * optimize mutations as well * Optimize live loader to better deal with conflicted requests. * Make a callback delete txns from Oracle. * Bug fix: Calculate raft checkpoint frequently. * Clarify snapshot rules. * Incremental rollups use skiplists too. * Stop rollups during drop operations. Co-authored-by: Ahsan Barkati <ahsanbarkati@gmail.com>
- Loading branch information
1 parent
60bec16
commit 1017c4e
Showing
22 changed files
with
591 additions
and
241 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.