feat(eds/store): cache proofs after first eds recompute on sampling #2429
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
walldiss
requested review from
renaynay,
Wondertan,
vgonkivs and
distractedm1nd
as code owners
walldiss
changed the title
|
Tests are broken because of rsmt2d bug. There is an open PR to fix the bug: celestiaorg/rsmt2d#236 |
|
Does this solution works with bridge? |
|
Needs support for bridges by passing nmt options through app's block extension code |
walldiss
changed the base branch from
storage_performance_improvements
to
v0.11.0-rc9
|
Added support for bridges and also made it to not pass proofs inside context when it is possible to. |
Wondertan
reviewed
Aug 3, 2023
Wondertan
reviewed
Aug 3, 2023
Codecov Report
@@ Coverage Diff @@
## release-11-rc9 #2429 +/- ##
==================================================
+ Coverage 51.75% 51.94% +0.18%
==================================================
Files 156 156
Lines 10010 10039 +29
==================================================
+ Hits 5181 5215 +34
+ Misses 4367 4366 -1
+ Partials 462 458 -4
|
Wondertan
approved these changes
Aug 4, 2023
walldiss
force-pushed
the
recompute_eds_once_on_sampling
branch
2 times, most recently
from
e16f534
to
df1a729
Compare
walldiss
force-pushed
the
recompute_eds_once_on_sampling
branch
from
df1a729
to
0dc1c2a
Compare
Wondertan
approved these changes
Aug 4, 2023
distractedm1nd
approved these changes
Aug 7, 2023
renaynay
reviewed
Aug 7, 2023
renaynay
approved these changes
Aug 7, 2023
renaynay
pushed a commit
to renaynay/celestia-node
that referenced
this pull request
Aug 23, 2023
…elestiaorg#2429) ## Overview Before this change eds was recomputed twice on sampling: 1. inside sampling client to verify data integrity 2. On storing EDS to storage to calculate merkle tree proof nodes. This PR adds `proofsAdder` cache, that will store proof nodes for later use with key features: - cache could be optionally added to context and will be respected by ipld/shrex getter to populate on eds recompute. - cache is safe from second recompute, since it will only allow single recompute write and all consequent recomputes over same cache will be noop. Also this PR reworks writeEDS to use use `proofsAdder` instead of creating new blockstore and iterating over all keys in it. This approach speeds up writeEDS even without pre collected cache. Since there are no more need for blockstore being created, I have simplified writeEDS and removed a lot of redundant code including whole struct of `writeSession`. The PR also adds benchmark that uses disk for badger store (existing benchmark uses in-memory datastore instead of badger). This PR brings significant performance increase. On disk store badger results are: - before PR store.Put operation: **~1.5s** - after PR store.Put with cached proofs: **800ms** - after PR store.Put without cached proofs: **900ms**
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Before this change eds was recomputed twice on sampling:
This PR adds
proofsAddercache, that will store proof nodes for later use with key features:Also this PR reworks writeEDS to use use
proofsAdderinstead of creating new blockstore and iterating over all keys in it. This approach speeds up writeEDS even without pre collected cache.Since there are no more need for blockstore being created, I have simplified writeEDS and removed a lot of redundant code including whole struct of
writeSession.The PR also adds benchmark that uses disk for badger store (existing benchmark uses in-memory datastore instead of badger).
This PR brings significant performance increase. On disk store badger results are: