RAPIDE & IPSL: first proof of concept #148
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jorropo
force-pushed
the
rapide
branch
3 times, most recently
from
6539fe9
to
7d30d50
Compare
Codecov Report
@@ Coverage Diff @@
## very-fast-downloads ipfs/go-libipfs#148 +/- ##
=======================================================
+ Coverage 18.50% 20.37% +1.86%
=======================================================
Files 96 103 +7
Lines 10337 10807 +470
=======================================================
+ Hits 1913 2202 +289
- Misses 8152 8320 +168
- Partials 272 285 +13
|
|
Is there a design doc or spec for rapide? The linked issues don't point me to one. |
|
@iand there is https://www.notion.so/pl-strflt/RAPIDE-3c4fb9e159ae471bb426bb81855cee16 which is a bit outdated (during the implementations I found new tricks which make the implementation much easier, the big arching idea is correct tho). |
|
Presentation of that work here: https://www.youtube.com/watch?v=JZheMPTxvLc |
Jorropo
force-pushed
the
rapide
branch
3 times, most recently
from
78269ef
to
719b5e6
Compare
Jorropo
force-pushed
the
rapide
branch
3 times, most recently
from
74a2a52
to
0932d3b
Compare
This still requires lots of work. This include a mostly complete compiler, (minus a few types and literals). More nodes and scopes will be required.
This include a working single threaded structure for RAPIDE. Todo: - Multithreading - dontGoThere map for serverDrivenWorker - Racing
This help avoiding situations where you go race someone else, in a shallow dag, and you get a block next to them in the traversal.
We do not have consensus nor usecase for the language part of IPSL, to avoid long debates for some not yet usefull pieces of code, I've removed it and we will handle merging this part seperately (or never merge it).
In 719b5e6 I changed `download.expand` to release the node lock before returning. This allows other workers to access the just expanded node while we walk up the chain ancestry. This mean that when we relock the node later to add it to our list of tasks, someone else may have downloaded some of the blocks already and thus they are not in the childrens slice and thus we wont add them to our list. This also solve the case where you give a Traversal that is not supported by the underlying protocol, in that case we want to kill the current download and retry deeper, while previously we would hard error on this and kill the worker.
This was referenced Feb 15, 2023
|
Moved to ipfs#1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This support the code for a proof of concept of RAPIDE which concurrently download from multiple gateways.
Todo:
dontGoThere map for serverDrivenWorkerIn memory benchmarks using fake latency andio.Readershaping.Include: #33
Fixes ipfs#4 ipfs#2