[WIP] feat: Add iroh-sync and integrate with gossip #1177
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Canceled the CI run, windows tests were running for 1.5 hrs. |
Closed
Frando
changed the title
This commit imports iroh-gossip into the repo. See n0-computer/iroh-sync#1 for the prehistory. * From that original PR comes my IO-less implementation of the gossip protocol (Plumtree broadcast trees and HyParView cluster membership). The implementations are quite strict according to the papers [1](https://asc.di.fct.unl.pt/~jleitao/pdf/dsn07-leitao.pdf) and [2](https://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf). If you find any deviations that are not documented, please report, this is likely a bug. * The combined protocol for Plumtree and Hyparview is additionally namespaced with a 32-byte topic ID to allow multiple broadcast trees (one per topic) to coexist. This means: Each topic is a seperate broadcast tree with seperate swarm membership and active connections (connections to the same peers are still shared on the network layer) This IO-less protocol implementation is in the `proto` module. This PR *also* adds my work from the past week: A `net` module that provides networking for the IO-less protocol impl, through the `MagicEndpoint` from iroh-net, with a (I think) quite straightforward actor implementation. The `net` module has a smoke test, and the PR also includes a `chat` example that adds signed messages on the trustless base layer. What this mostly needs is, I think, more testing and validation of the protocol implementation. For this to be practical, I think we should formulate some base usecase scenarios. The `proto` module has a very simple simulator and some tests that can take environment variables to increase the cluster size. They passed with 10.000 nodes. They also report some stats. The numbers look OK-ish, but I'd love to compare them properly with either other implementations or estimations from the original paper or other resoures. However, those are hard to find.
... and wire it up in net module to transmit your endpoint addresses
Frando
force-pushed
the
Frando/gossip
branch
2 times, most recently
from
17c15d2
to
4c8066e
Compare
|
Closing in favor of #1216 which includes all commits from this PR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR starts with a rebase of #1127 on top of #1149
The main addition so far is
iroh/src/sync/live.rswhich integrates iroh-sync and iroh-gossip to provide live synchronization of a iroh-sync document among peers.There's also an example in
iroh/examples/sync.rsthat showcases live editing a document with peers.There is no integration yet with
iroh-byteswhich also means that the content of entries inserted by peers is missing (only the hash is there). Integration with iroh-bytes will be next.Also there's lots of TODOs in the code.
And something seems to be off for the initial sync (that actually uses iroh-sync). It does not work reliably right now, looking into debugging next.a commit was missing, now fixed