[WIP] sync integration into ffi #1341
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#1324) * fix(iroh-net): portmapper priority follows described priority strategy * typo
* use the version of bao-tree/BLAKE3 that allows direct access to fast hashing * use published bao-tree and use the reexport instead of depending directly on blake3
* refactor: immutable DerpMap * refactor: remove ability to dynamically change the DerpMap This was not being done anyway * fix(iroh-net): ensure initial re-stun always happens * fix typo Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com> --------- Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
* feat: iroh-gossip initial commit This commit imports iroh-gossip into the repo. See n0-computer/iroh-sync#1 for the prehistory. * From that original PR comes my IO-less implementation of the gossip protocol (Plumtree broadcast trees and HyParView cluster membership). The implementations are quite strict according to the papers [1](https://asc.di.fct.unl.pt/~jleitao/pdf/dsn07-leitao.pdf) and [2](https://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf). If you find any deviations that are not documented, please report, this is likely a bug. * The combined protocol for Plumtree and Hyparview is additionally namespaced with a 32-byte topic ID to allow multiple broadcast trees (one per topic) to coexist. This means: Each topic is a seperate broadcast tree with seperate swarm membership and active connections (connections to the same peers are still shared on the network layer) This IO-less protocol implementation is in the `proto` module. This PR *also* adds my work from the past week: A `net` module that provides networking for the IO-less protocol impl, through the `MagicEndpoint` from iroh-net, with a (I think) quite straightforward actor implementation. The `net` module has a smoke test, and the PR also includes a `chat` example that adds signed messages on the trustless base layer. What this mostly needs is, I think, more testing and validation of the protocol implementation. For this to be practical, I think we should formulate some base usecase scenarios. The `proto` module has a very simple simulator and some tests that can take environment variables to increase the cluster size. They passed with 10.000 nodes. They also report some stats. The numbers look OK-ish, but I'd love to compare them properly with either other implementations or estimations from the original paper or other resoures. However, those are hard to find. * fix(gossip): make util pub * fix(gossip): better naming of cli arguments in chat example * chore: make clippy happy * feat: transmit opaque PeerData on join, forward and neighbor ... and wire it up in net module to transmit your endpoint addresses * example: improve gossip chat example * chore: clippy * fix(gossip): update dep version after rebase * fix: changes after rebase * change(gossip): cleanup and better debug impls * fix(gossip): disconnect handling * change: do not error on connect without addrs Before MagicEndpoint::connect would error if called without addrs or derp_region. This commit changes this by only warning and not bailing. This allows to connect to peers by peer id only if addresses or a derp_region were provided previously. The downside is that we lose a valuable error case. A more proper fix will need to move the check if a peer is dialable into the actor. * fix: adapt for changes in iroh-net, and use genawaiter * chore: clippy * docs: document public items in iroh-gossip also renames mod.rs files to modname.rs * tests: move gossip simulator into tests module * tests: make simulation tests work with top level protocol state * change: use derive_more::Debug * feat: add command to leave a topic * docs: document default plumtree config * docs: default values for hyparview * docs: add high-level description of iroh-gossip protocol * test/refactor: add unit test for plumtree optimization step * refactor: improve chat example and address review * fix: do not stop actor if conn closes before event is processed * docs: more links * fix: changes after rebase * change: remove unneeded serialize in iroh-net added earlier * fix: make tests work and cleanup some namings * chore: clippy * refactor: improve and cleanup chat example * refactor: remove unneeded InEvent variant in plumtree * refactor: improve gossip example further * docs for le people and le me * fmt * docs: plumtree * docs: hyparview * feat: validate gossip message ids on receive * chore: cleanup * feat: metrics, and backport fixes from sync-gossip-bytes branch * feat: expose which peer sent us a gossip message * fix: postcard is required * fix: only graft lazy peers in optimize_tree * chore: remove leftover comment * docs(hyparview): improve with divma's suggestions * refactor: impl IntoIterator for IndexSet * fix(hyparview): only trigger ForwardJoin if peer is not yet active * fix(hyparview): reply to shuffle if active_view.len() is 1 or 0 * refactor: rename methods for clarity * fix: do not auto-join topics on external requests * docs: typos * docs: fix * refactor: rename GossipHandle to just Gossip * refactor: less code, more derive * docx: fix * refactor: add util module for base32 parse/fmt * refactor: make PeerData optional * refactor: do not use shift_remove in indexset, not needed * test: add unittests for IndexSet * test: TimerMap and base32 utils * refactor: rename trait PeerAddress to PeerId an generic PA to PI * feat: evict messages after some time * chore: clippy & fmt * fix: renames in doc comments * fix: use rand_chacha in tests for predictable cross-arch results * chore: fmt * chore: remove unneeded line * test: try to fix cross-arch rngs * test: always recreate RNG for cross-arch compat * fix: change after rebase on main * chore: remove debug leftovers --------- Co-authored-by: Diva M <divma@protonmail.com>
* removes content support from iroh-sync * adds a quick-and-dirty writable database to iroh-bytes (will be replaced with a better generic writable database soon) * adds a `Downloader` to queue get requests for individual hashes from individual peers * adds a `BlobStore` that combines the writable db with the downloader * adds a `Doc` abstraction that combines an iroh-sync `Replica` with a `BlobStore` to download content from peers on-demand * updates the sync repl example to plug it all together * also adds very basic persistence to `Replica` (encode to byte string) and uses this in the repl example
* make the REPL in the sync example work properly with rustyline for editing and reading input, shell-style argument parsing and clap for parsing commands * add a docs store for opening and closing docs * add author to doc struct
uses flume channels to allow for combined sync and async usage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.