feat: iroh-gossip #1149

Frando · 2023-06-29T10:13:23Z

Description

This PR adds iroh-gossip. See n0-computer/iroh-sync#1 for the prehistory.

iroh-gossip exposes a pubsub-like API to broadcast messages on topics. Peers join a topic by connecting to at least one other peer. They can then broadcast messages to all other peers who joined a topic, and subscribe to messages. The protocol is based on HyParView, which gives us an overlay network of all nodes that joined a topic; and PlumTree to create an epidemic broadcast tree for reliably broadcasting messages to all online peers.

The new crate is structured as follows:

The protocol implementation (mod proto) as an IO-less state machine. It exposes a single method to handle InEvents and returns an iterator of OutEvents which must be handled by the implementor (register timers, send messages, emit events). The implementation is generic over a PeerAddress trait, which should be an identifier for a peer. In our case, it will be a PeerId.
- All protocol messages are namespaced by a TopicId, a 32 byte identifier. Topics are seperate swarms and broadcast scopes. The HyParView and PlumTree algorithms both work in the scope of a single topic. Thus, joining multiple topics increases the number of open connections to peers and the size of the local routing table.
- The high level protocol lives in state::State. It mostly dispatches messages based on the topic ID into the matching topic::State. This is the actual protocol state for a topic. It, again, mostly dispatches messages onto the actual protocols: hyparview::State and plumtree::State.
- The implementations of these two modules quite strictly follows their papers 1 and 2. If you find any deviations that are not documented, please report, this is likely a bug.
The net module provides networking for the IO-less protocol impl, through the MagicEndpoint from iroh-net, with a (I think) quite straightforward actor implementation.

The net module has a smoke test, and the PR also includes a chat example that adds signed messages on the trustless base layer.

What this mostly needs is, I think, more testing and validation of the protocol implementation. For this to be practical, I think we should formulate some base usecase scenarios.

The proto module has a very simple simulator and some tests that can take environment variables to increase the cluster size. They passed with 10.000 nodes. They also report some stats. The numbers look OK-ish, but I'd love to compare them properly with either other implementations or estimations from the original paper or other resoures. However, those are hard to find.

Notes & open questions

Right now the plumtree module builds an unbounded cache of all broadcast messages. This will have to change. The simplest might be to move to an LRU cache for now. In the future, we might want to expose a method for the module to request expired messages from a backing storage or so. (The module needs to have a cache of messages to reply for requests by other peers. However these will only come within a reasonable amount of time after message publication / reception.) this is now implemented with an expiring cache
Same for the list of received message IDs - they're also unbounded atm this is now implemented with an expiring cache
We should validate the numbers - either by more simulation tests or by actually comparing them to the paper. However I couldn't find any simulation code from the papers.

Change checklist

Self-review.
Documentation updates if relevant.
[ ]Tests if relevant. some tests are there, some more might be good

iroh-gossip/examples/chat.rs

iroh-gossip/README.md

iroh-gossip/src/lib.rs

iroh-gossip/src/proto/mod.rs

iroh-gossip/src/net/mod.rs

iroh-gossip/src/proto/mod.rs

iroh-gossip/src/proto/state.rs

dignifiedquire · 2023-07-18T09:41:57Z

iroh-gossip/src/proto/util.rs

@@ -0,0 +1,182 @@
+use rand::{


missing unit tests, ideally some proptests for the pieces in this file

I added basic unit tests for IndexSet, TimerMap and the new base32 utils.
What parts would you want to proptest? Can do, just not super sure if/where it would make sense here.

iroh-gossip/src/proto/topic.rs

iroh-gossip/src/proto/plumtree.rs

iroh-gossip/src/proto/mod.rs

@dignifiedquire

## Description This PR adds `iroh-sync`, a document synchronization protocol, to iroh, and integrates with `iroh-net`, `iroh-gossip` and `iroh-bytes`. * At the core is the `iroh-sync` crate, with a set reconciliation algorithm implemented by @dignifiedquire. See [the old iroh-sync repo](https://github.com/n0-computer/iroh-sync/) for the prehistory and #1216 for the initial PR (fully included in this PR, and by now outdated) * Iroh sync is integrated in the iroh node, with iroh-gossip, in the RPC interface, and the CLI. * `LiveSync` is the handle to an actor that integrates sync with [gossip](#1149 ) to broadcast and receive document updates from peers. For each open document a gossip swarm is joined with a `TopicId` derived from the doc namespace. * mod `download` contains the new downloader. It will be improved in #1344 . * mod `client` is the new high-level RPC client. It currently only has methods for dealing with docs and sync, other things should be added once we merged this. CLI commands for sync are in `commands/sync.rs`. Will be much better with #1356 . * `examples/sync.rs` has a REPL to modify and sync docs. It does a full setup without using the iroh console. Also includes code to sync directories, and a hammer command for load testing. * The PR also introduces `iroh::client::Iroh`, a wrapper around the RPC client, and `iroh::client::Doc`, a wrapper around RPC client for a single document ## Notes & open questions #### Should likely happen before merge: * [x] Make `iroh_sync::Store:::list_authors` and `list_replicas` return iterators `iroh-sync` *fixed in #1366 * * [ ] Add `iroh_sync::Store::close_replica` * [x] `ContentStatus` in `on_insert` callback is reported as `Ready` if the content is still `baomap::PartialEntry` (in-process download) *fixed in a8e8093* #### Can happen after merge, but before `0.6` release * [ ] Implement `AuthorImport` and `AuthorShare` RPC & CLI commands * [ ] sync store `list_namespaces` and `list_authors` internally collect, return iterator instead * [ ] Fix cross-compiles to arm/android. See cross-rs/cross#1311 * [ ] Ensure that fingerpring calculation is efficient and/or cached for large documents. Currently calculating the initial fingerprint iterates once over all entries in a document. * [ ] Make content downloads be more reliable * [ ] Add some way to download content from peers independent of the first insertion event for a remote entry. The downloader with retries is tracked in #1334 and 1344, but independent of that, we still would currently only ever try to queue a download when the `on_insert` callback triggers, which is only once. There should be a way, even if manual for now, to try to download missing content in a replica from peers. * [ ] during `iroh-sync` sync include info if content is available for each entry * [ ] Add basic peer management and persistence. Currently live sync will stop to do anything after a node restart. * [ ] Persist the addressbook of peers for a document, to reconnect when restarting the node * [ ] Implement `PeerAdd` and `PeerList` RPC & CLI commands. The latter needs changes in `iroh-net` to expose information of currently-connected peers and their peer info. * [ ] Make read-only replicas possible * [ ] Improve reliablity of replica sync. * sync is triggered on each `NeighborUp` event from gossip. check that we don't sync too much. * maybe include peer info in gossip messages, to queue syncs with those (but not all at once) * track and exchange the timestamp of last full sync for peers, to know if you missed gossiped message and react accordingly * add more tests with peers coming and leaving #### Open questions * [ ] `iroh_sync::EntrySignature` should the signatures include a namespace prefix? * [ ] do we want the 1:1 mapping of `NamespaceId`and gossip `TopicId`, or would the topic id as a hash be better? #### Other TODOs collected from the code * [ ] Port `hammer` and `fs` commands from REPL example to iroh cli * [ ] docs/naming: settle terminology about keypairs, private/secret/signing keys, public keys/identifiers and make docs and symbols consistent * [ ] Make `bytes_get` streaming in the RPC interface * [ ] Allow to unset the subscription on a replica * [ ] `iroh-sync` shouldn't depend on `iroh-bytes` only for `Hash` type -> #1354 * [ ] * [ ] Move `sync::live::PeerSource` to iroh-net or even better -> #1354 * [ ] `StoreInstance::put` propagate error and verify timestamp is reasonable. * [ ] `StoreInstance::get_range` implement inverted range * [ ] `iroh_sync`: Remove some items only used in tests (marked with #[cfg(test)]) * [ ] `iroh_sync` fs store: verify get method fetches all keys with this namespace * [ ] `ranger::SimpleStore::get_range`: optimize * [ ] `ranger::Peer` avoid allocs? * [ ] `fs::StoreInstance::get_fingerprint` optimize * [ ] `SyncEngine::doc_subscribe` remove unwrap, handle error ## Change checklist - [x] Self-review. - [x] Documentation updates if relevant. - [ ] Tests if relevant. --------- Co-authored-by: dignifiedquire <me@dignifiedquire.com> Co-authored-by: Asmir Avdicevic <asmir.avdicevic64@gmail.com> Co-authored-by: Kasey <klhuizinga@gmail.com>

* feat: iroh-gossip initial commit This commit imports iroh-gossip into the repo. See n0-computer/iroh-sync#1 for the prehistory. * From that original PR comes my IO-less implementation of the gossip protocol (Plumtree broadcast trees and HyParView cluster membership). The implementations are quite strict according to the papers [1](https://asc.di.fct.unl.pt/~jleitao/pdf/dsn07-leitao.pdf) and [2](https://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf). If you find any deviations that are not documented, please report, this is likely a bug. * The combined protocol for Plumtree and Hyparview is additionally namespaced with a 32-byte topic ID to allow multiple broadcast trees (one per topic) to coexist. This means: Each topic is a seperate broadcast tree with seperate swarm membership and active connections (connections to the same peers are still shared on the network layer) This IO-less protocol implementation is in the `proto` module. This PR *also* adds my work from the past week: A `net` module that provides networking for the IO-less protocol impl, through the `MagicEndpoint` from iroh-net, with a (I think) quite straightforward actor implementation. The `net` module has a smoke test, and the PR also includes a `chat` example that adds signed messages on the trustless base layer. What this mostly needs is, I think, more testing and validation of the protocol implementation. For this to be practical, I think we should formulate some base usecase scenarios. The `proto` module has a very simple simulator and some tests that can take environment variables to increase the cluster size. They passed with 10.000 nodes. They also report some stats. The numbers look OK-ish, but I'd love to compare them properly with either other implementations or estimations from the original paper or other resoures. However, those are hard to find. * fix(gossip): make util pub * fix(gossip): better naming of cli arguments in chat example * chore: make clippy happy * feat: transmit opaque PeerData on join, forward and neighbor ... and wire it up in net module to transmit your endpoint addresses * example: improve gossip chat example * chore: clippy * fix(gossip): update dep version after rebase * fix: changes after rebase * change(gossip): cleanup and better debug impls * fix(gossip): disconnect handling * change: do not error on connect without addrs Before MagicEndpoint::connect would error if called without addrs or derp_region. This commit changes this by only warning and not bailing. This allows to connect to peers by peer id only if addresses or a derp_region were provided previously. The downside is that we lose a valuable error case. A more proper fix will need to move the check if a peer is dialable into the actor. * fix: adapt for changes in iroh-net, and use genawaiter * chore: clippy * docs: document public items in iroh-gossip also renames mod.rs files to modname.rs * tests: move gossip simulator into tests module * tests: make simulation tests work with top level protocol state * change: use derive_more::Debug * feat: add command to leave a topic * docs: document default plumtree config * docs: default values for hyparview * docs: add high-level description of iroh-gossip protocol * test/refactor: add unit test for plumtree optimization step * refactor: improve chat example and address review * fix: do not stop actor if conn closes before event is processed * docs: more links * fix: changes after rebase * change: remove unneeded serialize in iroh-net added earlier * fix: make tests work and cleanup some namings * chore: clippy * refactor: improve and cleanup chat example * refactor: remove unneeded InEvent variant in plumtree * refactor: improve gossip example further * docs for le people and le me * fmt * docs: plumtree * docs: hyparview * feat: validate gossip message ids on receive * chore: cleanup * feat: metrics, and backport fixes from sync-gossip-bytes branch * feat: expose which peer sent us a gossip message * fix: postcard is required * fix: only graft lazy peers in optimize_tree * chore: remove leftover comment * docs(hyparview): improve with divma's suggestions * refactor: impl IntoIterator for IndexSet * fix(hyparview): only trigger ForwardJoin if peer is not yet active * fix(hyparview): reply to shuffle if active_view.len() is 1 or 0 * refactor: rename methods for clarity * fix: do not auto-join topics on external requests * docs: typos * docs: fix * refactor: rename GossipHandle to just Gossip * refactor: less code, more derive * docx: fix * refactor: add util module for base32 parse/fmt * refactor: make PeerData optional * refactor: do not use shift_remove in indexset, not needed * test: add unittests for IndexSet * test: TimerMap and base32 utils * refactor: rename trait PeerAddress to PeerId an generic PA to PI * feat: evict messages after some time * chore: clippy & fmt * fix: renames in doc comments * fix: use rand_chacha in tests for predictable cross-arch results * chore: fmt * chore: remove unneeded line * test: try to fix cross-arch rngs * test: always recreate RNG for cross-arch compat * fix: change after rebase on main * chore: remove debug leftovers --------- Co-authored-by: Diva M <divma@protonmail.com>

@dignifiedquire

## Description This PR adds `iroh-sync`, a document synchronization protocol, to iroh, and integrates with `iroh-net`, `iroh-gossip` and `iroh-bytes`. * At the core is the `iroh-sync` crate, with a set reconciliation algorithm implemented by @dignifiedquire. See [the old iroh-sync repo](https://github.com/n0-computer/iroh-sync/) for the prehistory and #1216 for the initial PR (fully included in this PR, and by now outdated) * Iroh sync is integrated in the iroh node, with iroh-gossip, in the RPC interface, and the CLI. * `LiveSync` is the handle to an actor that integrates sync with [gossip](#1149 ) to broadcast and receive document updates from peers. For each open document a gossip swarm is joined with a `TopicId` derived from the doc namespace. * mod `download` contains the new downloader. It will be improved in #1344 . * mod `client` is the new high-level RPC client. It currently only has methods for dealing with docs and sync, other things should be added once we merged this. CLI commands for sync are in `commands/sync.rs`. Will be much better with #1356 . * `examples/sync.rs` has a REPL to modify and sync docs. It does a full setup without using the iroh console. Also includes code to sync directories, and a hammer command for load testing. * The PR also introduces `iroh::client::Iroh`, a wrapper around the RPC client, and `iroh::client::Doc`, a wrapper around RPC client for a single document ## Notes & open questions #### Should likely happen before merge: * [x] Make `iroh_sync::Store:::list_authors` and `list_replicas` return iterators `iroh-sync` *fixed in #1366 * * [ ] Add `iroh_sync::Store::close_replica` * [x] `ContentStatus` in `on_insert` callback is reported as `Ready` if the content is still `baomap::PartialEntry` (in-process download) *fixed in a8e8093* #### Can happen after merge, but before `0.6` release * [ ] Implement `AuthorImport` and `AuthorShare` RPC & CLI commands * [ ] sync store `list_namespaces` and `list_authors` internally collect, return iterator instead * [ ] Fix cross-compiles to arm/android. See cross-rs/cross#1311 * [ ] Ensure that fingerpring calculation is efficient and/or cached for large documents. Currently calculating the initial fingerprint iterates once over all entries in a document. * [ ] Make content downloads be more reliable * [ ] Add some way to download content from peers independent of the first insertion event for a remote entry. The downloader with retries is tracked in #1334 and 1344, but independent of that, we still would currently only ever try to queue a download when the `on_insert` callback triggers, which is only once. There should be a way, even if manual for now, to try to download missing content in a replica from peers. * [ ] during `iroh-sync` sync include info if content is available for each entry * [ ] Add basic peer management and persistence. Currently live sync will stop to do anything after a node restart. * [ ] Persist the addressbook of peers for a document, to reconnect when restarting the node * [ ] Implement `PeerAdd` and `PeerList` RPC & CLI commands. The latter needs changes in `iroh-net` to expose information of currently-connected peers and their peer info. * [ ] Make read-only replicas possible * [ ] Improve reliablity of replica sync. * sync is triggered on each `NeighborUp` event from gossip. check that we don't sync too much. * maybe include peer info in gossip messages, to queue syncs with those (but not all at once) * track and exchange the timestamp of last full sync for peers, to know if you missed gossiped message and react accordingly * add more tests with peers coming and leaving #### Open questions * [ ] `iroh_sync::EntrySignature` should the signatures include a namespace prefix? * [ ] do we want the 1:1 mapping of `NamespaceId`and gossip `TopicId`, or would the topic id as a hash be better? #### Other TODOs collected from the code * [ ] Port `hammer` and `fs` commands from REPL example to iroh cli * [ ] docs/naming: settle terminology about keypairs, private/secret/signing keys, public keys/identifiers and make docs and symbols consistent * [ ] Make `bytes_get` streaming in the RPC interface * [ ] Allow to unset the subscription on a replica * [ ] `iroh-sync` shouldn't depend on `iroh-bytes` only for `Hash` type -> #1354 * [ ] * [ ] Move `sync::live::PeerSource` to iroh-net or even better -> #1354 * [ ] `StoreInstance::put` propagate error and verify timestamp is reasonable. * [ ] `StoreInstance::get_range` implement inverted range * [ ] `iroh_sync`: Remove some items only used in tests (marked with #[cfg(test)]) * [ ] `iroh_sync` fs store: verify get method fetches all keys with this namespace * [ ] `ranger::SimpleStore::get_range`: optimize * [ ] `ranger::Peer` avoid allocs? * [ ] `fs::StoreInstance::get_fingerprint` optimize * [ ] `SyncEngine::doc_subscribe` remove unwrap, handle error ## Change checklist - [x] Self-review. - [x] Documentation updates if relevant. - [ ] Tests if relevant. --------- Co-authored-by: dignifiedquire <me@dignifiedquire.com> Co-authored-by: Asmir Avdicevic <asmir.avdicevic64@gmail.com> Co-authored-by: Kasey <klhuizinga@gmail.com>

dignifiedquire reviewed Jun 29, 2023

View reviewed changes

iroh-gossip/examples/chat.rs Show resolved Hide resolved

Frando force-pushed the Frando/gossip branch from 9854c9a to f8a1422 Compare June 30, 2023 11:48

Frando force-pushed the Frando/magic-endpoint branch from 00f5539 to 0129484 Compare June 30, 2023 15:43

Frando force-pushed the Frando/gossip branch 4 times, most recently from 608c72a to 119d6b9 Compare July 6, 2023 09:17

Frando mentioned this pull request Jul 6, 2023

[WIP] feat: Add iroh-sync and integrate with gossip #1177

Closed

Frando force-pushed the Frando/gossip branch from d261a0a to b894f24 Compare July 7, 2023 10:07

Frando changed the base branch from Frando/magic-endpoint to main July 7, 2023 10:14

Frando force-pushed the Frando/gossip branch from 5264882 to 4cae498 Compare July 12, 2023 15:29

Frando mentioned this pull request Jul 17, 2023

feat: specify a DERP region for the peer you are trying to connect to #1222

Merged

3 tasks