Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] feat: Add iroh-sync and integrate with gossip #1177

Closed
wants to merge 20 commits into from
Closed

[WIP] feat: Add iroh-sync and integrate with gossip #1177

wants to merge 20 commits into from

Commits on Jul 12, 2023

  1. feat: iroh-gossip initial commit 

    This commit imports iroh-gossip into the repo. See n0-computer/iroh-sync#1 for the prehistory.

* From that original PR comes my IO-less implementation of the gossip protocol
  (Plumtree broadcast trees and HyParView cluster membership). The
  implementations are quite strict according to the papers
  [1](https://asc.di.fct.unl.pt/~jleitao/pdf/dsn07-leitao.pdf) and
  [2](https://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf). If you
  find any deviations that are not documented, please report, this is
  likely a bug.
* The combined protocol for Plumtree and Hyparview is additionally namespaced
  with a 32-byte topic ID to allow multiple broadcast trees (one per
  topic) to coexist. This means: Each topic is a seperate broadcast tree
  with seperate swarm membership and active connections (connections to
  the same peers are still shared on the network layer)

This IO-less protocol implementation is in the `proto` module.

This PR *also* adds my work from the past week: A `net` module that provides networking
for the IO-less protocol impl, through the `MagicEndpoint` from iroh-net, with a (I think) quite straightforward actor implementation.

The `net` module has a smoke test, and the PR also includes a `chat` example that adds signed messages on the trustless base layer.

What this mostly needs is, I think, more testing and validation of the
protocol implementation. For this to be practical, I think we should
formulate some base usecase scenarios.

The `proto` module has a very simple simulator and some tests that can
take environment variables to increase the cluster size. They passed
with 10.000 nodes. They also report some stats. The numbers look OK-ish,
but I'd love to compare them properly with either other implementations
or estimations from the original paper or other resoures. However, those
are hard to find.
    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    4aa6a9f View commit details
    Browse the repository at this point in the history

  2. fix(gossip): make util pub

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    da56cfd View commit details
    Browse the repository at this point in the history

  3. fix(gossip): better naming of cli arguments in chat example

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    e6245a9 View commit details
    Browse the repository at this point in the history

  4. chore: make clippy happy

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    f46b5c1 View commit details
    Browse the repository at this point in the history

  5. feat: transmit opaque PeerData on join, forward and neighbor 

    ... and wire it up in net module to transmit your endpoint addresses
    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    2200d94 View commit details
    Browse the repository at this point in the history

  6. example: improve gossip chat example

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    825aaaf View commit details
    Browse the repository at this point in the history

  7. chore: clippy

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    4fa3c12 View commit details
    Browse the repository at this point in the history

  8. fix(gossip): update dep version after rebase

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    45779ed View commit details
    Browse the repository at this point in the history

  9. fix: changes after rebase

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    2b3a596 View commit details
    Browse the repository at this point in the history

  10. change(gossip): cleanup and better debug impls

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    40bb716 View commit details
    Browse the repository at this point in the history

  11. fix(gossip): disconnect handling

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    ceb7029 View commit details
    Browse the repository at this point in the history

  12. feat(gossip): use genawaiter for subscribe_all

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    aec61a8 View commit details
    Browse the repository at this point in the history

  13. chore: clippy

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    57db3c9 View commit details
    Browse the repository at this point in the history

  14. fix(gossip): update deps after rebase

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    4cae498 View commit details
    Browse the repository at this point in the history

  15. feat: initial prototype of integrating iroh-sync

    @dignifiedquire @Frando
    dignifiedquire authored and Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    730a4b4 View commit details
    Browse the repository at this point in the history

  16. sync: start impl of multikey

    @dignifiedquire @Frando
    dignifiedquire authored and Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    98c1094 View commit details
    Browse the repository at this point in the history

  17. feat: integrate iroh-sync and iroh-gossip, add example

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    9e129d5 View commit details
    Browse the repository at this point in the history

  18. feat: make the live sync handler work with many docs

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    5e275ac View commit details
    Browse the repository at this point in the history

  19. chore: cleanup and clippy

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    fcb3026 View commit details
    Browse the repository at this point in the history

  20. chore: remove old code and add docs

    @Frando
    Frando committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    f15583b View commit details
    Browse the repository at this point in the history