Libp2p quic second attempt #2159

dvc94ch · 2021-07-26T10:23:23Z

Adds support for tls, removes the UnboundedSenders from the endpoint. The endpoint still has UnboundedReceivers, but that's not different than having a VecDeque. The transport dial api for example doesn't allow applying backpressure.

Closes paritytech/substrate#1334

mxinden · 2021-08-09T09:43:32Z

Thanks @dvc94ch for giving this another shot.

As someone not directly involved in the many Rust QUIC efforts, could you compare this implementation on a high level to the many unfinished efforts out there, namely:

//CC @kpp who might be able to add some additional context as well.

dvc94ch · 2021-08-09T10:18:12Z

well first demie wrote an implementation. then tomaka improved on it. it stalled mainly because of rustls/quinn-proto not having a new release for over a year. this implementation builds on previous work and resolves the open todo's. the tls implementation was updated to the latest spec and was tested with other implementations by kpp (correctness).

features it has that the old version didn't have are proper udp socket handling with ecn codes and gso (throughput). support for plugging in your own crypto (security). support for 0rtt encryption (latency). key logging to help debugging with wireshark (maintainability). other than correctness, throughput, latency, security and maintainability not really sure how a generic connection oriented transport can differentiate itself.

kpp · 2021-08-09T14:08:00Z

@tomaka ^

dvc94ch · 2021-08-09T15:11:01Z

anyone interested in performing a first round of review?

dvc94ch · 2021-08-11T11:23:17Z

@kpp if parity is interested in getting this merged, maybe you could give it a review?

kpp · 2021-08-11T20:05:48Z

Well, my issue is to finish @tomaka’s attempt.

dvc94ch · 2021-08-11T20:09:21Z

Why do you think that is a better starting point?

dvc94ch · 2021-08-11T23:25:29Z

Are you referring to https://github.com/kpp/rust-libp2p/tree/master/transports/tls-quic/src? It seems incomplete to me but the tls code does look quite neat.

kpp · 2021-08-12T03:38:42Z

Why do you think that is a better starting point?

That's an issue I was assigned.

the tls code does look quite neat.

Thank you! I will merge Pierre's code from https://github.com/tomaka/libp2p-rs/tree/quiccc-again/transports/quic into it.

dvc94ch · 2021-08-12T08:50:19Z

So parity decided to finish it's branch now. How convenient.

I assume that the most contentious part is that there is noise support in this branch and you probably think it'll make your code much simpler to support just TLS. It doesn't make much difference as you saw in the crypto trait. Although there isn't support for non ed25519 keys, it's probably not that hard to add.

Even in the quiccc branch there are lots of open todos and missing features. Are you planning on fixing those?

dvc94ch · 2021-08-12T09:30:26Z

In that case I'll continue maintaining libp2p-quic. I assume @mxinden will wait a few months to see what happens before deciding anything.

I'd strongly prefer for us to implement something that is already specified (i.e. TLS+QUIC) and not start creating parity-nquic that will get abandoned after 6 months.

Let's hope your branch doesn't get abandonned again in 6 months.

tomaka · 2021-08-12T13:18:02Z

So parity decided to finish it's branch now. How convenient.

My (and Parity's) position has never ever changed: we want QUIC+TLS, not QUIC+Noise, and the reason the branch was stalled is that either nobody is working on it or (for the longest time) cross-dependencies issues w.r.t. quinn-proto and rustls.

dvc94ch · 2021-08-12T14:46:51Z

This branch supports TLS too, although @kpp seems to have done some improvements and cleanup.

kpp · 2021-08-16T10:55:05Z

So parity decided to finish it's branch now. How convenient.

I talked to Pierre. Let's unite on your PR.

transports/quic/src/tls/certificate.rs

transports/quic/Cargo.toml

transports/quic/src/transport.rs

transports/quic/tests/smoke.rs

transports/quic/Cargo.toml

transports/quic/src/crypto.rs

transports/quic/src/lib.rs

transports/quic/tests/smoke.rs

dvc94ch · 2021-09-11T09:58:49Z

so last missing thing is merging master and getting the build green. great work @kpp and @Demi-Marie

kpp · 2021-09-11T10:11:34Z

Let's wait #2226, #2227 and possibly #2228 before merging master.

* protocols/gossipsub: Fix inconsistency in mesh peer tracking (libp2p#2189) Co-authored-by: Age Manning <Age@AgeManning.com> * misc/metrics: Add auxiliary crate to record events as OpenMetrics (libp2p#2063) This commit adds an auxiliary crate recording protocol and Swarm events and exposing them as metrics in the OpenMetrics format. * README: Mention security@ipfs.io * examples/: Add file sharing example (libp2p#2186) Basic file sharing application with peers either providing or locating and getting files by name. While obviously showcasing how to build a basic file sharing application, the actual goal of this example is **to show how to integrate rust-libp2p into a larger application**. Architectural properties - Clean clonable async/await interface ([`Client`]) to interact with the network layer. - Single task driving the network layer, no locks required. * examples/README: Give an overview over the many examples (libp2p#2194) * protocols/kad: Enable filtering of (provider) records (libp2p#2163) Introduce `KademliaStoreInserts` option, which allows to filter records. Co-authored-by: Max Inden <mail@max-inden.de> * swarm/src/protocols_handler: Impl ProtocolsHandler on either::Either (libp2p#2192) Implement ProtocolsHandler on either::Either representing either of two ProtocolsHandler implementations. Co-authored-by: Thomas Eizinger <thomas@eizinger.io> * *: Make libp2p-core default features optional (libp2p#2181) Co-authored-by: Max Inden <mail@max-inden.de> * core/: Remove DisconnectedPeer::set_connected and Pool::add (libp2p#2195) This logic seems to be a leftover of libp2p#889 and unused today. * protocols/gossipsub: Use ed25519 in tests (libp2p#2197) With f2905c0 the secp256k1 feature is disabled by default. Instead of enabling it in the dev-dependency, simply use ed25519. * build(deps): Update minicbor requirement from 0.10 to 0.11 (libp2p#2200) Updates the requirements on [minicbor](https://gitlab.com/twittner/minicbor) to permit the latest version. - [Release notes](https://gitlab.com/twittner/minicbor/tags) - [Changelog](https://gitlab.com/twittner/minicbor/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/twittner/minicbor/compare/minicbor-v0.10.0...minicbor-v0.11.0) --- updated-dependencies: - dependency-name: minicbor dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): Update salsa20 requirement from 0.8 to 0.9 (libp2p#2206) * build(deps): Update salsa20 requirement from 0.8 to 0.9 Updates the requirements on [salsa20](https://github.com/RustCrypto/stream-ciphers) to permit the latest version. - [Release notes](https://github.com/RustCrypto/stream-ciphers/releases) - [Commits](RustCrypto/stream-ciphers@ctr-v0.8.0...salsa20-v0.9.0) --- updated-dependencies: - dependency-name: salsa20 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * *: Bump pnet to v0.22 Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Max Inden <mail@max-inden.de> * *: Dial with handler and return handler on error and closed (libp2p#2191) Require `NetworkBehaviourAction::{DialPeer,DialAddress}` to contain a `ProtocolsHandler`. This allows a behaviour to attach custom state to its handler. The behaviour would no longer need to track this state separately during connection establishment, thus reducing state required in a behaviour. E.g. in the case of `libp2p-kad` the behaviour can include a `GetRecord` request in its handler, or e.g. in the case of `libp2p-request-response` the behaviour can include the first request in the handler. Return `ProtocolsHandler` on connection error and close. This allows a behaviour to extract its custom state previously included in the handler on connection failure and connection closing. E.g. in the case of `libp2p-kad` the behaviour could extract the attached `GetRecord` from the handler of the failed connection and then start another connection attempt with a new handler with the same `GetRecord` or bubble up an error to the user. Co-authored-by: Thomas Eizinger <thomas@eizinger.io> * core/: Remove deprecated read/write functions (libp2p#2213) Co-authored-by: Max Inden <mail@max-inden.de> * protocols/ping: Revise naming of symbols (libp2p#2215) Co-authored-by: Max Inden <mail@max-inden.de> * protocols/rendezvous: Implement protocol (libp2p#2107) Implement the libp2p rendezvous protocol. > A lightweight mechanism for generalized peer discovery. It can be used for bootstrap purposes, real time peer discovery, application specific routing, and so on. Co-authored-by: rishflab <rishflab@hotmail.com> Co-authored-by: Daniel Karzel <daniel@comit.network> * core/src/network/event.rs: Fix typo (libp2p#2218) * protocols/mdns: Do not fire all timers at the same time. (libp2p#2212) Co-authored-by: Max Inden <mail@max-inden.de> * misc/metrics/src/kad: Set query_duration lowest bucket to 0.1 sec (libp2p#2219) Probability for a Kademlia query to return in less than 100 milliseconds is low, thus increasing the lower bucket to improve accuracy within the higher ranges. * misc/metrics/src/swarm: Expose role on connections_closed (libp2p#2220) Expose whether closed connection was a Dialer or Listener. * .github/workflows/ci.yml: Use clang 11 (libp2p#2233) * protocols/rendezvous: Update prost (libp2p#2226) Co-authored-by: Max Inden <mail@max-inden.de> * *: Fix clippy warnings (libp2p#2227) * swarm-derive/: Make event_process = false the default (libp2p#2214) Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Age Manning <Age@AgeManning.com> Co-authored-by: Ruben De Smet <ruben.de.smet@rubdos.be> Co-authored-by: Thomas Eizinger <thomas@eizinger.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: rishflab <rishflab@hotmail.com> Co-authored-by: Daniel Karzel <daniel@comit.network> Co-authored-by: David Craven <david@craven.ch>

dvc94ch · 2021-09-14T22:57:41Z

@kpp, @DemiMarie can you approve the PR? I think all issues have been addressed.

@mxinden what are the next steps to get this merged?

kpp

So far so good

DemiMarie

I recommend running the tests thousands of times in a tight loop, using netem to simulate packet loss and/or reordering. This caught bugs I was working on my version, both in my code and in quinn-proto.

DemiMarie · 2021-09-15T21:24:47Z

transports/quic/src/tls/verifier.rs

+        _dss: &DigitallySignedStruct,
+    ) -> Result<HandshakeSignatureValid, TLSError> {
+        Err(TLSError::PeerIncompatibleError(
+            "Only TLS 1.3 certificates are supported".to_string(),


I believe this is not reachable as rustls should have aborted the handshake sooner, but panicking here is not critical.

mxinden · 2021-09-25T13:25:50Z

@mxinden what are the next steps to get this merged?

I hope to find some time next week to give this an in-depth review. Thanks for the ping.

mxinden

I am sorry for the delay here.

I still need more time for a full review. I got a couple questions which I am posting ahead of time. Let me know in case you would rather wait for the full review.

mxinden · 2021-10-01T17:06:42Z

transports/quic/src/crypto.rs

@@ -0,0 +1,199 @@
+use libp2p_core::PeerId;


For the sake of consistency, would you mind including license headers here?

mxinden · 2021-10-01T17:09:14Z

transports/quic/src/endpoint.rs

+        /// UDP address to connect to.
+        addr: SocketAddr,
+        /// The remotes public key.
+        public_key: C::PublicKey,


Do I understand correctly that one can thus not dial a remote without their PeerId?

mxinden · 2021-10-01T17:12:22Z

transports/quic/src/endpoint.rs

+
+#[derive(Debug)]
+pub struct TransportChannel<C: Crypto> {
+    tx: mpsc::UnboundedSender<ToEndpoint<C>>,


same here, this is the channel from libp2p not from the socket. other than erroring if the queue is full and trying to dial I don't see what we can do

I don't understand why the direction (libp2p -> socket) matters in regards to bounded or unbounded. Can you expand on that @dvc94ch?

mxinden · 2021-10-01T17:14:12Z

transports/quic/src/endpoint.rs

+            public_key,
+            tx,
+        };
+        self.tx.unbounded_send(msg).expect("endpoint has crashed");


Why can one not use a bounded channel here and await both the sending and the result through the oneshot?

I guess you can do that, but what's the point

The point would be to enable backpressure throughout the system. Through backpressure an upper layer component is paused when the lower layer is overwhelmed and woken up when the lower layer is ready to receive new items.

transports/quic/src/endpoint.rs

mxinden · 2021-10-01T17:25:52Z

transports/quic/src/endpoint.rs

+
+        if me.event_slot.is_none() && me.incoming_slot.is_none() {
+            let mut metas = [RecvMeta::default(); BATCH_SIZE];
+            let mut iovs = MaybeUninit::<[IoSliceMut; BATCH_SIZE]>::uninit();


I am assuming this is an optimization. If so, given the usage of unsafe code, what performance impact did this optimization show in benchmarks?

mxinden · 2021-10-01T17:30:48Z

transports/quic/src/muxer.rs

+            inner.connection.handle_event(event);
+        }
+
+        let _max_datagrams = inner.endpoint.max_datagrams();


Does this method call have any side effects?

no, this is when using quinn-proto from git which changes the api

https://github.com/quinn-rs/quinn/blob/main/quinn-proto/src/connection/mod.rs#L411

mxinden · 2021-10-01T17:32:56Z

transports/quic/src/muxer.rs

+                    // Break here so that the noise upgrade can finish.
+                    return Poll::Pending;


Can you expand on this? Why does this Future have to pause? Where is the waker of the task registered?

this is so the upgrade can complete
https://github.com/libp2p/rust-libp2p/pull/2159/files#diff-b1dcda9367774dd8a742457ff79bc528d030a8cc38d6791efefc78b30f3f2e03R222

mxinden · 2021-10-01T17:34:35Z

transports/quic/src/muxer.rs

+                }
+                Event::Stream(StreamEvent::Readable { id }) => {
+                    tracing::trace!("stream readable {}", id);
+                    if let Some(substream) = inner.substreams.get_mut(&id) {


Is there a scenario where one would receive a StreamEvent::Readable but not have a reference to it in inner.substreams? If not, how about an expect here to enforce this assumption?

transports/quic/src/transport.rs

Co-authored-by: Max Inden <mail@max-inden.de>

dvc94ch · 2021-10-02T10:56:02Z

I no longer work for actyx (they downsized to 3 employees). I'm currently working on a startup with @rklaehn and @wngr and as such my priorities have shifted and I won't be able to devote time to this effort for the time being.

kpp · 2021-10-02T11:16:37Z

I am back next week. 🌴

mxinden

Thank you for the comments @dvc94ch!

A couple of follow ups on the backpressure discussions. I am sorry for being stubborn here. Still I think this is an important aspect of the implementation. I am happy to expand further if need be. Thanks for bearing with me.

mxinden · 2021-10-06T14:19:40Z

transports/quic/src/endpoint.rs

+
+#[derive(Debug)]
+pub struct TransportChannel<C: Crypto> {
+    tx: mpsc::UnboundedSender<ToEndpoint<C>>,


can't prevent a user from dos'ing himself

How does an upper layer component know whether the lower layer is able to keep up. In other words, how does the upper layer know when to add more items and when to slow down? Backpressure through bounded channels enforces just that.

mxinden · 2021-10-06T14:22:31Z

transports/quic/src/endpoint.rs

+            public_key,
+            tx,
+        };
+        self.tx.unbounded_send(msg).expect("endpoint has crashed");


The point would be to enable backpressure throughout the system. Through backpressure an upper layer component is paused when the lower layer is overwhelmed and woken up when the lower layer is ready to receive new items.

mxinden · 2021-10-06T14:26:22Z

transports/quic/src/endpoint.rs

+        &self,
+        id: ConnectionHandle,
+    ) -> (ConnectionChannel<C>, mpsc::Sender<ConnectionEvent>) {
+        let (tx, rx) = mpsc::channel(12);


I wonder whether the queue is needed at all. As a rule of thumb I follow the rule of Queues exist only to handle burstiness. Is creating new connections a bursty task? If not, why introduce a queue?

mxinden · 2021-10-06T14:30:52Z

transports/quic/src/endpoint.rs

+    socket: UdpSocket,
+    crypto_config: Arc<CryptoConfig<C>>,
+    connections: FnvHashMap<ConnectionHandle, mpsc::Sender<ConnectionEvent>>,
+    outgoing: VecDeque<udp_socket::Transmit>,


queues are used throughout the codebase,

That is correct, yes, though I would argue that those places are either (a) not relevant (e.g. libp2p-ping with little to no bandwidth usage) or (b) a deficiency introduced in the past.

That said, lower layers of libp2p do enforce backpressure today, e.g. the TCP transport (libp2p-tcp) or the Yamux implementation..

mxinden · 2021-10-06T14:35:53Z

I no longer work for actyx (they downsized to 3 employees).

I am sorry to hear that.

I won't be able to devote time to this effort for the time being.

Understandable. I hope to be able to test this pull request sometime this or next week.

DemiMarie · 2021-10-06T16:04:11Z

@mxinden @dvc94ch I wholeheartedly agree that the unbounded queues and the VecDeque need to go.

kpp · 2021-10-14T11:11:50Z

I no longer work for actyx (they downsized to 3 employees). I'm currently working on a startup with @rklaehn and @wngr and as such my priorities have shifted and I won't be able to devote time to this effort for the time being.

Should I open a new PR because I can push to this PR but to David'd branch through an intermediate PR?

@dvc94ch David, you didn't add license headers neither to https://github.com/ipfs-rust/libp2p-quic nor here. Would you please post license headers you prefer? I appreciate your work on this project, I can't add "Parity 2021". That would be inconvenient.

dvc94ch · 2021-10-14T11:33:11Z

Feel free to open a new PR. I guess as license header you can add Cloudpeer 2021

kpp · 2021-10-14T11:40:12Z

dvc94ch · 2021-10-14T11:43:08Z

not officially founded yet, hq will be in romania, but I guess we can start establishing it as a good open source player.

kpp · 2021-10-14T11:43:58Z

Well, I believe the header should state the company you were working on while writing this code. Or your name if it was your pet project.

dvc94ch · 2021-10-14T11:44:46Z

then David Craven if you prefer

dvc94ch added 3 commits July 16, 2021 16:03

Add libp2p-quic.

7431703

Disable build on wasm.

7c65806

Update to v0.6.0

eb3bc68

dvc94ch force-pushed the libp2p-quic branch from 7fa60ea to 2c6a10c Compare July 26, 2021 10:29

Merge remote-tracking branch 'upstream/master' into libp2p-quic

f6c32fe

dvc94ch force-pushed the libp2p-quic branch from 2c6a10c to f6c32fe Compare July 26, 2021 10:35

dvc94ch added 2 commits August 9, 2021 12:01

Update to latest libp2p-quic.

a762d14

Update imports.

4a69421

Merge branch 'master' into libp2p-quic

a93cf96

Merge branch 'master' into libp2p-quic

a1c06ae

kpp reviewed Aug 16, 2021

View reviewed changes

transports/quic/src/tls/certificate.rs Outdated Show resolved Hide resolved

transports/quic/Cargo.toml Show resolved Hide resolved

kpp reviewed Aug 16, 2021

View reviewed changes

transports/quic/Cargo.toml Outdated Show resolved Hide resolved

transports/quic/Cargo.toml Outdated Show resolved Hide resolved

transports/quic/src/transport.rs Show resolved Hide resolved

transports/quic/tests/smoke.rs Show resolved Hide resolved

kpp reviewed Aug 16, 2021

View reviewed changes

transports/quic/Cargo.toml Outdated Show resolved Hide resolved

kpp reviewed Aug 16, 2021

View reviewed changes

transports/quic/src/crypto.rs Outdated Show resolved Hide resolved

transports/quic/src/lib.rs Outdated Show resolved Hide resolved

kpp mentioned this pull request Aug 24, 2023

Tracking issue for QUIC support paritytech/polkadot-sdk#536

Open

10 tasks

dvc94ch commented Aug 30, 2021

View reviewed changes

transports/quic/tests/smoke.rs Show resolved Hide resolved

kpp and others added 5 commits September 8, 2021 13:40

Use std HashMap instead of fnv's

f760494

Remove ToLibp2p to_keypair

9a88cf2

Move iovs init into a separate function

35ac0fb

Use mem::replace to unbind recv_buf

d665212

Update transports/quic/src/endpoint.rs

ee7a7f5

kpp and others added 2 commits September 14, 2021 17:11

Merge branch 'master' into libp2p-quic

314e1c5

kpp approved these changes Sep 15, 2021

View reviewed changes

DemiMarie approved these changes Sep 16, 2021

View reviewed changes

mxinden reviewed Oct 1, 2021

View reviewed changes

Update transports/quic/src/transport.rs

9b36bc1

Co-authored-by: Max Inden <mail@max-inden.de>

mxinden reviewed Oct 6, 2021

View reviewed changes

mxinden mentioned this pull request Oct 7, 2021

misc/multistream-select: Implement simultaneous open extension #2066

Closed

dvc94ch closed this Oct 14, 2021

kpp mentioned this pull request Oct 14, 2021

transports/quic: Add implementation based on quinn-proto #2289

Merged

7 tasks

elenaf9 mentioned this pull request Sep 9, 2022

[Tracking Issue] transports/quic: Add QUIC Transport #2883

Closed

15 tasks

		// Break here so that the noise upgrade can finish.
		return Poll::Pending;

Libp2p quic second attempt #2159

Libp2p quic second attempt #2159

Conversation

dvc94ch commented Jul 26, 2021 • edited Loading

mxinden commented Aug 9, 2021

dvc94ch commented Aug 9, 2021 • edited Loading

kpp commented Aug 9, 2021

dvc94ch commented Aug 9, 2021

dvc94ch commented Aug 11, 2021

kpp commented Aug 11, 2021

dvc94ch commented Aug 11, 2021

dvc94ch commented Aug 11, 2021

kpp commented Aug 12, 2021

dvc94ch commented Aug 12, 2021

dvc94ch commented Aug 12, 2021

tomaka commented Aug 12, 2021 • edited Loading

dvc94ch commented Aug 12, 2021

kpp commented Aug 16, 2021

dvc94ch commented Sep 11, 2021

kpp commented Sep 11, 2021

dvc94ch commented Sep 14, 2021

kpp left a comment

Choose a reason for hiding this comment

DemiMarie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden commented Sep 25, 2021

mxinden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dvc94ch commented Oct 2, 2021

kpp commented Oct 2, 2021

mxinden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden commented Oct 6, 2021

DemiMarie commented Oct 6, 2021

kpp commented Oct 14, 2021

dvc94ch commented Oct 14, 2021

kpp commented Oct 14, 2021

dvc94ch commented Oct 14, 2021

kpp commented Oct 14, 2021 • edited Loading

dvc94ch commented Oct 14, 2021

dvc94ch commented Jul 26, 2021 •

edited

Loading

dvc94ch commented Aug 9, 2021 •

edited

Loading

tomaka commented Aug 12, 2021 •

edited

Loading

kpp commented Oct 14, 2021 •

edited

Loading