feat(s2n-quic-dc): Reuse handshakes across application processes #2854

mehnazyunus · 2025-10-15T17:08:56Z

Release Summary:

Reuse handshakes across application processes to reduce memory utilization.

Resolved issues:

Description of changes:

This PR introduces a manager server and an application server, which are intended to run in separate processes. A client handshakes with and connects to the manager server. The manager server accepts incoming streams and transfers them via Unix domain sockets to the application server. The application server receives these streams and continues communicating with the client. The same handshake can be reused with multiple manager servers.

Call-outs:

Added in comments.

Testing:

Added happy path integration test. The client handshakes once and connects to two manager servers, reusing the same handshake. It then continues communication with separate application servers.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

mehnazyunus · 2025-10-15T17:21:42Z

dc/s2n-quic-dc/src/stream/server/tokio.rs

        let acceptor = udp::Acceptor::new(
            id,
            socket,
            &self.stream_sender,


The Start struct could be fully reused in the manager once we can move this stream sender for UDP to a behavior struct like for TCP. (manager server does not have a stream sender)

mehnazyunus · 2025-10-15T17:44:13Z

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

+                    return Err(Some(error)).into();
+                }
+                WorkerState::Sending { future, event_data } => match future.as_mut().poll(cx) {
+                    Poll::Ready(res) => match res {


I tried reusing the poll_send function here, but wasn't able to resolve the borrowing issues when calling the function with the future from the WorkerState, not sure if there is a way around that.

I'm not worried about it for the initial PR, but we can take a look together at doing so.

mehnazyunus · 2025-10-15T17:46:17Z

dc/s2n-quic-dc/src/stream/server/tokio/uds.rs

+        let recv_buffer: either::Either<_, Channel> = Either::A(recv_buffer);
+        let sub = self.env.subscriber();
+
+        let map = Map::new(


Using a placeholder map here to send to accept_stream, afaict it is only used for handling unexpected packets.

I think we'll want to avoid creating a new map for each stream -- that's pretty expensive.

Could we create a single placeholder map and reuse that for all streams?

Possibly? I'm not sure what we're using the map for. Maybe a placeholder would work, or we can edit the signatures to have Option<&Map> if that is better/easier.

mehnazyunus · 2025-10-15T17:48:38Z

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

+                    Poll::Ready(Ok(ControlFlow::Continue(())))
+                }
+                Err(err) => {
+                    debug!("Error sending message to socket {:?}", err);


Do we have to close the stream here, something like this?

Yes, that seems right.

dc/s2n-quic-dc/events/acceptor.rs

Mark-Simulacrum · 2025-10-15T18:00:57Z

dc/s2n-quic-dc/events/acceptor.rs

+    /// The ID of the stream
+    stream_id: u64,
+
+    /// The amount of time the TCP stream spent in the queue before being enqueued


Which queue is this? I suspect this is time from TCP accept() to UDS sendmsg()?

Yes, the "fresh" queue. Can change the comment to "before being sent over UDS"

Mark-Simulacrum · 2025-10-15T18:11:37Z

dc/s2n-quic-dc/events/acceptor.rs

+    sojourn_time: core::time::Duration,
+
+    /// The number of times the Unix domain socket was blocked on send
+    #[measure("blocked_count")]


Is this for sending this particular stream? It might make sense to switch this (or add) a #[counter(...)] as well.

Measure will emit histograms.

Mark-Simulacrum · 2025-10-15T18:16:18Z

dc/s2n-quic-dc/src/stream/server/application.rs

+    #[allow(dead_code)]
+    env: Environment<S>,
+    #[allow(dead_code)]
+    acceptor_rt: runtime::Shared<S>,


Why do we have the allow(dead_code) here? Are these only kept for a Drop impl or something?

Removed those fields here, but I'm not sure why we have it in the original server in the first place. fmt/clippy/build do not seem to complain without it.

dc/s2n-quic-dc/src/stream/server/application.rs

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

Mark-Simulacrum · 2025-10-15T18:42:14Z

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

+                    return Err(Some(error)).into();
+                }
+                WorkerState::Sending { future, event_data } => match future.as_mut().poll(cx) {
+                    Poll::Ready(res) => match res {


I'm not worried about it for the initial PR, but we can take a look together at doing so.

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

Mark-Simulacrum · 2025-10-15T18:58:04Z

dc/s2n-quic-dc/src/stream/server/tokio/uds.rs

+        let recv_buffer: either::Either<_, Channel> = Either::A(recv_buffer);
+        let sub = self.env.subscriber();
+
+        let map = Map::new(


I think we'll want to avoid creating a new map for each stream -- that's pretty expensive.

quic/s2n-quic-core/src/time/timestamp.rs

dc/s2n-quic-dc/src/stream/server/tokio/uds.rs

Mark-Simulacrum

We need to fix the packet format (i64 -> u64) but otherwise I think this is ready to land.

dc/s2n-quic-dc/src/packet/uds/decoder.rs

Mark-Simulacrum · 2025-10-21T15:17:28Z

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

+            if keys
+                .application
+                .opener
+                .on_decrypt_success(recv_buffer.into())


For a future PR: we will probably want to replace this with a call to decrypt(...), since we ought to verify that this is actually a valid packet before we mark it as used in the replay protection map.

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

dc/s2n-quic-dc/src/stream/server/tokio/uds.rs

mehnazyunus added 7 commits October 14, 2025 22:01

Move mpmc sender to poll behavior struct

9749b32

Move server builder methods to macros

e4eb9ca

Add Unix domain socket send event

75fde84

Add helper functions to access secret

8f4dcd4

Add manager server functionality

f26c672

Add application server functionality

8a1a9d3

Add shared cache happy path test

7c8b019

mehnazyunus requested a review from a team as a code owner October 15, 2025 17:08

mehnazyunus commented Oct 15, 2025

View reviewed changes

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs Show resolved Hide resolved

mehnazyunus commented Oct 15, 2025

View reviewed changes

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs Show resolved Hide resolved

mehnazyunus commented Oct 15, 2025

View reviewed changes

mehnazyunus requested a review from Mark-Simulacrum October 15, 2025 17:49

mehnazyunus force-pushed the shared-cache branch from f29e579 to 1efc867 Compare October 15, 2025 18:07

Fix clippy warnings

ed64761

mehnazyunus force-pushed the shared-cache branch from 1efc867 to ed64761 Compare October 15, 2025 18:16

Mark-Simulacrum reviewed Oct 15, 2025

View reviewed changes

mehnazyunus added 9 commits October 15, 2025 22:48

Add counter for blocked_count

4a75860

Decrypt packet for dedup check

fe198e1

Fix integration test

e7b29b0

Remove acceptor addr from application server

a0fd7f4

Use different socket paths in test

cf206ad

Remove acceptor runtime and mpmc channel from application server

3821caf

Add queue_time to packet

b209b71

Init manager server

8ea84c3

Implement manager server

6777e07

Mark-Simulacrum reviewed Oct 17, 2025

View reviewed changes

quic/s2n-quic-core/src/time/timestamp.rs Outdated Show resolved Hide resolved

Use a single map across streams

4e9967b

Make uds receiver cloneable

d8eca96

Mark-Simulacrum reviewed Oct 20, 2025

View reviewed changes

dc/s2n-quic-dc/src/stream/server/tokio/uds.rs Outdated Show resolved Hide resolved

mehnazyunus added 3 commits October 20, 2025 22:00

Add encode timestamp to packet

da15ead

Use CLOCK_MONOTONIC when not on linux

e2122e8

Add event on uds receive

931e875

Mark-Simulacrum reviewed Oct 21, 2025

View reviewed changes

Mark-Simulacrum requested a review from boquan-fang October 21, 2025 15:23

Use u64 for encode time; use mut packet

2056480

Mark-Simulacrum approved these changes Oct 21, 2025

View reviewed changes

boquan-fang approved these changes Oct 21, 2025

View reviewed changes

Mark-Simulacrum merged commit 44f24af into aws:main Oct 21, 2025
195 of 196 checks passed

feat(s2n-quic-dc): Reuse handshakes across application processes #2854

feat(s2n-quic-dc): Reuse handshakes across application processes #2854

Uh oh!

Conversation

mehnazyunus commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release Summary:

Resolved issues:

Description of changes:

Call-outs:

Testing:

Uh oh!

Uh oh!

Uh oh!

mehnazyunus Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mehnazyunus Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mehnazyunus commented Oct 15, 2025 •

edited

Loading

mehnazyunus Oct 15, 2025 •

edited

Loading

mehnazyunus Oct 15, 2025 •

edited

Loading