Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

platform: add GSO support #759

Merged
merged 3 commits into from
Jul 23, 2021
Merged

platform: add GSO support #759

merged 3 commits into from
Jul 23, 2021

Conversation

camshaft
Copy link
Contributor

@camshaft camshaft commented Jul 23, 2021

This change adds GSO for Linux. It passes the segment size through the cmsg(3) interface. A couple things to note:

  • I had to add the ability to dynamically disable GSO when getting an EIO error from the OS. I initially tried to probe just localhost but that led to a false positive, which made all of the interop tests fail.
  • A few of the platform internals had to be changed to accommodate the ability to offload segments. The nice thing is from the transport crate's perspective, basically nothing changed!
  • I ran into an issue with MTU probing where we wrote a packet with 1200 bytes and then we try to push another for the probe but the queue has clamped the payload to 1200 bytes since it has the same destination address. I got around this by adding a can_gso to the tx::Message trait that tells the IO layer if it can be used as part of a segment.
  • I went through and converted all of the storage data structures to be pinned to make sure they have stable pointers. It's unlikely given our usage, but in case they get reallocated it would result in a bunch of dangling pointers.

Overall these changes produce some good results. Here's the before/after of CPU usage to saturate a 1GbE:

Before

s2n-quic cpu

After

s2n-quic-qns cpu

Here's the flamegraphs for the same test:

Before

After

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@camshaft camshaft force-pushed the camshaft/gso branch 2 times, most recently from 463448e to 43d2a90 Compare July 23, 2021 05:42
@camshaft camshaft marked this pull request as ready for review July 23, 2021 07:34
quic/s2n-quic-platform/src/buffer/vec.rs Show resolved Hide resolved
quic/s2n-quic-platform/src/features/gso.rs Outdated Show resolved Hide resolved
quic/s2n-quic-platform/src/features/gso.rs Outdated Show resolved Hide resolved
quic/s2n-quic-platform/src/features/gso.rs Show resolved Hide resolved
quic/s2n-quic-platform/src/message/mmsg.rs Show resolved Hide resolved
quic/s2n-quic-platform/src/message/msg.rs Outdated Show resolved Hide resolved
quic/s2n-quic-platform/src/message/queue/slice.rs Outdated Show resolved Hide resolved
quic/s2n-quic-platform/src/socket/mmsg.rs Show resolved Hide resolved
Co-authored-by: Wesley Rosenblum <55108558+WesleyRosenblum@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants