Proposal: Zero allocation connectionless sockets

[scalablecory]

This proposal eliminates allocations for connectionless use of Socket. It augments the SocketAddress class to allow reuse across operations, becoming a high-perf alternative to EndPoint.

Rationale and Usage

APIs which need to translate between IPEndPoint and native sockaddr structures are performing a large amount of defensive copying and layering workarounds.

This affects UDP performance and contributes to excessive GC. A simple example is:

Socket socket = ...;
byte[] buffer = ...;
var remoteEndPoint = new IPEndPoint(IPAddress.Any, 0);

socket.ReceiveFrom(buffer, 0, buffer.Length, SocketFlags.None, ref remoteEndPoint);
socket.SendTo(buffer, 0, buffer.Length, SocketFlags.None, remoteEndPoint);

These two calls allocate 12 times:

• 3x IPEndPoint
• 3x IPAddress
• 3x SocketAddress
• 3x byte[] ...6x if IPv6

See also: #30196

New usage has 0 allocations:

Socket socket = ...;
byte[] buffer = ...;
var remoteAddress = new SocketAddress(AddressFamily.InterNetwork);

socket.ReceiveFrom(buffer, 0, buffer.Length, SocketFlags.None, remoteAddress);
socket.SendTo(buffer, 0, buffer.Length, SocketFlags.None, remoteAddress);

Proposed API

class Socket
{
	public int ReceiveFrom(Span<byte> buffer, SocketFlags socketFlags, SocketAddress socketAddress);
	public ValueTask<int> ReceiveFromAsync(Memory<byte> buffer, SocketFlags socketFlags, SocketAddress socketAddress, CancellationToken cancellationToken = default);

	public int SendTo(ReadOnlySpan<byte> buffer, SocketFlags socketFlags, SocketAddress socketAddress);
	public ValueTask<int> SendToAsync(ReadOnlyMemory<byte> buffer, SocketFlags socketFlags, SocketAddress socketAddress, CancellationToken cancellationToken = default);
}

class SocketAsyncEventArgs
{
	// Only one of RemoteEndPoint or RemoteAddress must be specified.
	public SocketAddress RemoteAddress { get; set; }
}

class SocketAddress
{
	// If we can merge System.Net.Primitives and System.Net.Sockets, these two methods are unnecessary. That would be ideal.
	public static void GetBuffer(SocketAddress address, out byte[] buffer);
	public static void SetSockaddrSize(SocketAddress address, int size);
}

class EndPoint
{
	public virtual void SerializeTo(SocketAddress socketAddress); // Already has "SocketAddress Serialize()"; default would be to call that and copy.
}

Details

• It is intended that UDP servers will use SocketAddress as a dictionary key to lookup client state, to avoid first converting to EndPoint.
  • It is assumed that users will only rarely care to actually get the IP/port/etc. from the SocketAddress. This duty continues to be delegated to EndPoint.
  • If users ever want to deserialize a SocketAddress into an EndPoint, they can already use EndPoint.Create.
  • Need to ensure only the actual sockaddr structure is compared/hashed, not the entire byte buffer.
• SocketAddress is currently duplicated in System.Net.Primitives and System.Net.Sockets to avoid exposing its internal buffer. This change will allow avoiding duplication.
• This relies on users not using a SocketAddress until the I/O is finished.
  • This is a bit safer with EndPoint as we can take defensive copies before methods return.
• We can currently do some optimizations to avoid all allocations for SendTo and SendToAsync IPv4/IPv6 with some special casing, so this API would primarily be to optimize ReceiveFrom variants as well as (less important) allowing non-IPv4/IPv6 protocols to benefit. Still, if we were to add an API for ReceiveFrom we would probably want an API on SendTo for symmetry.

Open Questions

• We've put some effort into not doing something like this before. It would be great to understand why. Currently:
  • We duplicate the SocketAddress class in multiple assemblies to avoid exposing its buffer, and have a step to marshal (byte-by-byte) between the two implementations.
  • Tons of APIs take EndPoint, it's a nice abstraction that we wanted here despite performance implications.
• It isn't immediately obvious from the API surface that socketAddress is written to by ReceiveFrom. Is there a better way we can indicate this?
• The two new methods on SocketAddress exist purely because System.Net.Sockets needs access to internals in System.Net.Primitives. Any thoughts on how to avoid exposing these "pubternal" bits?
  • One option is to merge System.Net.Primitives and System.Net.Sockets; I don't see harm in this but that is a much larger discussion :)
• If we merge the Primitives and Sockets assemblies, we can get rid of some of the allocations for ReceiveFrom without making any API changes. It's not a perfect solution but might be good enough.

Related Issues

There are two additional issues to update our APIs with ValueTask/Span/Memory that this will need to be consistent with:

• Update Socket class add Span overloads for Socket datagram functions #938
• Update UdpClient class UdpClient - add Span support #864

[stephentoub]

If the core issue is repeated conversions from EndPoint to SocketAddress, why not cache it on the EndPoint instead?

And aren't some of the allocations you mention a side-effect of what used to be one implementation being split between multiple assemblies? If that split is resulting in a perf impact, we should consider moving types into the same assembly to avoid the overhead, rather than adding new APIs.

Also, we separately want ValueTask/Memory-based APIs for UDP. Are you proposing that these be them? Would we not also have EndPoint-based APIs?

This feels to me like it's breaking an abstraction that shouldn't be broken, and we should have exhausted every other avenue before doing so.

[scalablecory]

If the core issue is repeated conversions from EndPoint to SocketAddress, why not cache it on the EndPoint instead?

Yes, I considered this option. We've been careful to make defensive copies when using EndPoint, which indicates to me that this could be a subtly breaking behavior change if sendto was suddenly not doing this. Also, given that EndPoint has Serialize and Create to enable EndPoint / sockaddr translation, it looks like a decision was made to explicitly not bake things into a single class -- I'd love some background knowledge on this if you've got it.

And aren't some of the allocations you mention a side-effect of what used to be one implementation being split between multiple assemblies? If that split is resulting in a perf impact, we should consider moving types into the same assembly to avoid the overhead, rather than adding new APIs.

Agreed, this is one of my open questions. I don't really see a downside to merging at least System.Net.Primitives and System.Net.Sockets, but I don't have the background on why they were split in the first place. If you have any knowledge here it would be helpful. Doing so would help us cut out many of these allocations.

Also, we separately want ValueTask/Memory-based APIs for UDP. Are you proposing that these be them?

No, that is not part of this proposal. We should of course make both proposals consistent if it moves forward.

This feels to me like it's breaking an abstraction that shouldn't be broken, and we should have exhausted every other avenue before doing so.

Agreed, this is one of my open questions. Hoping to see some collaboration on those potential avenues.

[scalablecory]

Related:

• span-based UDP using EndPoint add Span overloads for Socket datagram functions #938.
• span-based UdpClient UdpClient - add Span support #864

[wfurt]

User may not care about port but you alway have to specify one for sending.

socket.SendTo(buffer, 0, buffer.Length, SocketFlags.None, remoteAddress);

will not work unless you use Connect() before. And when you do, you do not need the remoteAddress/remoteEndPoint at all. Did I miss something?

[geoffkizer]

If you have any knowledge here it would be helpful.

I think there are two historical things going on here. @stephentoub may have more context.

(1) These APIs were originally designed a long time ago and at the time, minimizing allocations was not a major priority, unfortunately.
(2) During the assembly refactoring for .NET Core, some classes that were originally all in the same assembly got moved into separate assemblies. In some cases this has led to unfortunate code duplication and/or inefficiencies introduced to avoid private access to certain classes.

We can and should fix these issues. In other words, there's some bad legacy here we can improve on, I think.

[geoffkizer]

It seems to me that we need to consider the send case and the receive case separately here.

On the send side, there's no reason we need to do any copies at all. All we need to do is produce the native sockaddr bytes when we do the native call. Currently, that means we need to call IPEndPoint.Serialize which will allocate a SocketAddress etc. But if we just had something like IPEndPoint.Serialize(Span sockAddrBuffer) then I think we could avoid allocation in this path entirely without changing the top-level API. I'm glossing over details here so please correct me if this is more complicated than I'm describing

On the receive side, we ultimately need to return the endpoint info somehow. In your proposal we're mutating an existing SocketAddress, so you are correct that this API does not allocate; but presumably every user of this API will convert the SocketAddress into an IPEndPoint, which will cause allocation at that point, right? I'm not sure what to do about this, but if the goal is truly zero allocation then we need to consider the common usage, not just the API itself.

[scalablecory]

User may not care about port but you alway have to specify one for sending.

@wfurt SocketAddress is an entire sockaddr_in/etc. structure, it includes a port.

[scalablecory]

presumably every user of this API will convert the SocketAddress into an IPEndPoint, which will cause allocation at that point, right?

@geoffkizer I don't think this needs to be a common use. Really you'd use the address to lookup a client's state in a dictionary -- you don't need an EndPoint to do this, but can instead use SocketAddress directly to avoid any conversions. Outside of this, the main reason I can think of to want ip/port is for display purposes... in which case, yes, you'd need to convert to IPEndPoint to get.

[wfurt]

ok. that would make sense. I look at .Net API and I did not see port there. Would you envision adding Port property to it as well?

[wfurt]

https://docs.microsoft.com/en-us/dotnet/api/system.net.socketaddress?view=netcore-2.2

[scalablecory]

Would you envision adding Port property to it as well?

Note it doesn't have an IP property either.

My vision is to keep SocketAddress opaque, and force users to use EndPoint.Serialize and EndPoint.Create to shuffle to/from EndPoint if they need to manipulate the IP/port. My expectation is that this would not be done once per op, but rather once per endpoint at the start of a "connection", so it's not needed.

[scalablecory]

if we just had something like IPEndPoint.Serialize(Span sockAddrBuffer)

@geoffkizer yea, I thought of that. I think keeping the APIs symmetrical is worthwhile, though. Plus, there's some (probably ultimately inconsequential, but...) benefit to avoiding a SerializeTo(span) overhead on every call to SendTo. Less important now if we end up taking another direction with QUIC, but that's a fair amount of redundant calls for such a packet-heavy thing.

[geoffkizer]

A couple related questions that I don't quite understand:

(1) Why does SocketAddress even exist today? Are there any APIs that take it? Is it just so a user can get the bytes for the native sockaddr structure? When would they even need to do this?
(2) Why is the IPEndPoint on ReceiveFrom passed by ref, instead of out?

[geoffkizer]

Really you'd use the address to lookup a client's state in a dictionary -- you don't need an EndPoint to do this, but can instead use SocketAddress directly to avoid any conversions.

Yeah, that's a good point.

I think keeping the APIs symmetrical is worthwhile, though.

I agree but I'm not quite sure the best way to do this.

[scalablecory]

(1) Why does SocketAddress even exist today? Are there any APIs that take it? Is it just so a user can get the bytes for the native sockaddr structure? When would they even need to do this?

I think it is to allow custom implementations of EndPoint to serialize random non-IP sockaddr without 1st party support.

[geoffkizer]

@scalablecory Yeah that makes sense.

[scalablecory]

(2) Why is the IPEndPoint on ReceiveFrom passed by ref, instead of out?

Looks like it's to work within EndPoint's design:

• To know how much to allocate for a sockaddr.
• To then call EndPoint.Create (instance method, that knows how to deserialize for a specific sockaddr format) to create a new EndPoint based on the sockaddr you get from recvfrom.

[geoffkizer]

Ok, but you already had to bind the socket to a LocalEndPoint, right? So why can't you figure that out from the LocalEndPoint?

[scalablecory]

Good point. I can't account for that.

Perhaps just an oversight in the design, or LocalEndPoint was made after ReceiveFrom was made.

[GSPP]

Has it been measured what performance impact these allocations have? I wonder if it matters with real network communication, even just from a CPU usage standpoint.

[scalablecory]

@GSPP unfortunately we only have synthetic benchmarks to look at right now. Having good real-world benchmarks for Sockets is a long-term goal. Realistically, high-frequency protocols like QUIC, uTP, and latency-sensitive games (Unity engine) will see excessive gen-0 GC.

[kripergvg]

@GSPP Here https://github.com/dotnet/corefx/issues/39317 I said that in our game server 40% of our allocations are from from UDP Socket. It is a disaster for us because we use GC.TryStartNoGCRegion and maintenance mode.

[geoffkizer]

@scalablecory I do think this is the right general approach. That is, have APIs that take/return a "sockaddr"-like argument.

I think my main concern here is that the way SocketAddress works is a little weird. One example (that you pointed out) is it's not obvious that ReceiveFrom would fill in the provided SocketAddress. But I'm not sure what to do about this.

[geoffkizer]

One other thought.

It seems like there's some low-hanging fruit here in terms of allocation that we could improve without any new actual API. This would help issues like #30196. Seems like we should try to make this better first before we introduce new API; that will provide benefit to existing customers.

[scalablecory]

@geoffkizer thanks, I appreciate the thought you're putting into this.

One example (that you pointed out) is it's not obvious that ReceiveFrom would fill in the provided SocketAddress. But I'm not sure what to do about this.

Yea I'm not sure either. I wonder if we have any similar APIs we can lean on for inspiration... I'll have to go looking. @bartonjs can you lend us your API design muscle? If we have these two APIs, can you think of a clean way to help us distinguish between SendTo which only reads a SocketAddress, and ReceiveFrom which writes to it?

public int ReceiveFrom(Span<byte> buffer, SocketFlags socketFlags, SocketAddress socketAddress);
public int SendTo(ReadOnlySpan<byte> buffer, SocketFlags socketFlags, SocketAddress socketAddress);

It seems like there's some low-hanging fruit here in terms of allocation that we could improve without any new actual API. This would help issues like #30196. Seems like we should try to make this better first before we introduce new API; that will provide benefit to existing customers.

I've created dotnet/corefx#41039 to merge Primitives and Sockets, which would allow us to get rid of most of these allocations without an API change (and without using the InternalsVisibleTo escape hatch).

Lets decide on that one before moving forward with this one.

[geoffkizer]

Can you help me understand why merging the assemblies helps?

[geoffkizer]

Just looking at the SendTo code....

(1) We always make a "snapshot", i.e. copy, of the IPEndPoint -- which means a copy of the IPAddress as well. We only actually use the snapshot when _rightEndPoint is null, which I believe is uncommon. (I'm actually kind of unclear why we need _rightEndPoint at all here, but let's assume we do.) So at the very least we could just avoid the copy unless _rightEndPoint is null. (There's also some weirdness around dual mode that I'm going to ignore for now.)
(2) We then create an Internals.SocketAddress from the endpoint. But it looks like we special case IPEndPoint here already so we don't need to create a "real" SocketAddress first, anyway. So it doesn't seem like assembly merging actually helps here.
(3) We can and should avoid allocating an Internals.SocketAddress here entirely by reworking the code to generate the appropriate native sockaddr directly on the stack. This could work a couple different ways but it seems doable.

So I think for SendTo at least, we could get to 0 allocation without any API changes at all.

ReceiveFrom is harder, and as the API works it will still require at least an allocation for the returned IPEndPoint and IPAddress. But it seems like we could do better there too.

[geoffkizer]

I gotta say, I really don't understand why we set _rightEndPoint in SendTo.

The main purpose of _rightEndPoint seems to be to store the local endpoint. But in SendTo, we're setting it to the remote endpoint. This seems really weird.

[FakeByte]

Of course, it is undeniable that his intention may just be to show that the built-in API is really not good enough. So much so that he had to choose to wrap the Api himself. If I misunderstood what he meant, then I apologize for that.

Thats what I meant, using a native wrapper is of course not an ideal solution, needing to compile the native dll for each platform separately is very annoying. But for the performance it is worth it at the moment.

[Turnerj]

for me a "quick win" solution would not be enough to convince me to switch to using the built in socket api

Just to be clear with my "quick win" comment, I'm referring to @antonfirsov's comment of work load available for changes in .NET 8. I'd rather even some improvements that we can get in this coming release than just waiting till some release in the future where all the changes required can be tackled at once.

Whatever changes that can be made to .NET 8 given the workload might not be enough to get you to switch to using the managed implementation but it will benefit everybody already using it. When future releases contain more improvements, it may still get to a point where you actually can with minimal or no performance degradation.

Really, all I'm talking about is time to get some changes in now and some more later rather than getting them all at once.

But for the performance it is worth it at the moment.

I am kinda curious how much better throughput/lower latency you're actually finding the native sockets over the .NET implementation. Is it like 2x faster or 10x faster? More? This is relating back to @antonfirsov's comment, if the managed implementation was 2x faster/half the allocations than it is currently, does that even remotely close the gap?

[azureskydiver]

As much as I also agree that getting any quick wins as soon as possible is a great idea, I think some people are discounting the time and effort it will take for some people to switch from their current set of wrappers back to the .NET APIs. If the custom wrapper was written with a completely different API style/approach than the .NET version, obviously it is not a drop in replacement, nor will it be a "simple refactor" .
And after all the effort to switch back to the .NET version, you'll need to do a full set of benchmarking and profiling again to prove to management that it was worth that time and and effort to do the switch back, and to throw away the previous custom work. Furthermoree, you can't just tell your test team/organization that "it's okay, you don't need to do a full regression test".

[sgf]

Thats what I meant, using a native wrapper is of course not an ideal solution, needing to compile the native dll for each platform separately is very annoying. But for the performance it is worth it at the moment.

Well, I misunderstood you, and I apologize for that.

[sgf]

https://github.com/enclave-networks/Enclave.FastPacket
https://github.com/JohannesDeml/NetworkBenchmarkDotNet
https://github.com/enclave-networks/research.udp-perf

here is some repositories maybe little help

[wfurt]

I have PoC for the

class Socket
{
	public int ReceiveFrom(Span<byte> buffer, SocketFlags socketFlags, SocketAddress socketAddress);
	public ValueTask<int> ReceiveFromAsync(Memory<byte> buffer, SocketFlags socketFlags, SocketAddress socketAddress, CancellationToken cancellationToken = default);

	public int SendTo(ReadOnlySpan<byte> buffer, SocketFlags socketFlags, SocketAddress socketAddress);
	public ValueTask<int> SendToAsync(ReadOnlyMemory<byte> buffer, SocketFlags socketFlags, SocketAddress socketAddress, CancellationToken cancellationToken = default);
}

The socketAddress is updated in place - assuming it was created for correct address family and has sufficient capacity. If not, the call would fail e.g. there is no attempt to resize the underlying buffer. And of course you cannot pass same instance to multiple calls.

From the conversation above: for the DNS server:
https://github.com/TurnerSoftware/DinoDNS/blob/main/src/TurnerSoftware.DinoDNS/Connection/Listeners/UdpQueryListener.cs
One can reduce allocations significantly by using the new API directly and allocating SocketAddress inside of the receive loop. It cannot be above the loop as the code does not await HandleRequestAsync. Probably best option would be to make it part of the transitData. Aside from that the SocketAddress is used as opaque object so it should just work.

The Mirroring is somewhat more complicated: https://github.com/MirrorNetworking/Mirror/blob/d226d577c21e557ae09086e0ddc1da63d2704e38/Assets/Mirror/Transports/KCP/kcp2k/highlevel/KcpServer.cs#L149

If the RawReceiveFrom is not called concurrently, one can simply create global SocketAddress for IPv6 - it is ok to pass bigger buffer - my assumption is that the receive call would set SocketAddress.Size to reflect the actually received amount e.g. v4 vs v6 or file length for UDS. The connectionID seems to be simple GetHashCode. That should work on SocketAddress as well - with the caveat that of course different endpoint can possibly produce same hash code. (but that problem already exists) #86872 would make it easier to do extra validation but it feels like it is not necessary strictly speaking.

At the moment I don't have the originally proposed change for SocketAsyncEventArgs. It is not too difficult but for now I did not do it to keep the scope as small as possible. The work is mostly around validation as the field is already used internally and we would need to define and verify behavior for cases when somebody would also specify RemoteEndPoint .

At least for projects discussed above this should be sufficient. If this is good enough for majority here I can drive it for API review and push it to 8.0 before the gates close.

I opened separate #86872 to provide some more convenient methods around SocketAddress but the goal goes beyond UDP. @antonfirsov opened #78993 and that should make it easier for anybody who want's to interact with (IP)EndPoint as some point. While both are somewhat related, we may or may not get them into 8.0. As the work is not excessive I'd be happy to tentatively push both to 8.0 and ditch them if we run out of time.

Lastly #86513 talks about cost of pinning. We had case when the GCHandles fragmented memory and caused OOM. Hopefully that should not be problem for the synchronous version but we talk with @stephentoub about possibility to use NativeMemory in some cases to remove work from GC.

[FakeByte]

@wfurt
The API you proposed looks really nice, changing socket address in place is a simple solution and solves the allocation problems.

[miwarnec]

@wfurt That's amazing. This will elevate C# to a whole new class of use cases.
Super excited about this, and it's great to see that C# devs care so much.

[32haojiufangjia]

什么时候可以完成这项0GC的工作啊？？？期待

[EDIT]
Bing translation:
when will this 0GC job be done??? expect

[sgf]

it seems blocked

#87397
#86872

i think that means new api won't be release with .net 8.0

[antonfirsov]

Note that the label is blocking, not "blocked". This means that the issue is high-priority (blocking the release) which pushes it to the top of the API review backlog.

[sgf]

Note that the label is , not "blocked". This means that the issue is high-priority (blocking the release) which pushes it to the top of the API review backlog.blocking

👍 great.

[wfurt]

This was long journey but the end is near. #87397 added sync & async overloads with SocketAddress. It is possible now to pass it to SendTo & ReceiveFrom in and avoid allocations. Same object will be updated in place e.g. it cannot be shared by multiple threads. On receive, the Size is updated to reflect valid bytes e.g. one will need to slice the Buffer to get valid address. #89808 should help on Windows with avoiding GCHandle & pinning. #90086 also reduce allocation for legacy SendTo & Receive but there is still room for improvements in 9.0.

What missing is direct support for SocketAsyncEventArgs. There did not seems to be any compelling reasons besides completeness. And #86872 did not get the helper functions approved. If anybody needs API to extract useful data from the SocketAddress please comment here or on #86872.

All this should be available ion RC1 build -> just barely made it in for 8.0.
Since changes above should fix most use cases I'm going to close this issue. It would be great if people can pick up RC1 and experiment as everything got in pretty late.

[macaba]

Thank you, it's appreciated.

Should I use new SocketAddress(AddressFamily.InterNetworkV6); to ensure the internal buffer is large enough for a ReceiveFrom that could receive both V4 and V6?

[wfurt]

yes, it should have at least what the given socket can receive. Or more generically

Socket socket = new Socket(SocketType.Dgram, ProtocolType.Udp)
SocketAddress socketAddress = new SocketAddress(socket.Address.Family);

it will default to DualMode when it can and fall-back to IPv4 if needed.
One more note that the the old ReceiveFrom would do work to map IPv4 to IPv6V4Mapped.
The new API leaves the buffer exactly as given by OS. e.g. if you do EndPoint.Create(socketAddress) you may end up with different address and if you care you need to do the mapping yourself.