How do we handle transferring large amounts of data?

[Mushroom]

So with the advent of #38, we tested a few large factories (including my main save which has multiple dyson spheres and factories all over the place) - the performance with LiteNetLib is ....less than stellar.

On my save, the starter planet (mostly abandoned, most of the stuff is done in a different system) is 10,449,864 bytes in size (uncompressed). Even with the MTU rising to the maximum size of 1432 bytes, that's over 7k packets just to transfer one factory!

On a loaded connection, this means it takes 20 seconds for the factory to load in - this is not really acceptable. The transfer is the bulk of the time taken, host export takes 255ms and client import takes 150ms.

So, it's likely that we need to look at alternatives to LiteNetLib (this is something they admit themselves here: RevenantX/LiteNetLib#114 - "TCP is better for large data").

I had a search and came up with a couple of alternatives:

• We roll our own with the Socket library in C# - we get no benefits of the communities behind many of the networking libs, but it is the fastest option.
• We use https://github.com/vis2k/Telepathy - this appears to be designed for transferring large amounts of data in the context of Unity MMOs.
• We use https://github.com/nxrighthere/ValveSockets-CSharp (which uses https://github.com/ValveSoftware/GameNetworkingSockets as a backend) - this is it's own protocol, written by valve. Not sure how performant the C# wrapper is, but should be able to handle large amounts of data, and has the backing of Valve.

Does anyone want to benchmark these? I've never been great at benchmarking, and my skillset for making sure things are fair in them in C# is limited, most of my experience is in embedded C.

If anyone wants the copy of the file: https://hunter2.watchingyour.tv/Second_COPY.dsv

[Flapperkewiet]

I found 2 more networking libraries, not sure if they are any good (especially since they are not actively being developed anymore):
lidgren
ForgeNetworkingRemastered

[Mushroom]

Lidgren is a no, but forge doesn't look too out of date.

Been having a play with the valve one today, interop isn't fun, but it seems to be the most promising.

[Mushroom]

So I spent a bit of time yesterday evening and my lunch today investigating the various options. Ultimately it boils down to the fact that a lot of game networking libraries just aren't designed for sending large amounts of data.

However, in my various trials, most of which failed spectacularly as I hit various limits, I actually came out with a surprising winner - WebSockets. The overhead is actually minimal for small amounts of data, and it handles large amounts of data flawlessly.

The particular lib that I used for it was WebSocketSharp, and it transferred in a very basic test 65mb in just under a second, which was considerably faster than liteletlib is currently.

Therefore, I propose we move over to websockets instead - it's not like this is a twitch shooter where every microsecond counts, and we get the added benefit of the in-order nature of TCP and the reliability for free.

Opinions? @hubastard @sp00ktober @Flapperkewiet

[hubastard]

What's your internet speed btw ?

[Mushroom]

My internet speed is 400MBit down, and 35MBit up, but this was all tested on my local machine

[Flapperkewiet]

Doesn't system have a websocket, too? Why use a library?

[Mushroom]

Not in .NET 3.5, and not a server :)

[sp00ktober]

Maybe we also should consider asking @brokenmass to help here as he did a bunch of work on reducing file sizes by optimizing the game data as it seems.
For example this mod

[hongbii]

What about chop the data into several parts, transfer it parallelly, and combine it at the destination?

[Mushroom]

We actually ended up re-implementing the network layer with websockets, where this sort of thing is taken care of at a protocol level, it got merged a few days ago.