Dedicated game server TURN Server

[dbotha]

Hi @paullouisageneau, hope you are well :)

Hopefully this isn't too off topic...

I think I've mentioned in the past that I'm using libdatachannel to facilitate networking in a realtime multiplayer game. Currently one player is chosen to run the server and all other players connect directly to the player. This works great however now I'm looking to add dedicated server support.

I've got a dedicated server build of my game working well but I have some tricky network conditions I need to work around. Essentially I want to use Kubernetes (and Agones) to manage a cluster of dedicated game servers. For reasons I don't fully understand yet (I'm still very new to Kubernetes) it looks like the game server runs on a Kubernetes node behind what is essentially a symmetric NAT. I've confirmed this by firing two STUN messages at different STUN servers from the same UDP socket - the binding responses have the same public IP but different ports. I'd love to be able to change this NAT behaviour but it may not be possible. It is possible for me to control port forwarding manually if I know ahead of time what UDP ports are required - this isn't too helpful though as each RTCDataChannel is going to be communicating on some random port. And the random port range is too large to forward everything.

Now I know I can work around this entirely with an external TURN server but that's going to add latency which is undesirable in a realtime multiplayer game. So I've been thinking about potential solutions.

I think I've dreamt up something that might work but I was hoping to run it by someone with a lot more WebRTC knowledge. Basically I'd implement my own TURN server from scratch that runs alongside the dedicated game server in the same container (added latency will thus be minimal). This TURN server will communicate with remote players on a single UDP socket with well defined port (so that I can ensure it's forwarded into the container) - messages coming in from the remote players will be demux'd based on their source address and then forwarded to the game server process that's running alongside it in the fashion WebRTC would expect from a TURN server. It'd function exactly like a real TURN server other than the single UDP socket + demuxing for relay messages to/from external players. This means I'd be able to tell the dedicated game server & the remote players about this TURN server and it should just work with the WebRTC machinery. This could be extended to TCP too but tbh I'm mostly interested in UDP and can live with lower connection success rate at the start if a connection over UDP can't be established.

I don't think out of the box TURN servers would work for me either due to my weird single UDP socket + demuxing requirement.

I've yet to really dig into the TURN spec but was hoping for some kind of nod that what I'm attempting isn't too bonkers 😅

Kind regards,

Deon

[dbotha]

Reading through the TURN RFC and I'm now thinking the below may be possible using an off the shelf TURN server rather than writing my own.

The game clients will act as the TURN Clients i.e. create the allocation on the TURN server, etc.

The Game Server will act as the peer with which the clients want to communicate. The Game Server itself wouldn't need to be a TURN client and if my understanding is correct not even need to know that the TURN server is a TURN server.

[dbotha]

Actually this might not work given the TURN Server & the Game Server share the same IP & would talk to eachother via localhost - I feel like any off the shelf TURN server would filter allocation requests for localhost peers...

[dbotha]

Nevermind - I can't believe it but it looks like it actually works using coturn 🎉 Sorry if you read all of the above...

[dbotha]

Out of interest does libdatachannel support listening on a single UDP/TCP port as per this answer: https://stackoverflow.com/a/68349888/406920 ?

[paullouisageneau]

@dbotha libdatachannel now supports multiplexing multiple WebRTC peer connections on a single UDP socket with the new enableIceUdpMux option 🎉 (see #555)

[dbotha]

Oh wow absolutely amazing, thanks @paullouisageneau !

[paullouisageneau]

Ha ha, it seems you got the answer to your own question!

Indeed, a TURN server can be deployed in the backend to allow exposing only a single UDP and/or TCP port to WebRTC clients. It's not the classic usage but it's something I've seen in production a couple times. Only the client requires the TURN server to be set up, the game server will communicate directly with the TURN server as it'll relay on the local network address. Additionally, you should force ICE relay on the client.

There is another way to achieve even shorter latency. libdatachannel does not support listening on a single port (mainly because It wouldn't follow the ICE protocol requirements). However, you can set a port range in the configuration and forward the port range on the NAT. In that case, instead of setting a STUN server, you can manually override emitted host candidates with the external address to optimize connection establishment.

[dbotha]

Thanks for the validation & helpful suggestion @paullouisageneau - I'll definitely consider that as another option. Although forwarding ranges (for reasons I don't yet understand) is not possible in Agones+Kubernetes. Although I could work around it by adding several individual forwarding rules to mimic ranges, but that's a bit cumbersome. I suspect routing through the TURN sever is going to add minimal latency in this case as it's always going to be running right next to the game server. If anyone else stumbles upon this discussion I'll be updating my StackOverflow question with an answer on the approach I took.

[paullouisageneau]

Sure, if you can't forward a port range, I think a TURN server is definitely the way to go.