[TRACKER] Networking improvements #13947
Comments
Most of the ideas seems cool :) |
|
@groud: For exposing ENet: It just seems a shame that multiple people had to work on their own Godot ENet implementation when it is already there, just not exposed sufficiently. E.g. https://github.com/PerduGames/gdnet3, https://github.com/Faless/godot-enet-better etc. Thanks for the feedback! |
|
Regarding 2: Would the best way be to implement a "heartbeat", a packet that gets periodically sent by each peer at a configurable interval to measure latency (and make that information then available on the peer/SceneTree)? |
|
I think "Support for movement interpolation & reconciliation (lag compensation, see article)" can be highly game-specific, and probably can't be implemented as-is, since the implementation methods may vary a lot depending on the game genre. Providing built-in support for Hermite interpolation could be a good idea, and it can be used in areas other than networking, see this article for more details. |
|
According to reduz, ENet channels are not supported on mobile. So if they can be exposed at all, this probably has to be taken into account. |
|
I'd like to help with launching multiple instances for server and clients. Does anyone have a good idea of how this should work? I can think of two solutions for this issue.
|
|
Opening some old wounds 😉 ...
@reduz sorry about bringing this up again, I was kinda dragged to it, but I really care about point 6 (and a to point 5 too 😉 ). |
|
@Faless Your patch for issue 6 is exactly what I was looking for. It's currently a huge pain point when developing games with dedicated servers. |
|
Some answers I guess
|
|
@reduz Just to clarify, can you explain what you mean by "implementation specific"? |
|
By the way, nothing prevents from adding extra functions to MultiplayerNetworkPeerENet, even if not in the base class. Something like set_send_channel() may work (not sure if there is any point on checking what channel the data has been received on, since RPC already handles this) |
|
you could do something like. |
|
@mhilbrunner Not sure if all APIs measure ping, latency, packet loss, etc. the same way. Discovery is also completely implementation specific. For IPV4 you can send packets to all your subnet to test, or broadcast. IPV6 is likely different, but you have already other network classes to do this, does not need to be part of the multiplayer API. |
|
@reduz The above idea is interesting. But couldn't it potentially cause weird behaviour/race conditions in case of multithreading? Just to compare to another engine, Unity has the following (the others are not applicable): #2 (Ping): https://docs.unity3d.com/ScriptReference/Network.GetAveragePing.html The story for UE4 likely looks similar; I know it can launch a configurable number of clients at "Play" and supports interpolation, at least. For now, I'll look into ping, discovery and the channels. I personally don't need NAT traversal currently :) If someone wants to chime in on any of these or help out, PRs welcome, of course :) As well as helpful pointers. ENet seems to already have support for average pings, which may not be optimal but could be a starting point. |
|
@mhilbrunner no, because the networking API is not meant to be used from multiple threads Regarding those functions, ping may make sense given you need it to work on both ends, which is why I suggested checking if ENet has something like this. But discovery, interpolation and host, we can provide a script in the Asset Library, given there isn't a single way to do it. |
ENet does something internally, but I've read around it's not to be used as a good latency metric. Addiontally, we might want to have it on other libraries too (e.g. Websocket, WebRTC). Given that we already have a small protocol for managing peers (see
@reduz , this is something that at the same time adds a new feature, makes debug easier, unclutters SceneTree, unclutters Object and most of all moves the Multiplayer related code in it's own class/classes where it belongs, I really can't understand why you are against it. I'd love to do it myself and show it to you if there's the smallest chance you would change your mind (pretty please 😉 ). |
|
Current API is meant to be simple, and I see no problem in adding a tool for launching (or even debugging) multiple instances to test. Making a worse API just for the hack of testing all scenes together does not seem good to me. Again, feel free to make a separate library via gdnative if you want something more specific. |
And that is not going to change. The idea is to keep the same API, with the small addition of also allowing a node to the same work that is now done by SceneTree. The API itself will not change, current demos will keep working as they do now with no change.
That is not going to be possible without huge and unmaintainable copy-paste from core files, unless we separate the Multiplayer protocol from the core. I guess I'll try to make a PR that adds no more than few lines of code and respect those two conditions (same API, separated protocol) to at least allow for a gdnative implementation without rewriting a lot of code, I'm confident that once you see it, you will like it, as again there will be no change to the API (nor the current SceneTree behaviour). |
|
Changes need to be made on a use case basis. If this work is only to make it easier to test multiplayer code, then It's definitely undesired. |
The many developers that kindly asked to allow for running both client and server in the same instance because their game requires it, or because it makes their code way better decoupled and not redundant. EDIT: |
|
The current design is not meant to run as client and server with two simultaneous instances. The server, as things are now, can run as a client without any difficulty and without being redundant. If debugging is the problem, then the debugger can be modified to support launching and debugging multiple instances at the same time (it should be pretty easy to do this). Again, what you are proposing is considerably more complex than what is in there and I don't really see any advantage. Supplying it externally should really not be a problem. |
|
If all your miss are the RPC functions, a wrapper could easily be done so they call your node instead of SceneTree, then you can handle them as you want. |
|
Regarding Quake III, this is really ancient multiplayer design. I'm not sure why you would want to go with that approach. Reminds me of Torque, which was completely unusable. |
EDIT: As I said at the beginning, it's an old wound, I don't want to argue about this, I'll see if I can at least unclutter SceneTree and Node to have the networking part separated in the code. We'll see then if it's worth merging or not, at this point I won't mind any wasted work |
|
Let me rephrase myself. I see zero advantage over the current approach, which is simpler and can perfectly act as server authoritative. Running both a client and a server in the same process only makes implementation more complex and provides nothing in exchange. Why should a considerably more bothersome approach need to be supported when the default one provided serves fine for large majority of use cases? |
Because it's not considerably more bothersome, because the current situation makes your life impossible in some (minority) of use cases and makes people that hates singletons like me cry 😭 . 😉 |
|
UPNP is known to be unreliable, it won't work for everybody out of the box since some router don't support it or comes with it disabled by default. Also, I asked for an alternative since I have a problem which seems to be related to the UPNP library used in Godot #25008 I'm kind of lost at this point and don't know what is the right approach to let someone host a game on Godot without having to do the port-forwarding himself. NB: I'm new to network |
Of course, because that was finally acknowledged as a security issue and now most routers come with it disabled.
You can't with 100% reliability, as you can't with any other piece of software, because there will always be some case where the router/network configuration does not allow it. |
That is basically why every game I know of tells you which port you need to open to host a match, so if you don't have working UPNP IGD you can manually open them. |
|
@Faless I understand that, though Warframe and Supreme Commander FAF both use UPNP and work without it. I'm asking for alternatives, what are these games solution to this problem? I see SupComFAF is talking about proxy but why would a server be magically allowed access on your router? I'm quite confused. (see the 'magic' part, that's where I'm lost ;) ) |
Use UPnP, if that fails, (optionally try NAT punch-through, if that fails) ask the user to manually open ports. SupComFAF in fact has a |
I see there's a sub-issue opened for adding documentation / tutorials around interpolation/reconciliation, but no issue opened around adding any engine-level support for this. Was this concluded to be a non-goal for the engine (and if so, why)? Regarding authority - it seems that the current approach is relying on clients to compute the physics and send the results to the server. I still dont fully understand how that would work and how potential conflicts between what multiple clients computed would be resolved. Furthermore, it seems very prone to cheaters / compromised clients. Is there an issue # to track adding support for relying fully on server to simulate all the physics and having client-side prediction? Thanks! |
I guess the discussion is still ongoing, the question is mainly if this can be done in a general way that is useful in common cases. What I've encountered with my RT game is that some interpolations depends on game mechanics (e.g. custom integration), so it's kinda hard to generalize. The way I do it is to set, in the clients, all the physics objects to static, disable the collisions, attach a Tween node that interpolate position/velocity/acceleration (sent by the server) but also apply the Works very smoothly in my case, but I'm not sure if this could be good as a general approach. I do not like reconciliation, reconciliation makes the game worse for people on good connections, and it might even help cheaters. It's something I try to avoid unless it's absolutely necessary.
That depends on the way you implement your network code. The multiplayer example in the demos is non-authoritative (i.e. peers send their own positions/calls and the others accepts them blindly). You can do an authoritative server by e.g. having the players only send input state to the server/host which applies them to the player nodes (and is master on all the other nodes too) You can, if you want, implement other techniques like consensus algorithms, turn-based strategies like in some RTS networking, etc. The decision is left to the game dev, as there's no fit for all solution. We'll be discussing networking/multiplayer improvements at the upcoming Godot events, and we'll update the roadmap in regard. As much as we understand the wishes for an easy-to-use system for some common networking cases, generalizing them is not an easy task, and sadly not always possible. |
|
#19110 talks about some potential man in the middle attacks, but it occurred to me that Godot multiplayer may also be vulnerable to spoofed RPCs (a peer sends an RPC that looks like it came from another peer). This is a more pressing issue because then an attacker doesn't need any special power other than knowing another user's IP address, and then they could do things like disconnect the other player or make them send in-game resources to the attacker. Knowing someone's IP address is plausible since people may both host and join games, and they need to publish their IP address to host. Protection against this kind of attack wouldn't need a full DTLS implementation (though I think a DTLS implementation would also protect against this). I think it would be enough to include an HMAC of each RPC, using a key (unique for each client) sent from the server to the client in plaintext at the beginning of the connection. Obviously this wouldn't protect against MITM attacks, but an attacker wouldn't be able to forge packets unless they can eavesdrop on the connection. |
There are checks for that.
And you need to be able to spoof your IP address which every legit ISP will not allow you to do, plus ENet has it's own extra session id, and you'll need to know a bunch of other parameters (like path caches, sequence numbers, etc). I don't think implementing hmac would make sense under those circumstances. |
|
This stack exchange post (https://security.stackexchange.com/questions/1062/why-dont-isps-filter-on-source-address-to-prevent-spoofing) suggests ISPs typically didn't filter spoofed packets in 2010. Do you have a source reporting which ISPs currently filter spoofed packets? The session ID in ENet is only 8 bits and the sequence number is 16 bits, so those could eventually be guessed. I am not sure whether they are initialized randomly or not, but if not, they could be guessed even more reliably. Do you happen to know if there is an official specification of ENet's protocol? I had trouble answering these questions to myself since I couldn't find one. I don't really understand Godot's plumbing that well, so I couldn't say whether path caches would be sufficiently unguessable. |
|
As stated in a more recent answer to that question an increasing number of
ISP have realized the impact of ip spoofing in terms of costs, and are thus
implementing anti-spoofing techniques.
Additionally, knowing the IP of the server is not really enough, you would
also need the IP of the client. Not only that, you would also need to know
the randomly assigned UDP port of the client (if we are talking about enet).
That said, if there's any proof of concept that works by just knowing the
server IP/port let me know.
About enet, the only documentation I know of is here:
http://enet.bespin.org/ , the rest I gathered from the source code.
…On Wed, Jun 5, 2019, 18:48 raymoo ***@***.***> wrote:
This stack exchange post (
https://security.stackexchange.com/questions/1062/why-dont-isps-filter-on-source-address-to-prevent-spoofing)
suggests ISPs typically didn't filter spoofed packets in 2010. Do you have
a source reporting which ISPs currently spoof packets?
The session ID in ENet is only 8 bits and the sequence number is 16 bits,
so those could eventually be guessed. I am not sure whether they are
initialized randomly or not, but if not, they could be guessed even more
reliably. Do you happen to know if there is an official specification of
ENet's protocol? I had trouble answering these questions to myself since I
couldn't find one.
I don't really understand Godot's plumbing that well, so I couldn't say
whether path caches would be sufficiently unguessable.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#13947?email_source=notifications&email_token=AAM4C3RZXHJE5D64PMKFPBTPY7U6TA5CNFSM4EGQQ3K2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXAKMPA#issuecomment-499164732>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAM4C3T77WQAVNDT4BMMHILPY7U6TANCNFSM4EGQQ3KQ>
.
|
An increasing number is not the same as "every legit ISP". Without specific figures it could mean 10% were before, and 25% are now.
When I said an attacker knows the IP address of another user, I meant another client. In the same post I give a reason why it's plausible that an unprivileged attacker would know the IP address of another player.
The port number is 16-bit, which could be brute forced. However, the combination of port number + sequence number would be more difficult to brute force if both the port number and initial sequence number are assigned randomly. From Wikipedia I know that Linux has a port number range of around 16 bits, and Windows around 15 bits. I think on at least Linux it is assigned in some random way and probably can't be predicted easily unless the attacker can observe the other client's connection status. I don't know if ENet initializes its initial sequence number randomly, but if it does then I think the attacks I was worried about would take too long to be useful in a game.
I was hoping there was a specification of the protocol independent from the implementation, but thank you for checking. |
|
An increasing number is not the same as "every legit ISP". Without
specific figures it could mean 10% were before, and 25% are now.
A non-complete list:
https://www.manrs.org/isps/participants/
Feel free to go over it, don't expect ISP names there, but bigger
consortiums, e.g. for Italy MiX includes all ISPs I know and some I've
never heard of.
When I said an attacker knows the IP address of another user, I meant
another client. In the same post I give a reason why it's plausible that an
unprivileged attacker would know the IP address of another player.
You mean the host+join scenario? Well, in that case, the join part is most
likely on loopback, which is pretty hard to spoof.
I don't know if ENet initializes its initial sequence number randomly, but
if it does then I think the attacks I was worried about would take too long
to be useful in a game.
Now that I better think of it, sequence number is only added to reliable
and unreliable-ordered packets, so we can't really count on that.
I would still prefer to be sure that this is actually exploitable in a real
case scenario before adding more code, but in any case, if we are going to
implement it, it should be limited to the ENet peer, not affecting the
whole MultiplayerAPI.
|
Thanks. This doesn't really give a proportion, but I do recognize some large US ISPs when I searched.
I mean where a player sometimes hosts a game for other people to join, and sometimes joins other people's games. If they advertise their games in (for example) a Discord server, other players may know their IP address from previous interactions.
Does this mean it wouldn't affect RPC calls? Or do you just mean that the code changes should be localized to the ENet peer? |
|
I mean where a player sometimes hosts a game for other people to join, and
sometimes joins other people's games. If they advertise their games in (for
example) a Discord server, other players may know their IP address from
previous interactions.
It seems to me it's a case that requires some quite special conditions and
a lot of effort from the attacker, with limited chance of success, and
little to gain.
I wouldn't worry too much about it, as we are talking about games here, not
mission critical systems.
Does this mean it wouldn't affect RPC calls? Or do you just mean that the
code changes should be localized to the ENet peer?
That it should be localized to the ENet peer and made optional. Other
multiplayer peers (websocket and upcoming webrtc) don't need it, making
that a waste of precious bytes and computational power.
… |
It's a pretty common situation for some types of games, such as fighting games without matchmaking, where people typically post their IP address and invite people to join.
It only requires the effort once to be reused by the attacker, and maybe other attackers who obtain the exploiting program from the original attacker.
If there's a command that doesn't require a sequence number, and is a valid command at any point in the game (such as a client notifying disconnection), then it could be (I am guessing) exploited reliably.
I don't think this is true, at least for moderately-successful games. Some potential rewards (using the example of disconnecting a player):
Sounds good to me. |
|
If there's a command that doesn't require a sequence number, and is a
valid command at any point in the game (such as a client notifying
disconnection), then it could be (I am guessing) exploited reliably.
Well, I tried to explain you why this is not a trivial thing to do
(filtering, sequencing, session IDs, ecc), but since you prefer making
assumption and guesses that it's easy to do, there's clearly no point in
arguing, as you clearly made your final decision already.
Sounds good to me.
While I still don't think it's going to be much useful, feel free to open a
PR, I'll make sure to review it.
|
I thought you said that not all packets used by Godot had sequence numbers? And session IDs are only a byte and probably not random. |
|
But you assumed that no sequence was used for "disconnect" messages, and
you are now assuming that sessionId is not random
…On Tue, Jun 11, 2019, 15:50 raymoo ***@***.***> wrote:
(filtering, sequencing, session IDs, ecc), but since you prefer making
assumption and guesses that it's easy to do, there's clearly no point in
arguing, as you clearly made your final decision already.
I thought you said that not all packets used by Godot had sequence
numbers? And session IDs are only a byte and probably not random.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#13947?email_source=notifications&email_token=AAM4C3V66PNWILGEL2WNOO3PZ63SXA5CNFSM4EGQQ3K2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXNMJII#issuecomment-500876449>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAM4C3S45SQI776365B75ALPZ63SXANCNFSM4EGQQ3KQ>
.
|
|
It'd be cool to have an encryption switch for the high-level API which would solve 2 problems at once. But I'm not using it, so don't expect a PR from me :] |
|
It'd be cool to have an encryption switch for the high-level API which
would solve 2 problems at once. But I'm not using it, so don't expect a PR
from me :]
Yeah, encryption would be much more useful for a general case, and I'm
planning to investigate that (again, at ENet level) once I implement DTLS
support in Godot.
… —
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#13947?email_source=notifications&email_token=AAM4C3UTTGDKJYCSKWS3JQLPZ7HAXA5CNFSM4EGQQ3K2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXNWYTI#issuecomment-500919373>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAM4C3SJFCWOGCAZNNAAN43PZ7HAXANCNFSM4EGQQ3KQ>
.
|
ENet itself allows receiving the ENet disconnect packet as unsequenced. It looks like it has some sort of check (https://github.com/lsalzman/enet/blob/0eaf48eeb0d94a18d079378d8b76d588832ce838/protocol.c#L478) in unsequenced packets, but seems like it would accept a large range of values, so is probably meant for some other purpose than security. In particular it looks like if the command had unsequencedGroup set to 0, it would always be accepted (I could be wrong).
I looked at the code for determining session IDs (https://github.com/lsalzman/enet/blob/0eaf48eeb0d94a18d079378d8b76d588832ce838/protocol.c#L340, https://github.com/lsalzman/enet/blob/0eaf48eeb0d94a18d079378d8b76d588832ce838/protocol.c#L340) and I believe it starts at zero for each peer and counts up with extra connections. I'm not sure, but in any case it looks pretty deterministic. |
|
The only realistic and well-known vulnerability in the ENet is the Blind Masquerade, which is explained in RFC 4960. We solved this by using four-way handshake and cookies. A similar attempt to solve this was done in this fork and further commits there, but no idea if his implementation works in practice. For everything else we use encryption based on the Noise protocol. |
|
In case someone is interested, this service provide spoofing tests and stats: |
|
These graphs measure how many client addresses are capable of spoofing as a different source address. The portion is low but significant (a bit smaller than a quarter), so an attacker whose ISP performs filtering can look for a VPS in a region or behind an ISP that doesn't filter source addresses. Thanks for finding these statistics. I referenced the abstract for the paper for that project for this post: https://www.usenix.org/legacy/events/sruti05/tech/beverly.html |
I believe this should be in an API for |
|
The second issue was closed(Method to get networking stats): #8763 (comment) |
|
Note: For any items not having a proposal open yet, please open one proposal per feature (don't lump features together). Feature and improvement proposals for the Godot Engine are now being discussed and reviewed in a dedicated Godot Improvement Proposals (GIP) (godotengine/godot-proposals) issue tracker. The GIP tracker has a detailed issue template designed so that proposals include all the relevant information to start a productive discussion and help the community assess the validity of the proposal for the engine. The main (godotengine/godot) tracker is now solely dedicated to bug reports and Pull Requests, enabling contributors to have a better focus on bug fixing work. Therefore, we are now closing all older feature proposals on the main issue tracker. If you are interested in this feature proposal, please open a new proposal on the GIP tracker following the given issue template (after checking that it doesn't exist already). Be sure to reference this closed issue if it includes any relevant discussion (which you are also encouraged to summarize in the new proposal). Thanks in advance! |
This issue is to collect and discuss improvements to the networking in Godot 3.
Methods on(likely not possible)IPto get loopback, link local, or external addresses instead of all of them as an array (withget_local_addresses())Lobby / LAN host autodiscovery using broadcasting(possible through broadcasting usingPacketPeerUDP, fixed here: Set SO_BROADCAST socket option for UDP sockets. #19378)The text was updated successfully, but these errors were encountered: