sctp over udp encapsulation does not work under NAT network

[gidcs]

I am trying usrsctp right now.
As far as I understand, usrsctp should work fine even client is behind NAT when udp encapsulation is using.
However, I found that demo programs does not work when client is behind NAT.
(Note that: WebRTC is working fine and discard_server is not reside in the LAN.)
The 4-way handshake always stuck at the second step and the client will keep retry and retry.
Is that a normal case for the demo programs?

The log of client and discard_server is shown as below:

$ ./client SERVER_IP 9 0 22222 11111
##show nothing

  $ ./discard_server 11111 22222
  [0.000] SCTP: add HMAC id 1 to list
  [0.000] SCTP: added chunk 193 (0xc1) to Auth list
  [0.000] SCTP: added chunk 128 (0x80) to Auth list
  [0.000] Bind called port: 9
  [0.000] Addr: [0.000] IPv6 address: 0:0:0:0:0:0:0:0:port:9 scope:0
  [0.000] Main hash to bind at head:0x557afedd8a70, bound port:9 - in tcp_pool=0
  [22.287] recv_function_udp: Received 112 bytes.[22.287]  - calling sctp_common_input_processing with off=12
  [22.287] stcb:(nil) inp:0x557afeddd120
  [22.287] stcb is (nil)
  [22.287] Ok, Common input processing called, m:0x7f5a78000c30 iphlen:0 offset:12 length:112 stcb:(nil)
  [22.287] sctp_process_control: iphlen=0, offset=12, length=112 stcb:(nil)
  [22.287] Its an INIT of len:100 vtag:0
  [22.287] sctp_process_control: processing a chunk type=1, len=100
  [22.287] SCTP_INIT
  [22.287] sctp_handle_init: handling INIT tcb:(nil)
  [22.287] sctp_handle_init: sending INIT-ACK
  [22.287] Check for unrecognized param's
  [22.288] Hit default param 8004
  [22.288] move on
  [22.288] Select source addr for:[22.288] IPv4 address: NAT_IP:52809
  [22.288] ifn from route:(nil) ifn_index:1
  [22.288] ifn_index:1 name:lo is emit interface
  [22.288] Is destination preferred:[22.288] IPv4 address: 127.0.0.1:0
  [22.288] src_loop:1 src_priv:0 src_glob:0
  [22.288] dest_loop:0 dest_priv:0 dest_glob:1
  [22.288] NO:6
  [22.288] Found 0 preferred source addresses for intf:lo
  [22.288] Trying Plan B
  [22.288] Examine interface enp3s0f0
  [22.288] Is destination preferred:[22.288] IPv4 address: SERVER_IP:0
  [22.288] src_loop:0 src_priv:0 src_glob:1
  [22.288] dest_loop:0 dest_priv:0 dest_glob:1
  [22.288] YES
  [22.288] Found ifn:(nil) 1 preferred source addresses
  [22.288] num preferred:1 on interface:0x557afeddc5b0 cur_addr_num:0
  [22.288] Is destination preferred:[22.288] IPv4 address: SERVER_IP:0
  [22.288] src_loop:0 src_priv:0 src_glob:1
  [22.288] dest_loop:0 dest_priv:0 dest_glob:1
  [22.288] YES
  [22.288] Calling ipv4 output routine from low level src addr:AAAAAAAA
  [22.288] Destination is BBBBBBBB
  [22.288] RTP route is 0x7f5a780142c0 through
  [22.288] IP output returns 0
  [25.323] recv_function_udp: Received 112 bytes.[25.324]  - calling sctp_common_input_processing with off=12
  [25.324] stcb:(nil) inp:0x557afeddd120
  [25.324] stcb is (nil)
  [25.324] Ok, Common input processing called, m:0x7f5a78000c30 iphlen:0 offset:12 length:112 stcb:(nil)
  [25.324] sctp_process_control: iphlen=0, offset=12, length=112 stcb:(nil)
  [25.324] Its an INIT of len:100 vtag:0
  [25.324] sctp_process_control: processing a chunk type=1, len=100
  [25.324] SCTP_INIT
  [25.324] sctp_handle_init: handling INIT tcb:(nil)
  [25.324] sctp_handle_init: sending INIT-ACK
  [25.324] Check for unrecognized param's
  [25.324] Hit default param 8004
  [25.324] move on
  [25.324] Select source addr for:[25.324] IPv4 address: NAT_IP:52809
  [25.324] ifn from route:(nil) ifn_index:1
  [25.324] ifn_index:1 name:lo is emit interface
  [25.324] Is destination preferred:[25.324] IPv4 address: 127.0.0.1:0
  [25.324] src_loop:1 src_priv:0 src_glob:0
  [25.324] dest_loop:0 dest_priv:0 dest_glob:1
  [25.324] NO:6
  [25.324] Found 0 preferred source addresses for intf:lo
  [25.324] Trying Plan B
  [25.324] Examine interface enp3s0f0
  [25.324] Is destination preferred:[25.324] IPv4 address: SERVER_IP:0
  [25.324] src_loop:0 src_priv:0 src_glob:1
  [25.324] dest_loop:0 dest_priv:0 dest_glob:1
  [25.324] YES
  [25.324] Found ifn:(nil) 1 preferred source addresses
  [25.324] num preferred:1 on interface:0x557afeddc5b0 cur_addr_num:0
  [25.324] Is destination preferred:[25.324] IPv4 address: SERVER_IP:0
  [25.324] src_loop:0 src_priv:0 src_glob:1
  [25.324] dest_loop:0 dest_priv:0 dest_glob:1
  [25.324] YES
  [25.324] Calling ipv4 output routine from low level src addr:AAAAAAAA
  [25.324] Destination is BBBBBBBB
  [25.324] RTP route is 0x7f5a780142c0 through
  [25.324] IP output returns 0
  [31.398] recv_function_udp: Received 112 bytes.[31.398]  - calling sctp_common_input_processing with off=12
  [31.398] stcb:(nil) inp:0x557afeddd120
  [31.398] stcb is (nil)
  [31.398] Ok, Common input processing called, m:0x7f5a78000c30 iphlen:0 offset:12 length:112 stcb:(nil)
  [31.398] sctp_process_control: iphlen=0, offset=12, length=112 stcb:(nil)
  [31.398] Its an INIT of len:100 vtag:0
  [31.398] sctp_process_control: processing a chunk type=1, len=100
  [31.398] SCTP_INIT
  [31.398] sctp_handle_init: handling INIT tcb:(nil)
  [31.398] sctp_handle_init: sending INIT-ACK
  [31.398] Check for unrecognized param's
  [31.398] Hit default param 8004
  [31.398] move on
  [31.398] Select source addr for:[31.398] IPv4 address: NAT_IP:52809
  [31.398] ifn from route:(nil) ifn_index:1
  [31.398] ifn_index:1 name:lo is emit interface
  [31.398] Is destination preferred:[31.398] IPv4 address: 127.0.0.1:0
  [31.398] src_loop:1 src_priv:0 src_glob:0
  [31.398] dest_loop:0 dest_priv:0 dest_glob:1
  [31.398] NO:6
  [31.398] Found 0 preferred source addresses for intf:lo
  [31.398] Trying Plan B
  [31.398] Examine interface enp3s0f0
  [31.398] Is destination preferred:[31.398] IPv4 address: SERVER_IP:0
  [31.398] src_loop:0 src_priv:0 src_glob:1
  [31.398] dest_loop:0 dest_priv:0 dest_glob:1
  [31.398] YES
  [31.398] Found ifn:(nil) 1 preferred source addresses
  [31.398] num preferred:1 on interface:0x557afeddc5b0 cur_addr_num:0
  [31.398] Is destination preferred:[31.398] IPv4 address: SERVER_IP:0
  [31.398] src_loop:0 src_priv:0 src_glob:1
  [31.398] dest_loop:0 dest_priv:0 dest_glob:1
  [31.398] YES
  [31.398] Calling ipv4 output routine from low level src addr:AAAAAAAA
  [31.398] Destination is BBBBBBBB
  [31.398] RTP route is 0x7f5a780142c0 through
  [31.398] IP output returns 0
  [43.545] recv_function_udp: Received 112 bytes.[43.545]  - calling sctp_common_input_processing with off=12
  [43.545] stcb:(nil) inp:0x557afeddd120
  [43.545] stcb is (nil)
  [43.545] Ok, Common input processing called, m:0x7f5a78000c30 iphlen:0 offset:12 length:112 stcb:(nil)
  [43.545] sctp_process_control: iphlen=0, offset=12, length=112 stcb:(nil)
  [43.545] Its an INIT of len:100 vtag:0
  [43.545] sctp_process_control: processing a chunk type=1, len=100
  [43.545] SCTP_INIT
  [43.545] sctp_handle_init: handling INIT tcb:(nil)
  [43.545] sctp_handle_init: sending INIT-ACK
  [43.545] Check for unrecognized param's
  [43.545] Hit default param 8004
  [43.545] move on
  [43.546] Select source addr for:[43.546] IPv4 address: NAT_IP:52809
  [43.546] ifn from route:(nil) ifn_index:1
  [43.546] ifn_index:1 name:lo is emit interface
  [43.546] Is destination preferred:[43.546] IPv4 address: 127.0.0.1:0
  [43.546] src_loop:1 src_priv:0 src_glob:0
  [43.546] dest_loop:0 dest_priv:0 dest_glob:1
  [43.546] NO:6
  [43.546] Found 0 preferred source addresses for intf:lo
  [43.546] Trying Plan B
  [43.546] Examine interface enp3s0f0
  [43.546] Is destination preferred:[43.546] IPv4 address: SERVER_IP:0
  [43.546] src_loop:0 src_priv:0 src_glob:1
  [43.546] dest_loop:0 dest_priv:0 dest_glob:1
  [43.546] YES
  [43.546] Found ifn:(nil) 1 preferred source addresses
  [43.546] num preferred:1 on interface:0x557afeddc5b0 cur_addr_num:0
  [43.546] Is destination preferred:[43.546] IPv4 address: SERVER_IP:0
  [43.546] src_loop:0 src_priv:0 src_glob:1
  [43.546] dest_loop:0 dest_priv:0 dest_glob:1
  [43.546] YES
  [43.546] Calling ipv4 output routine from low level src addr:AAAAAAAA
  [43.546] Destination is BBBBBBBB
  [43.546] RTP route is 0x7f5a780142c0 through
  [43.546] IP output returns 0

[gidcs]

I have go through it and found that my main interface was added into the asoc.sctp_restricted_addrs and this is why the four-way handshake always stuck. However, if I deactivate the virbr0 in my client computer, the demo program works without any issue. It is a quite strange behavior.

[tuexen]

Can you figure out why the addresses where added?

[nxrighthere]

In my case, a connection cannot be established even if I create address/port translation rule directly in the router. Tried three different machines with Windows 10 and different routers, no luck.

Debug logs

SCTP: add HMAC id 1 to list
Bind called port: 50704
Addr: IPv6 address: 0:0:0:0:0:0:0:0:port:50704 scope:0
Main hash to bind at head:0000000004928630, bound port:50704 - in tcp_pool=0
Allocate an association for peer:IPv4 address: 2.93.125.139:9500
Port:9500
Adding an address (from:1) to the peer: IPv4 address: 2.93.125.139:9500
Association 000000000C0B1070 now allocated
Sending INIT
Sending INIT - calls lowlevel_output
Select source addr for:IPv4 address: 2.93.125.139:9500
ifn from route:0000000000000000 ifn_index:1
No ifn emit interface?
Trying Plan B
Examine interface {8B0878A2-ABFC-4045-AC13-B242D16856CE}
Is destination preferred:IPv4 address: 192.168.1.2:0
src_loop:0 src_priv:1 src_glob:0
dest_loop:0 dest_priv:0 dest_glob:1
NO:7
Found ifn:0000000000000000 0 preferred source addresses
No preferred -- skipping to next
Trying Plan C: find acceptable on interface
Jump to Plan D - no emit_ifn
Trying Plan D looked_at is 0000000000000000
IPv4 address: 192.168.1.2:0
dst_is_loop:0 dest_is_priv:0
ifa->src_is_loop:0 dest_is_priv:0
ifa->src_is_loop:0 dest_is_glob:1
address is acceptable
Trying Plan C: find acceptable on interface
Jump to Plan D - no emit_ifn
Trying Plan D looked_at is 0000000000000000
IPv4 address: 192.168.1.2:0
dst_is_loop:0 dest_is_priv:0
ifa->src_is_loop:0 dest_is_priv:0
ifa->src_is_loop:0 dest_is_glob:1
address is acceptable
IPv4 address: 192.168.1.2:0
dst_is_loop:0 dest_is_priv:0
ifa->src_is_loop:0 dest_is_priv:0
ifa->src_is_loop:0 dest_is_glob:1
address is acceptable
Calling ipv4 output routine from low level src addr:c0a80102
Destination is 25d7d8b
RTP route is 0000000004945C10 through
IP output returns 0
Timer type 2 goes off
Error count for 000000000C0B1BF0 now 1 thresh:5
Overall error count for 000000000C0B10C8 now 1 thresh:8 state:1
Sending INIT
Sending INIT - calls lowlevel_output
Calling ipv4 output routine from low level src addr:c0a80102
Destination is 25d7d8b
RTP route is 0000000004945C10 through
IP output returns 0
Timer now complete (type = 2)
Timer type 2 goes off
Error count for 000000000C0B1BF0 now 2 thresh:5
Overall error count for 000000000C0B10C8 now 2 thresh:8 state:1
Sending INIT
Sending INIT - calls lowlevel_output
Calling ipv4 output routine from low level src addr:c0a80102
Destination is 25d7d8b
RTP route is 0000000004945C10 through
IP output returns 0
Timer now complete (type = 2)
Timer type 2 goes off
Error count for 000000000C0B1BF0 now 3 thresh:5
Overall error count for 000000000C0B10C8 now 3 thresh:8 state:1
Sending INIT
Sending INIT - calls lowlevel_output
Calling ipv4 output routine from low level src addr:c0a80102
Destination is 25d7d8b
RTP route is 0000000004945C10 through
IP output returns 0
Timer now complete (type = 2)
Timer type 2 goes off
Error count for 000000000C0B1BF0 now 4 thresh:5
Overall error count for 000000000C0B10C8 now 4 thresh:8 state:1
Sending INIT
Sending INIT - calls lowlevel_output
Calling ipv4 output routine from low level src addr:c0a80102
Destination is 25d7d8b
RTP route is 0000000004945C10 through
IP output returns 0
Timer now complete (type = 2)
Timer type 2 goes off
Error count for 000000000C0B1BF0 now 5 thresh:5
Overall error count for 000000000C0B10C8 now 5 thresh:8 state:1
Sending INIT
Sending INIT - calls lowlevel_output
Calling ipv4 output routine from low level src addr:c0a80102
Destination is 25d7d8b
RTP route is 0000000004945C10 through
IP output returns 0
Timer now complete (type = 2)
Timer type 2 goes off
Error count for 000000000C0B1BF0 now 6 thresh:5
Overall error count for 000000000C0B10C8 now 6 thresh:8 state:0
Sending INIT
Sending INIT - calls lowlevel_output
Calling ipv4 output routine from low level src addr:c0a80102
Destination is 25d7d8b
RTP route is 0000000004945C10 through
IP output returns 0
Timer now complete (type = 2)
Timer type 2 goes off
Error count for 000000000C0B1BF0 now 7 thresh:5
Overall error count for 000000000C0B10C8 now 7 thresh:8 state:0
Sending INIT
Sending INIT - calls lowlevel_output
Calling ipv4 output routine from low level src addr:c0a80102
Destination is 25d7d8b
RTP route is 0000000004945C10 through
IP output returns 0
Timer now complete (type = 2)

[tuexen]

Which addresses do you have configured on which interface on the machine behind the NAT. It seems that it owns 192.168.1.2. Do you see packets going out that machine? If yes, are you using UDP encapsulation and which SCTP and UDP port numbers are you using? Do you see packets being received by the external host owning 2.93.125.139?

[nxrighthere]

Which addresses do you have configured on which interface on the machine behind the NAT. It seems that it owns 192.168.1.2.

Yes, the machine behind the NAT owns 192.168.1.2.

Do you see packets going out that machine?

Yes, SCTP is repeatedly sending INIT until ABORT packet.

If yes, are you using UDP encapsulation and which SCTP and UDP port numbers are you using?

Server and client are encapsulated using 9925 port in usrsctp_init(), and this port number is used for usrsctp_setsockopt() w/ SCTP_REMOTE_UDP_ENCAPS_PORT, client is bound to in6addr_any with randomly generated port, server is listening on all interfaces and 9500 port which is used for usrsctp_connect() on the client side.

Do you see packets being received by the external host owning 2.93.125.139?

Nope, it seems that it doesn't receive anything.

[tuexen]

But you see an SCTP packet some random source port to the destination port number 9500 encapsulated in a UDP packet using 9925 as the source and destination port number encapsulated in IPv4 using 192.168.1.2 as the source address and 2.93.125.139 as the destination address, right? If that is true and if the checksums (IPv4 header, UDP, SCTP) are correct, the problem is not with the SCTP implementation on the client side. I would try to capture on the uplink of the NAT box to see if it really sends the packet upstream. If that is the case, are you sure that your external host (the one owning 2.93.125.139) is protected by any kind of firewall? I just tried to initiate an association with that server, but I don't get any response (no ICMP (Destination unreachable, port unreachable) or SCTP ABORT or SCTP INIT-ACK)...

[nxrighthere]

But you see an SCTP packet some random source port to the destination port number 9500 encapsulated in a UDP packet

Yes.

using 9925 as the source and destination port number encapsulated in IPv4 using 192.168.1.2 as the source address and 2.93.125.139 as the destination address, right

Nope, the source port is random as well while the destination is 9925. In the local network everything works fine, the problem occurs only when an external IP address is used with a device behind NAT. To make sure that the problem is not with the router I just checked another UDP transport which works fine using the same port and translation rule.

I just tried to initiate an association with that server, but I don't get any response (no ICMP (Destination unreachable, port unreachable) or SCTP ABORT or SCTP INIT-ACK)...

The server was down, but I can keep it running for a while if you want.

[nxrighthere]

I just realized that 9925 port number for NAT translation rule should be used instead of 9500, only then the client is able to establish a connection.

So manual address translation helps, but SCTP itself doesn't map addresses (UPNP/PMP)? It seems that I misunderstood the NAT friendliness...

[tuexen]

I just realized that 9925 port number for NAT translation rule should be used instead of 9500, only then the client is able to establish a connection.

So manual address translation helps, but SCTP itself doesn't map addresses (UPNP/PMP)?

Outgoing SCTP/UDP/IP packets should use the port number provided in the usrsctp_init() call as the UDP source port number. If that is not happening, then it is a bug. Are you seeing a different behaviour?

Behind a NAT box, an SCTP client should be able to initiate a SCTP/UDP based communication with a host in the Internet without any additional configuration on the NAT box. The SCTP does not interact with middle-boxes using UPNP/PMP. Why do you think this is necessary?

[nxrighthere]

Outgoing SCTP/UDP/IP packets should use the port number provided in the usrsctp_init() call as the UDP source port number. If that is not happening, then it is a bug. Are you seeing a different behaviour?

Everything is correct, yes, I was a bit confused...

Behind a NAT box, an SCTP client should be able to initiate a SCTP/UDP based communication with a host in the Internet without any additional configuration on the NAT box.

Yes, it works as expected, for the client no external steps are required. I have a different problem actually which directly related to:

The SCTP does not interact with middle-boxes using UPNP/PMP. Why do you think this is necessary?

My case is a quite common thing in multiplayer games where a user is hosting a game server which is listening for incoming connections behind NAT and this requires to manually (or by using UPNP/PMP) set translation rule to the address behind NAT. Some UDP transport layers provide this feature out of the box due to the popular demand (see for example) and this one is a popular implementation.

[tuexen]

Outgoing SCTP/UDP/IP packets should use the port number provided in the usrsctp_init() call as the UDP source port number. If that is not happening, then it is a bug. Are you seeing a different behaviour?

Everything is correct, yes, I was a bit confused...

OK, great. Thanks for the clarification.

Behind a NAT box, an SCTP client should be able to initiate a SCTP/UDP based communication with a host in the Internet without any additional configuration on the NAT box.

Yes, it works as expected, for the client no external steps are required. I have a different problem actually which directly related to:

The SCTP does not interact with middle-boxes using UPNP/PMP. Why do you think this is necessary?

My case is a quite common thing in multiplayer games where a user is hosting a game server which is listening for incoming connections behind NAT and this requires to manually (or by using UPNP/PMP) set translation rule to the address behind NAT. Some UDP transport layers provide this feature out of the box due to the popular demand (see for example) and this one is one of the most popular implementations.

Understood. But usrsctp is just an SCTP implementation. You can combine it with NAT traversal tools if you want. This is what WebRTC does, for example, by combining it that STUN/TURN/ICE. But this is not part of any SCTP specification...

[nxrighthere]

Yup, I'll stick with a sort of signaling server as well I think, just was a bit confused with NAT-friendly mode. 👌

[tuexen]

If there is still an issue, please re-open.