Question for ip-mptcp command

[VenkateswaranJ]

I have a server-client connected with two interfaces.

-----client---------------------server---
10.10.10.2 <----------> 10.10.10.3
20.20.20.2 <----------> 20.20.20.3

client side mptcp config

$ ip mptcp endpoint show
20.20.20.2 id 3 fullmesh dev enp0s9 
$ ip mptcp limits show
add_addr_accepted 2 subflows 2

server side mptcp config

$ ip mptcp endpoint show
20.20.20.3 id 3 fullmesh dev enp0s9 
$ ip mptcp limits show
add_addr_accepted 0 subflows 2

When I connect the client to the server it creates 2 subflows (one is the initial subflow)

Question:
I delete the endpoint from the client side.

sudo ip mptcp endpoint delete id 3

This removes the subflow from client and server (remove address command)

Now I added the endpoint again to the client.

sudo ip mptcp endpoint add 20.20.20.2 dev enp0s9 fullmesh

But the server won't signal its second interface address 20.20.20.3 to create a new subflow. Is it the correct behaviour?

If I understand correctly, the server announces its address only once when the MPTCP connection is established and will not announce it again. Is that correct?

Here is the output from ip mptcp monitor

$ sudo ip mptcp monitor
[         CREATED] token=917c83e8 remid=0 locid=0 saddr4=10.10.10.3 daddr4=10.10.10.2 sport=5000 dport=47930
[     ESTABLISHED] token=917c83e8 remid=0 locid=0 saddr4=10.10.10.3 daddr4=10.10.10.2 sport=5000 dport=47930
[  SF_ESTABLISHED] token=917c83e8 remid=1 locid=1 saddr4=20.20.20.3 daddr4=20.20.20.2 sport=5000 dport=57783 backup=0
[         REMOVED] token=917c83e8 remid=1
[       SF_CLOSED] token=917c83e8 remid=1 locid=1 saddr4=20.20.20.3 daddr4=20.20.20.2 sport=5000 dport=57783 backup=0

As you can see, after the SF_CLOSED there is no SF_ESTABLISHED

Also, I wonder why the ss command only shows the initial subflow. It never shows the second subflow.

$ ss -M
State               Recv-Q               Send-Q                               Local Address:Port                               Peer Address:Port                
ESTAB               0                    0                                       10.10.10.3:5000                                 10.10.10.2:47930               

[matttbe]

Hello,

I have a server-client connected with two interfaces.

Just to know what you expect, can each address reaches any address of the other peer? e.g. 10.10.10.2 <-> 20.20.20.3?

Or here you can only have max 2 subflows in this setup?

client side mptcp config

$ ip mptcp endpoint show
20.20.20.2 id 3 fullmesh dev enp0s9 
$ ip mptcp limits show
add_addr_accepted 2 subflows 2

Mmh, fullmesh alone, without subflow in this case (on the endpoints used for the additional subflows) doesn't make sense, it means:

• when looking at creating new subflows, the endpoint will not be picked (because subflow is not set)
• when an ADD_ADDR is received, only the endpoints with the fullmesh will be taken (if there is at least one, otherwise the source address is picked using the routing table).

So here, I would expect the kernel not to create any additional subflows, except if the server sends an ADD_ADDR (which is not the case, when looking below).

FYI, the fullmesh flag can make sense to be used alone (without subflow) if you want the address used by the initial subflow to create a subflow to each address received via ADD_ADDR from the other peer.

Maybe this update of the website could help better understanding how to use the PM works?
multipath-tcp/mptcp.dev#12
Do not hesitate to suggest modifications!

server side mptcp config

$ ip mptcp endpoint show
20.20.20.3 id 3 fullmesh dev enp0s9 
$ ip mptcp limits show
add_addr_accepted 0 subflows 2

fullmesh doesn't make sense here when used alone: nothing to create, and typically it is the server that is sending ADD_ADDR not the one receiving them. If you want the server to announce the endpoint, you need to use the signal flag.

The fullmesh flag is there to create more subflows, not to announce more addresses.

When I connect the client to the server it creates 2 subflows (one is the initial subflow)

To be honest, I'm not sure how you ended up with 2 subflows with what you described: even in the 'monitor' output, I don't see any received ANNOUNCED (or is it taken from the server side), so how can it know it can establish a subflow from 20.20.20.3 to 20.20.20.2?

Question: I delete the endpoint from the client side.

sudo ip mptcp endpoint delete id 3

This removes the subflow from client and server (remove address command)

Now I added the endpoint again to the client.

sudo ip mptcp endpoint add 20.20.20.2 dev enp0s9 fullmesh

But the server won't signal its second interface address 20.20.20.3 to create a new subflow. Is it the correct behaviour?

Was it even announcing the endpoint before? Can you try with the signal flag instead?

If I understand correctly, the server announces its address only once when the MPTCP connection is established and will not announce it again. Is that correct?

When a RM_ADDR, it should make the ID as "available" again, then when re-added, it should send the ADD_ADDR again I think. Can you try with the signal flag please?

Here is the output from ip mptcp monitor

$ sudo ip mptcp monitor
[         CREATED] token=917c83e8 remid=0 locid=0 saddr4=10.10.10.3 daddr4=10.10.10.2 sport=5000 dport=47930
[     ESTABLISHED] token=917c83e8 remid=0 locid=0 saddr4=10.10.10.3 daddr4=10.10.10.2 sport=5000 dport=47930
[  SF_ESTABLISHED] token=917c83e8 remid=1 locid=1 saddr4=20.20.20.3 daddr4=20.20.20.2 sport=5000 dport=57783 backup=0
[         REMOVED] token=917c83e8 remid=1
[       SF_CLOSED] token=917c83e8 remid=1 locid=1 saddr4=20.20.20.3 daddr4=20.20.20.2 sport=5000 dport=57783 backup=0

As you can see, after the SF_CLOSED there is no SF_ESTABLISHED

Again, best to retry with the proper flags.

Also, I wonder why the ss command only shows the initial subflow. It never shows the second subflow.

$ ss -M
State               Recv-Q               Send-Q                               Local Address:Port                               Peer Address:Port                
ESTAB               0                    0                                       10.10.10.3:5000                                 10.10.10.2:47930               

-M will only list the MPTCP connections, so only the addresses used by the application, the ones of the initial subflow. (Note that -i will show more info, like the number of subflows, etc.).
If you want to see the subflows, you can use ss -ti as root.

For more details: https://www.mptcp.dev/debugging.html#ss