-
-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issues with backup flow #315
Comments
Hi Monika, Thank you for the bug report.
Did you disable the service not to have it started at boot time or did you stop it later? I mean: did you check there was no rules previously added by this daemon?
What are the (IP) routes and rules you have in place? (
I wonder if this has not been "fixed" in v5.17.
The ideal for us would be to install a recent kernel, ideally compiled from our For Ubuntu, there are packages regularly built: https://kernel.ubuntu.com/~kernel-ppa/mainline/ |
By chance, do you have any answers to the questions from my previous message? |
Hi Matt, I tried to use a recent kernel 6.1.2-v8+. I compiled the kernel source from https://github.com/raspberrypi/linux When I try to mptcpize iperf, I get the following error I tried to enable mptcp support with iperf3 - mptcpize enable iperf3 But still the issue exists. So stuck here and not able to check the backup flow issue faced earlier with 5.15 kernel version. Thanks |
Hi Monika, If you got |
If you used the default config for a RPI and MPTCP is not enabled by default, please ask them to enable it by default as it is done in most distribution. And it makes sense to have that on a RPI. |
Hi Matt, |
mmh, strange... Could you run import socket
s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, socket.IPPROTO_MPTCP)
s.bind(("::1", 1234))
s.listen(1) Do you have the same I'm not sure how else you could get |
Executing "import socket gives the same error - OSError: [Errno 93] Protocol not supported. Can you brief me about SELinux rules? |
If I'm not mistaken, there are 3 possible errors when creating a socket:
I don't know all paths producing a If you can see the SELinux can be used to not allow you to create MPTCP sockets but it needs manual configuration, I guess it is not done automatically on a RPI and you didn't do that. You could cause some similar errors with BPF, another libc (or changing the behaviour of One last thing to check maybe is if you can see If not, I'm not sure where else to look at... |
Thank you so much for a comprehensive response. |
I would like to add that AF_INET6 with TCP works fine and it fails only with MPTCP. I missed to add CONFIG_MPTCP_IPV6 I guess. |
I thought you did but if not, yes it would be better to enable this and recompile the kernel. Maybe apps create a v6 sockets (in fact, a mix v4/v6) even if v4 is used after. |
Hmmm.. I get it. Compilation in progress. I will keep you posted. Much appreciate your support. |
Hi Matt, |
Hi @Monika-Research, could you resolve the problem? |
Hi @AlejandraOliver, I still couldn't get backup to working even with kernel version 6.1.2 |
Hi @matttbe, I am still facing issue with backup flow. Any leads? Thanks, |
What is the issue exactly (there are a few in the comments)? You set the backup flag on one endpoint but it is not taken into account? Do you mind trying to reproduce the bug on a recent v6.3 kernel release (last one is the 6.3.5 from what I can see) with a recent version of Then we will need the output of |
Hi Matt,
Yes, it is backup=0 always in the "ip mptcp monitor" and data is transmitted over that subflow.
Sure. I compiled the export branch and it is 6.4 I believe. should be fine? |
Strange, looks like a bug or a config issue.
Yes, using the export branch (make sure it is up to date) is even better! |
Hi @matttbe , I have compiled the kernel from the repository and have mptcp working correctly, but I have a question with schedulers. I'm testing with them and I would like to set one of the interfaces as a backup so that it will only be activated in case the others go down. Do you know the command I should use to do it,? I've tried
but I think it's not working. Thanks. |
@AlejandraOliver you need to specify if this endpoint will be used to create subflow (
As mentioned in my previous comment, make sure both the kernel ( |
FYI, you can see how we are validating this feature there: mptcp_net-next/tools/testing/selftests/net/mptcp/mptcp_join.sh Lines 2571 to 2634 in 15f31c7
|
ip mptcp endpoint add dev subflow backup ## client Hi @matttbe, I don't quite understand what depends on whether you use subflow or signal. I have a virtual scenario in VirtualBox with 2 Ubuntu 22.04 LTS machines. On both I have the v6.3-rc6 kernel from the export branch compiled and installed. In addition, I have already configured the routing, and I have also compiled and loaded in the kernel the different types of schedulers (bpf_red, bpf_rr,bpf_first and bpf_bkup). Each machine has 4 interfaces: MACHINE1 (client) MACHINE2 (server) My goal is for both machines to communicate via the 3 interfaces (eth1, eth2 and eth3). The problem is that when testing with the schedulers, for example the redundant, if I put one of the interfaces as a backup and the others go down, this interface is not reactivated. I would be very grateful if you could show me an example of how you would configure the 3 interfaces in client and server, with one of them as a backup. Like, what path manager commands (ip mptcp endpoint add dev subflow backup- So I think It's maybe because I don't set the backup subflow correctly, or a configuration problem not yet solved in the export branch. Thanks a lot in advance!!! |
The path-manager needs to know if the device is interested by either:
So in short, you should use the
These schedulers are just test toys (maybe with bugs) and except the last one, they don't take into account the backup flag if I'm not mistaken. Except if you want to improve these schedulers (see #75), we don't recommend using them for the moment.
Something like that (quickly done, not tested): client:
server:
If you still have the issue, you know what to share ;-) |
@matttbe Following the configurations you have mentioned, I still have the problem that even if the endpoint is set as backup, data is transmitted. I have the following configuration:
Server:
If I captured in Wireshark, I see the following things:
In 1, interfaces of different network should not create subflow, I think. I have a question apart of this. If all the interfaces between the 2 machines were in the same internal network, I need to configure any type of routing before doing these path management commands? Or is simply having each interface assigned with an IP address is enough? Thank you very much. |
Thanks! You can also put the binaries in a .zip and upload it here ;-) The traces you made with tcpdump don't contain the SYN (and a Also the output of And finally, I guess you have
mmh, a few things:
We need routing rules to tell the kernel that the packets from a specific IP address have to be routed through the device linked to this IP address. So in your case, to have the traffic from 172.16.0.1 using the eth3 device and not another one to be able to receive the replies. Typically, a default route is configured which means that all the traffic is routed through a specific interface. There are also routes per other IP addresses you have to say that the traffic within the same prefix is routed through the right interface. If the two devices have IPs within the same prefix, you certainly already have such routes:
But typically you don't have rules to force routing the traffic from a specific IP to the interface linked to this IP:
e.g.
|
only the endpoints you added. You can always use |
I disabled mptcpd and now the output of Client:
Server
|
looks good now. We can see the |
Now, no data is retransmitted initially through the backup subflow (in Wireshark the backup flag is set to 1 in both directions). This is perfect but, when eth1 and eth2 go down, it should be reactivated, right? |
It should yes. It doesn't? with the default scheduler? |
No, when I put down eth1 and eth2, eth3 is still in backup. I'm using default scheduler |
What do you mean by saying "it is still in backup"? |
I mean that data is not transmitted on that interface (I'm seeing this with ifstat) |
From what I've been seeing in Wireshark and ifstat, it reactivates for a very short period of time (in wireshark two packets are sent with the PUSH flag set) and then it cuts off and resets to zero. |
do you mean a RST is sent? Could you share the packet traces please?
|
The Wireshark capture is in my repo with name prueba.rar. The output of The output of net.mptcp.add_addr_timeout = 120 I think what happens is that it is reactivated to transmit the packets that were left unsent in the downed subflows, and when it does so, it stops transmitting new data. |
One thing about that. If I want to set the same enviroment but with using mptcpd, I need to install one of the kernel of the export branch? Otherwise, installing a official kernel >v5.6, I wouldn't have any kind of scheduler, only the default scheduling that is preinstalled in the kernel, right? Thanks!! |
Thank you for the new info. I started to write this reply before but I got interrupted. I didn't check the traces and log yet but here is already what I already wrote (and I have a few meetings coming, not sure I can check them soon)
Please next time use .gz or .zip (or .xz or .7z but they are not accepted by GitHub I think) but not a proprietary file format.
Interesting. I don't think we have tests where we disable one interface while data are being transferred. We should I suppose.
mptcpd doesn't handle the different packet schedulers (yet?), only the path-managers. |
Okay, I will keep checking it. Thank you for your time! |
Hi, I have some dependency on kernel version and ip version with our products/devices. But I really wanted to get the backup option working. What can be done in this case? Can someone point to the exact patches required to get it working? Much appreciate guidance. Thank you. |
In case when is64 == 1 in emit(A64_REV32(is64, dst, dst), ctx) the generated insn reverses byte order for both high and low 32-bit words, resuling in an incorrect swap as indicated by the jit test: [ 9757.262607] test_bpf: #312 BSWAP 16: 0x0123456789abcdef -> 0xefcd jited:1 8 PASS [ 9757.264435] test_bpf: #313 BSWAP 32: 0x0123456789abcdef -> 0xefcdab89 jited:1 ret 1460850314 != -271733879 (0x5712ce8a != 0xefcdab89)FAIL (1 times) [ 9757.266260] test_bpf: #314 BSWAP 64: 0x0123456789abcdef -> 0x67452301 jited:1 8 PASS [ 9757.268000] test_bpf: #315 BSWAP 64: 0x0123456789abcdef >> 32 -> 0xefcdab89 jited:1 8 PASS [ 9757.269686] test_bpf: #316 BSWAP 16: 0xfedcba9876543210 -> 0x1032 jited:1 8 PASS [ 9757.271380] test_bpf: #317 BSWAP 32: 0xfedcba9876543210 -> 0x10325476 jited:1 ret -1460850316 != 271733878 (0xa8ed3174 != 0x10325476)FAIL (1 times) [ 9757.273022] test_bpf: #318 BSWAP 64: 0xfedcba9876543210 -> 0x98badcfe jited:1 7 PASS [ 9757.274721] test_bpf: #319 BSWAP 64: 0xfedcba9876543210 >> 32 -> 0x10325476 jited:1 9 PASS Fix this by forcing 32bit variant of rev32. Fixes: 1104247 ("bpf, arm64: Support unconditional bswap") Signed-off-by: Artem Savkov <asavkov@redhat.com> Tested-by: Puranjay Mohan <puranjay12@gmail.com> Acked-by: Puranjay Mohan <puranjay12@gmail.com> Acked-by: Xu Kuohai <xukuohai@huawei.com> Message-ID: <20240321081809.158803-1-asavkov@redhat.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Recent additions in BPF like cpu v4 instructions, test_bpf module exhibits the following failures: test_bpf: #82 ALU_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times) test_bpf: #83 ALU_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times) test_bpf: #84 ALU64_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times) test_bpf: #85 ALU64_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times) test_bpf: #86 ALU64_MOVSX | BPF_W jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times) test_bpf: #165 ALU_SDIV_X: -6 / 2 = -3 jited:1 ret 2147483645 != -3 (0x7ffffffd != 0xfffffffd)FAIL (1 times) test_bpf: #166 ALU_SDIV_K: -6 / 2 = -3 jited:1 ret 2147483645 != -3 (0x7ffffffd != 0xfffffffd)FAIL (1 times) test_bpf: #169 ALU_SMOD_X: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times) test_bpf: #170 ALU_SMOD_K: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times) test_bpf: #172 ALU64_SMOD_K: -7 % 2 = -1 jited:1 ret 1 != -1 (0x1 != 0xffffffff)FAIL (1 times) test_bpf: #313 BSWAP 16: 0x0123456789abcdef -> 0xefcd eBPF filter opcode 00d7 (@2) unsupported jited:0 301 PASS test_bpf: #314 BSWAP 32: 0x0123456789abcdef -> 0xefcdab89 eBPF filter opcode 00d7 (@2) unsupported jited:0 555 PASS test_bpf: #315 BSWAP 64: 0x0123456789abcdef -> 0x67452301 eBPF filter opcode 00d7 (@2) unsupported jited:0 268 PASS test_bpf: #316 BSWAP 64: 0x0123456789abcdef >> 32 -> 0xefcdab89 eBPF filter opcode 00d7 (@2) unsupported jited:0 269 PASS test_bpf: #317 BSWAP 16: 0xfedcba9876543210 -> 0x1032 eBPF filter opcode 00d7 (@2) unsupported jited:0 460 PASS test_bpf: #318 BSWAP 32: 0xfedcba9876543210 -> 0x10325476 eBPF filter opcode 00d7 (@2) unsupported jited:0 320 PASS test_bpf: #319 BSWAP 64: 0xfedcba9876543210 -> 0x98badcfe eBPF filter opcode 00d7 (@2) unsupported jited:0 222 PASS test_bpf: #320 BSWAP 64: 0xfedcba9876543210 >> 32 -> 0x10325476 eBPF filter opcode 00d7 (@2) unsupported jited:0 273 PASS test_bpf: #344 BPF_LDX_MEMSX | BPF_B eBPF filter opcode 0091 (@5) unsupported jited:0 432 PASS test_bpf: #345 BPF_LDX_MEMSX | BPF_H eBPF filter opcode 0089 (@5) unsupported jited:0 381 PASS test_bpf: #346 BPF_LDX_MEMSX | BPF_W eBPF filter opcode 0081 (@5) unsupported jited:0 505 PASS test_bpf: #490 JMP32_JA: Unconditional jump: if (true) return 1 eBPF filter opcode 0006 (@1) unsupported jited:0 261 PASS test_bpf: Summary: 1040 PASSED, 10 FAILED, [924/1038 JIT'ed] Fix them by adding missing processing. Fixes: daabb2b ("bpf/tests: add tests for cpuv4 instructions") Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Link: https://msgid.link/91de862dda99d170697eb79ffb478678af7e0b27.1709652689.git.christophe.leroy@csgroup.eu
Hi Team,
Below is my setup details:
2 rpis - Ubuntu 22.04 server, 5.15.0-1016 kernel
Clients IP addresses - 192.168.11.2/24, 192.168.12.2/24
Server IP addresses - 192.168.11.3/24, 192.168.12.3/24
Stopped mptcpd and using ip mptcp to set path management rules
My aim is to create a primary flow between 192.168.11.2 & 192.168.11.3 and a backup subflow between 192.168.12.2 & 192.168.12.3 network. That is, in the middle of a data transmission, if I put down the primary interface, I am expecting the backup flow to come as a savior. However I failed to create a backup flow from 192.168.12.2 to 192.168.12.3.
Following is the configuration used
Client config:
ip mptcp limits set add_addr_accepted 1 subflows 1
ip mptcp endpoint add 192.168.12.2 dev bat1 subflow backup
Server config:
ip mptcp limits set add_addr_accepted 1 subflows 1
ip mptcp endpoint add 192.168.12.3 dev bat1 signal backup
The above config results in following
[SF_ESTABLISHED] token=d262648 remid=0 locid=1 saddr4=192.168.12.2 daddr=192.168.11.3 sport=41215 dport=5201 backup=1 ifindex=9
[SF_ESTABLISHED] token=753246 remid=0 locid=1 saddr4=192.168.12.2 daddr=192.168.12.3 sport=41860 dport=5201 backup=0
I checked the configurations suggested in the following discussions
#237
#191
However I couldn't get 192.168.12.2-192.168.12.3 as backup flow.
In case I remove signal option from server as suggested in one of the links above, I get a backup flow from 192.168.12.2-192.168.11.3.
Requesting your suggestions please in creating and putting the correct backup flow to use when the primary flow fails during a transmission.
Thanks,
Monika
The text was updated successfully, but these errors were encountered: