scheduler: react when subflow-level events pop up (ACK/RTO) #343

matttbe · 2023-02-01T16:41:45Z

When discussing about the packet scheduler API at the last meeting, it sounds very likely the current packet scheduler will not react by queuing more packets if some subflows only events are emitted, e.g. new TCP ACKs are received only acking things at TCP-level but not at MPTCP level.

The scheduler should probably be called when such events happen.

This can be checked with packetdrill: a ACK is received at TCP level and the scheduler might not send anything while it should (there is more room available).

Hints:

hooking at TCP window level changes might be enough:
- maybe too frequent?
- might affect perf?
or hook in MPTCP options to do more checks there?:
- quite heavy, the core should filter events

geliangtang · 2023-09-18T09:12:55Z

How about invoking __mptcp_push_pending() right after mptcp_pm_nl_work() in mptcp_worker(). Something like:

        mptcp_pm_nl_work(msk);
 
+       __mptcp_push_pending(sk, 0);
+
        mptcp_check_send_data_fin(sk);
        mptcp_check_data_fin_ack(sk);
        mptcp_check_data_fin(sk);

matttbe · 2023-11-09T20:45:53Z

How about invoking __mptcp_push_pending() right after mptcp_pm_nl_work() in mptcp_worker().

I forgot to reply to this one: we talked about that suggestion at the weekly meeting on the 19th of Sept

It doesn't seem OK:

we don't want to do such thing from the mptcp worker (we need to limit the work there)
the scheduler might need to react each time an ACK/RTO is received/fired: more frequently than when the worker is invoked

The current packet scheduler will not react by queuing more packets if some subflows only events are emitted, e.g. new TCP ACKs are received only acking things at TCP-level but not at MPTCP level. The scheduler should be called when such events happen. ack_update_msk() is invoked when an ACK is received, so it's the right place to call the scheduler by invoking __mptcp_check_push(). mptcp_subflow_timeout() is implemented to call the scheduler when a subflow timeout happens. But I'm not sure where is the right place to invoke it. I mean where is the right place when a RTO is fired. So I temporarily do it in mptcp_worker(). Closes: #343 Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn> Message-Id: <433320c3a9db77bea53a34fc9c43a3c7e3320399.1709693691.git.tanggeliang@kylinos.cn>

The current packet scheduler will not react by queuing more packets if some subflows only events are emitted, e.g. new TCP ACKs are received only acking things at TCP-level but not at MPTCP level. The scheduler should be called when such events happen. ack_update_msk() is invoked when an ACK is received, so it's the right place to call the scheduler by invoking __mptcp_check_push(). mptcp_subflow_timeout() is implemented to call the scheduler when a subflow timeout happens. But I'm not sure where is the right place to invoke it. I mean where is the right place when a RTO is fired. So I temporarily do it in mptcp_worker(). Closes: multipath-tcp#343 Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>

scheduler: implement a "opportunistic retransmission" The goal of the "opportunistic retransmission" is to quickly reinject packets when we notice the window has just been closed on one path, see the section 4.2 of: https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final125.pdf This is implemented in mptcp.org, see mptcp_rcv_buf_optimization(). With the current API in the Upstream kernel, a new scheduler doesn't have the ability to trigger a reinjection. Currently there are only hooks to initiate reinjection when the MPTCP RTO fires. The packet scheduler should be able to get more info: not just when MPTCP cwnd close or the seq num has increased (max allowed MPTCP level seq num to be sent == last ack + (...)) but also when there is a RTO at subflow level: maybe linked to scheduler: react when subflow-level events pop up (ACK/RTO) multipath-tcp#343 Note that the packet scheduler never significantly queue more than what the cwnd of a subflow would accept: currently, the in-kernel only accepts to queue up to the MPTCP level cwnd (a few more bytes due to round-up) Closes: multipath-tcp#332 Signed-off-by: Geliang Tang <geliang@kernel.org>

The current packet scheduler will not react by queuing more packets if some subflows only events are emitted, e.g. new TCP ACKs are received only acking things at TCP-level but not at MPTCP level. The scheduler should be called when such events happen. ack_update_msk() is invoked when an ACK is received, so it's the right place to call the scheduler by invoking __mptcp_check_push(). mptcp_subflow_timeout() is implemented to call the scheduler when a subflow timeout happens. But I'm not sure where is the right place to invoke it. I mean where is the right place when a RTO is fired. So I temporarily do it in mptcp_worker(). Closes: multipath-tcp#343 Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>

scheduler: implement a "opportunistic retransmission" The goal of the "opportunistic retransmission" is to quickly reinject packets when we notice the window has just been closed on one path, see the section 4.2 of: https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final125.pdf This is implemented in mptcp.org, see mptcp_rcv_buf_optimization(). With the current API in the Upstream kernel, a new scheduler doesn't have the ability to trigger a reinjection. Currently there are only hooks to initiate reinjection when the MPTCP RTO fires. The packet scheduler should be able to get more info: not just when MPTCP cwnd close or the seq num has increased (max allowed MPTCP level seq num to be sent == last ack + (...)) but also when there is a RTO at subflow level: maybe linked to scheduler: react when subflow-level events pop up (ACK/RTO) multipath-tcp#343 Note that the packet scheduler never significantly queue more than what the cwnd of a subflow would accept: currently, the in-kernel only accepts to queue up to the MPTCP level cwnd (a few more bytes due to round-up) Closes: multipath-tcp#332 Signed-off-by: Geliang Tang <geliang@kernel.org>

The current packet scheduler will not react by queuing more packets if some subflows only events are emitted, e.g. new TCP ACKs are received only acking things at TCP-level but not at MPTCP level. The scheduler should be called when such events happen. ack_update_msk() is invoked when an ACK is received, so it's the right place to call the scheduler by invoking __mptcp_check_push(). mptcp_subflow_timeout() is implemented to call the scheduler when a subflow timeout happens. But I'm not sure where is the right place to invoke it. I mean where is the right place when a RTO is fired. So I temporarily do it in mptcp_worker(). Closes: multipath-tcp#343 Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>

scheduler: implement a "opportunistic retransmission" The goal of the "opportunistic retransmission" is to quickly reinject packets when we notice the window has just been closed on one path, see the section 4.2 of: https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final125.pdf This is implemented in mptcp.org, see mptcp_rcv_buf_optimization(). With the current API in the Upstream kernel, a new scheduler doesn't have the ability to trigger a reinjection. Currently there are only hooks to initiate reinjection when the MPTCP RTO fires. The packet scheduler should be able to get more info: not just when MPTCP cwnd close or the seq num has increased (max allowed MPTCP level seq num to be sent == last ack + (...)) but also when there is a RTO at subflow level: maybe linked to scheduler: react when subflow-level events pop up (ACK/RTO) multipath-tcp#343 Note that the packet scheduler never significantly queue more than what the cwnd of a subflow would accept: currently, the in-kernel only accepts to queue up to the MPTCP level cwnd (a few more bytes due to round-up) Closes: multipath-tcp#332 Signed-off-by: Geliang Tang <geliang@kernel.org>

The current packet scheduler will not react by queuing more packets if some subflows only events are emitted, e.g. new TCP ACKs are received only acking things at TCP-level but not at MPTCP level. The scheduler should be called when such events happen. ack_update_msk() is invoked when an ACK is received, so it's the right place to call the scheduler by invoking __mptcp_check_push(). mptcp_subflow_timeout() is implemented to call the scheduler when a subflow timeout happens. But I'm not sure where is the right place to invoke it. I mean where is the right place when a RTO is fired. So I temporarily do it in mptcp_worker(). Closes: multipath-tcp#343 Signed-off-by: Geliang Tang <tanggeliang@kylinos.cn>

scheduler: implement a "opportunistic retransmission" The goal of the "opportunistic retransmission" is to quickly reinject packets when we notice the window has just been closed on one path, see the section 4.2 of: https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final125.pdf This is implemented in mptcp.org, see mptcp_rcv_buf_optimization(). With the current API in the Upstream kernel, a new scheduler doesn't have the ability to trigger a reinjection. Currently there are only hooks to initiate reinjection when the MPTCP RTO fires. The packet scheduler should be able to get more info: not just when MPTCP cwnd close or the seq num has increased (max allowed MPTCP level seq num to be sent == last ack + (...)) but also when there is a RTO at subflow level: maybe linked to scheduler: react when subflow-level events pop up (ACK/RTO) multipath-tcp#343 Note that the packet scheduler never significantly queue more than what the cwnd of a subflow would accept: currently, the in-kernel only accepts to queue up to the MPTCP level cwnd (a few more bytes due to round-up) Closes: multipath-tcp#332 Signed-off-by: Geliang Tang <geliang@kernel.org>

matttbe · 2024-03-14T10:40:51Z

From the last meeting:

What is important before sending any changes to netdev, is to have a "performance environment":
- to measure the differences, see the impact, etc.
- Might be good to add such tests in the selftests:
  - We can depend on external tools, e.g. TCP Big selftests uses netperf
  - But maybe not bigger tools like transperf even if it would analyse already a few things for us
  - udpgso_bench_[rt]x.c: might be extended
- @matttbe is going to look at that
- It should run on HW devices and virtual ones:
  - not the same utilisation of the CPU
  - some ideas: using RPS in RX, and adding netem delay in TX, to force not having the same context being reused, different usage of the CPU
It looks like this work on the scheduler is challenging, and it would be better to prepare this work, split it, etc.:
- So if someone wants to work on that, best to come up with a plan, with the different steps, what will be changed in the architecture, etc.
- The goal is not to spend a long time in the reviews, and completely reworking the code each time, etc.