Routing configuration issues in the LKL + MP_TCP solution

[mxi1]

Hi @thehajime , I notieced there had already one LKL + MP_TCP code base. I tired this on my desktop and found some problems for the routing configuration of the LKL-hijacked MP_TCP app, hope you have some instructions for them.

From the MP_TCP wiki pages, this link Manual configuration shows how to configure the routing policies for the interfaces. It mentioned we could create specific poclies entries for the specific interface/address+gateway, like this:

ip rule add from 10.1.1.2 table 1
ip route add 10.1.1.0/24 dev eth0 scope link table 1
ip route add default via 10.1.1.1 dev eth0 table 1

ip rule add from 10.1.2.2 table 2
ip route add 10.1.2.0/24 dev eth1 scope link table 2
ip route add default via 10.1.2.1 dev eth1 table 2

ip route add default scope global nexthop via 10.1.1.1 dev eth0

From my understanding, the LKL-hijacked ip route command will manage the LKL kernel's router table, so we can use the router-related commands/APIs to add/update the router table when initiliazing the hijack environment.

However, for controlling the policy table, it seems the ip rule commands can't access the policy table in the hijacked environment. When I am running the command ip rule list, under the Android shell, the output looks like this:

RTNETLINK answers: Operation not supported on transport endpoint
Dump terminated

I tried to run something like hijacked ip rule add iif wlan0 xxx xxx commands, the results is similar, like this:

RTNETLINK answers: Operation not supported on transport endpoint

Besides, I also searched the C/C++ API documentations for ip rule, and find this: Is there an C/C++ API for adding/removing rules?.

If convenient, could you give some guides on this problem? Thanks.

[thehajime]

The set of patches toward mptcp-enable LKL, including multiple routing table support, is now under preparation. The following is a WIP patch, which may fix issue you faced (I assume you're using https://github.com/libos-nuse/lkl-linux/tree/lkl_4.11-mptcp_v0.92).

motomuman@2c155a8

We (@motomuman is actually working) hope to have a PR very soon.
The rest of patch after this would be to support multiple interface/gateways configurations, which will be also proposed later.

Besides, I also searched the C/C++ API documentations for ip rule, and find this: Is there an C/C++ API for adding/removing rules?.

well, we just used standard netlink message (LKL_RTM_NEWRULE etc) to create a rule.

[mxi1]

Thanks, I will check the patches for it.

[mxi1]

@thehajime I applied the patch, and also verified Motomuman's netlink-policy-routing branch, here are what I get:

• The "ip rule xxx"-like commands can run under lkl-hijacked mode;
• The lkl_set_ipv4_gateway function doesn't work as we image, and it doesn't create new table for the specific interface.

From the code, it should create one new table with its ifindex * 2 as the table ID, and create specific rules to lookup this table. In my devel env, there's no such one table, and no rules, either.

The result for lkl-hijacked ip rule list command:

0:	from all lookup local 
32766:	from all lookup main 
32767:	from all lookup default 

The result for lkl-hijacked ip route show command:

default via 192.168.13.1 dev eth0 
192.168.13.0/24 dev eth0  proto kernel  scope link  src 192.168.13.2 

Can you or @motomuman help to re-check the netlink-policy-routing branch for my issue? Thank you very much!

[motomuman]

Hi @mxi1, could you show me commands you executed?
If "LKL_HIJACK_NET_IFGATEWAY" is set, init function call "lkl_if_set_ipv4_gateway".
"lkl_if_set_ipv4_gateway" calls "lkl_if_add_rule_from_saddr" and create rule table.
In my environment, I got following output.

motomu@netexp2  > git log -1
commit 524ab614cb5eb54977c5d46d856339af7ea18750
Author: Motomu Utsumi <motomu@hongo.wide.ad.jp>
Date:   Sun Oct 15 16:56:26 2017 +0900

    lkl: Support multiple routing tables
    
    This change allows lkl to have multiple routing tables.
    Multiple routing tables are used for mptcp configuration, for example.
    Also rewrite route modification code with netlink instead of ioctl.
    
    Reviewed-by: Hajime Tazaki <thehajime@gmail.com>
    Signed-off-by: Motomu Utsumi <motomuman@gmail.com>

motomu@netexp2 > sudo LKL_HIJACK_NET_IFTYPE=tap LKL_HIJACK_NET_IFPARAMS=tap7 LKL_HIJACK_NET_IP=10.0.0.5 LKL_HIJACK_NET_NETMASK_LEN=24 LKL_HIJACK_NET_IFGATEWAY=10.0.0.1 ./tools/lkl/bin/lkl-hijack.sh ip rule show
0:	from all lookup local 
32765:	from 10.0.0.5 lookup 4 
32766:	from all lookup main 
32767:	from all lookup default

motomu@netexp2 > sudo LKL_HIJACK_NET_IFTYPE=tap LKL_HIJACK_NET_IFPARAMS=tap7 LKL_HIJACK_NET_IP=10.0.0.5 LKL_HIJACK_NET_NETMASK_LEN=24 LKL_HIJACK_NET_IFGATEWAY=10.0.0.1 ./tools/lkl/bin/lkl-hijack.sh ip route show table 4
default via 10.0.0.1 dev enp3s0 
10.0.0.0/24 dev enp3s0 

[mxi1]

@motomuman Thanks for your reply. It's my fault, I didn't notice the LKL_HIJACK_NET_IFGATEWAY had already replaced the LKL_HIJACK_NET_GATEWAY param.

[motomuman]

Both LKL_HIJACK_NET_GATEWAY and LKL_HIJACK_NET_IFGATEWAY are configurable parameters in this branch.
To configure lkl (with multiple interfaces) for mptcp, both are necesarry(https://multipath-tcp.org/pmwiki.php/Users/ConfigureRouting).
If LKL_HIJACK_NET_IFGATEWAY is set, lkl creates rule table and configures local link and gateway route on the table. Correspond to following.

  # This creates two different routing tables, that we use based on the source-address.
  ip rule add from 10.1.1.2 table 1

  # Configure the two different routing tables
  ip route add 10.1.1.0/24 dev eth0 scope link table 1
  ip route add default via 10.1.1.1 dev eth0 table 1

If LKL_HIJACK_NET_GATEWAY is set, lkl configures default route. Correspond to following.

  # default route for the selection process of normal internet-traffic
  ip route add default scope global nexthop via 10.1.1.1 dev eth0

[mxi1]

@motomuman Thanks for your patient answer. 👍 ,I will try it tomorrow.