katran with bonding

[actaeon]

Hi folks,

I have a bunch of servers that use 802.3ad bonding. Can I just run a separate instance of katran for each member ethernet nic? (dividing up the interrupts and such correctly, of course.) If I understand correctly, Katran is low enough in the stack where it wouldn't know or care about the bonding. Is this correct?

Thanks!

[tehnerd]

1. for sep instance on per link - most likely it's possible (we didn't try, but i dont think there is any limitation inside which would block you to do so)
2. as for bonding - for XDP in "driver mode" (best possible performance) - yeah it's lower in a stack then bonding driver. however there is a chance (we didn't try; would appreciate if you try to do so) that on bonding interface it could work in "generic xdp" mode (where xdp hook is located right after kernel's skb allocation; it's easy slower then "driver" but still have pretty good speed)

[tehnerd]

you dont need to specify how to attach katran ("driver" or "generic" xdp) it would try best one by default and would fallback to generic if driver doesn't work

[actaeon]

I'll try some stuff and see what works, including the driver mode on the bond interface. Should I just keep this issue open and report back? It will probably be a week or so.

[tehnerd]

sure. we can leave this open

[tehnerd]

Just wonde - have you tried to do it with bonding?

[actaeon]

Yes, I have it working with bonding by running a separate Katran instance on each link, no issues. You asked if I could try with generic XDP on the bond interface itself, and I haven't done that yet. (I kinda forgot about it.) I'll devote some time to it this week.

[tehnerd]

I could try to modify xdproot program so you will be able to run separate instance of it but it will use single program array (/sys/fs/bpffs/jmp_eth0) and you will need just a single instance of katran registred there. Would that simplify your tests?

[tehnerd]

(Xdproot in this context is a small xdp program described in “starting katran is shared mode” from EXAMPLE.md)

[actaeon]

You kinda lost me on the last couple of comments. I think I understand how Katran can operate in "shared" mode using the xdproot binary and knowing the program position of the katran BPF code. But I don't understand what running two instances of the xdproot binary will get me...?

[tehnerd]

correct me if i'm wrong, but afai can understand the way it works today in your environment is.
you have something line bond0 which is based on top of eth0/eth1 (as example). and you just start two instances of katran_server with -intf eth0 and eth1. the problem with that approach is to keep this katran_servers in sync (e.g. if you add VIP to one - you will need to add to another etc). what could be done instead is to modify xdproot in a way, that you would attach xdproot to interface eth0 and eth1 (as two sep binaries/runs) but you will need to run just a single instance of katran_server (which would be attached/registred in shared (by xdproot's) bpf's program array). w/ single instance of katran server all additional memory waste and complexity in config duplication would go away

[actaeon]

Thanks for the explanation. Yes, you are correct - that's my situation exactly. If I could run a single instance of Katran in the way you described it would be fantastic. If you're willing to make such a change, I'm happy to test it or contribute in whatever way I can.

[tehnerd]

Sure. I’m on LPC next week so could take some time to add this. But yeah will try to add this feature in a week two

[tehnerd]

sorry for the late update. with latest commit i've added a support for root xdp program which could be installed on multiple interface and reuse same program array. the way it should work is (from https://github.com/facebookincubator/katran/blob/master/EXAMPLE.md "starting katran is shared mode"):
your bundle interface contains multiple physical links. lets for simplicity imagine that it is eth0 eth1

1. you need to modify script : https://github.com/facebookincubator/katran/blob/master/install_xdproot.sh in a way that it would install first on eth0 and then on eth1 interfaces. also you need for both interfaces to reuse same path to pinned maps "-bpfpath=/sys/fs/bpf/${attachment_point}"
2. then you need to start katran in "shared mode" the only difference compare to regular mode is that you need to add "-map_path /sys/fs/bpf/{attachment_point} -prog_pos=2" in the end of the command. where map_path is the same as in install_xdproot.sh script's -bpfpath=/sys/fs/bpf/{attachment_point}

let me know if you have any questions

[actaeon]

This is fantastic, thank you. I'll test as soon as I can, probably next week.

[whl739]

Hi, guys,
I have an issue with bonding.
I wrote a simple balancer-user program with katran's bpf files, it will load bpf file and set vip and rs.
It only works when running seprate instance on per link in "generic xdp" mode, but when in "driver xdp"
mode, sometimes good, sometimes bad.
I don't know why, can someone help me?

[tehnerd]

@whl739 could you please open separate isssue (so we won’t mix two environment etc in a same issue) and provide this info

1. ethtool -i (eg ethtool -i eth0)
2. uname -mrs
3. version of katran you are running (last commit hash from “git log”)
4. what command you are using to start it
5. better description of “sometimes good sometimes bad”. What exactly is going on? What works what does not
6. output of “ip link” when katran is running

[whl739]

@tehnerd thanks, i opened a new issue: #20.

[tehnerd]

few other people seems like had no issue w/ running xdp on bonding by attaching to physical interface directly. please reopen this one or create a new issue if you will have any questions