katran xdp issue when running with bonding in driver mode #20

whl739 · 2018-12-25T07:32:17Z

I have a server that use 802.3ad bonding, and there's an issue in driver mode.

Since there were some troubles with building katran's dependences on my vm(out of memory when building fizz), so i wrote a balancer-user program(modified from kernel's bpf sample: xdp_tx_iptunnel_user.c) and complied it with katran's bpf files. The 'balancer_user.c' and some modifies are here:
balancer_user.c.txt
balaner_kern.diff.txt

It is only used to do a test. It will load katran's bpf file and set vip and rs, and only works when running seprate instance on per link in "generic xdp" mode.

Below is my env.
katran's version is 4c1fe70.

$ ifconfig
bond1: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST>  mtu 1500
        inet 172.30.4.3  netmask 255.255.255.0  broadcast 172.30.4.255
        inet6 fe80::ec6:ccff:fe9e:a9ce  prefixlen 64  scopeid 0x20<link>
        ether aa:bb:cc:dd:ee:ff  txqueuelen 1000  (Ethernet)
        RX packets 23856910322  bytes 2133122139232 (1.9 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 33499171263  bytes 22037784326757 (20.0 TiB)
        TX errors 0  dropped 642 overruns 0  carrier 0  collisions 0

eth0: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether aa:bb:cc:dd:ee:ff  txqueuelen 1000  (Ethernet)
        RX packets 1214891  bytes 766042605 (730.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1353364  bytes 719038863 (685.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether aa:bb:cc:dd:ee:ff  txqueuelen 1000  (Ethernet)
        RX packets 23202  bytes 14578863 (13.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25149  bytes 14607499 (13.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2797883824  bytes 6603725657284 (6.0 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2797883824  bytes 6603725657284 (6.0 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tunl0: flags=193<UP,RUNNING,NOARP>  mtu 1480
        inet 172.30.4.253  netmask 255.255.255.255
        tunnel   txqueuelen 1000  (IPIP Tunnel)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

"172.30.4.253" is the vip.

$ cat /etc/sysconfig/network-scripts/ifcfg-bond1
#IP Config for bond1:
DEVICE=bond1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
BONDING_OPTS="mode=4 miimon=100 ad_select=1 updelay=200 xmit_hash_policy=2"
IPV6INIT=no
TYPE=Unknown
NM_CONTROLLED=no
IPADDR='172.30.4.3'
NETMASK='255.255.255.0'
GATEWAY='172.30.4.1'

$ cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2+3 (2)
...

$ ethtool -i eth0
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x800003e2
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

$ uname -mrs
Linux 4.18.8-1.el7.elrepo.x86_64 x86_64

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
5: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether aa:bb:cc:dd:ee:11 brd ff:ff:ff:ff:ff:ff
6: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff

Here are my commands:

$ ./balancer 
Start a XDP prog which encapsulates incoming packets
in an IPv4/v6 header and XDP_TX it out.  The dst <VIP:PORT>
is used to select packets to encapsulate

Usage: ./balancer [...]
    -i <ifindex> Interface Index
    -a <vip-service-address> IPv4 or IPv6
    -p <vip-service-port> Service  port
    -s <source-ip> Used in the IPTunnel header
    -d <dest-ip> Used in the IPTunnel header, ip range (e.g. 1.1.1.1,2.2.2.2) is also allowed
    -m <dest-MAC> Used in sending the IP Tunneled pkt
    -T <stop-after-X-seconds> Default: 0 (forever)
    -P <IP-Protocol> Default is TCP
    -S use skb-mode
    -N enforce native mode
    -h Display this help

It works fine in generic mode(with '-S'):

sudo ./balancer -i 2 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -S

sudo ./balancer -i 3 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -S

the output is:

sum:    318928 pkts, rate:    159464 pkts/s
sum:   4213744 pkts, rate:   2106872 pkts/s
sum:   4241320 pkts, rate:   2120660 pkts/s
sum:   4251751 pkts, rate:   2125875 pkts/s
sum:   4269300 pkts, rate:   2134650 pkts/s
sum:   4288777 pkts, rate:   2144388 pkts/s
sum:   4295820 pkts, rate:   2147910 pkts/s

When using driver mode(with '-N'), it won't work properly.

sudo ./balancer -i 2 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -N

sudo ./balancer -i 3 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -N

sometimes it works:

$ curl 172.30.4.253/t -v
* About to connect() to 172.30.4.253 port 80 (#0)
*   Trying 172.30.4.253...
* Connected to 172.30.4.253 (172.30.4.253) port 80 (#0)
> GET /t HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.30.4.253
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.14.1
< Date: Tue, 25 Dec 2018 07:15:00 GMT
< Content-Type: application/octet-stream
< Content-Length: 2
< Connection: keep-alive
< 
* Connection #0 to host 172.30.4.253 left intact
OK

and sometimes not:

$ curl 172.30.4.253/t -v
* About to connect() to 172.30.4.253 port 80 (#0)
*   Trying 172.30.4.253...
* Connection timed out
* Failed connect to 172.30.4.253:80; Connection timed out
* Closing connection 0
curl: (7) Failed connect to 172.30.4.253:80; Connection timed out

I tried curl many times, almost 70% was failed.

And also i tried to run with "bond1" in generic mode, it couldn't work.

sudo ./balancer -i 6 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 10.210.9.100,10.210.9.101,10.210.9.102 -m 68:05:ca:21:d6:e5  -S

the result is:

$ curl 172.30.4.253/t -v
* About to connect() to 172.30.4.253 port 80 (#0)
*   Trying 172.30.4.253...
* Connection refused
* Failed connect to 172.30.4.253:80; Connection refused
* Closing connection 0
curl: (7) Failed connect to 172.30.4.253:80; Connection refused

The text was updated successfully, but these errors were encountered:

tehnerd · 2018-12-26T00:13:55Z

i'm not even sure why it is working w/ generic mode.
there is multiple errors/missconfigurations. few of em are:

mac address (in startup command) must be mac address of default router, not interface one
in userspace program you have union ipaddr. which is not what katran is using for real's ip address description (flag field is missing)
i have no idea is it even correct they way you started to do "key = hash % (*real_pos);"
...

if you have memory issue with building fizz - please enable swap partition on your VM (so build process will use disk, if VM doesn't have enough ram). and run proper userspace + bpf code after that. w/o this modification. after that we can try to figure out whats going on

whl739 · 2018-12-26T08:46:30Z

the mac is exactly the router's address(i pasted a fake address here).
the union is just used for parsing command line args, it is used to fill real_definition struct, and the flag field is just 0.
i changed the semantic of ch_rings, the '*real_pos' means the number of real servers, it works fine.
actually the swap is enabled on my vm, still not enough(1g ram, 1g swap), i don't know why include this library, seems useless.

i will increase my vm's memory and recompile katran, and test it again. but i still consider that the result is same with this.

whl739 · 2018-12-28T12:57:23Z

Finally, i tried the example of katran, the result is same with above.
I ran separate instance on per link.

$ sudo ./katran_server_grpc -balancer_prog ./balancer_kern.o -default_mac aa:bb:cc:dd:ee:ff \
      -lru_size=10000 -hc_forwarding=0 -intf=eth0 -port 50051
I1228 20:06:56.609061 53274 KatranGrpcService.cpp:67] Starting Katran
E1228 20:06:56.609275 53274 BpfLoader.cpp:166] Can't read section size for index: 2
I1228 20:06:56.609289 53274 BpfLoader.cpp:448] Skipping section: 2 of file: ./balancer_kern.o
I1228 20:06:56.627182 53274 BpfLoader.cpp:367] relocation for non existing prog w/ idx 7
E1228 20:06:56.640859 53274 BpfLoader.cpp:103] Can't find map w/ name: lpm_src_v4
E1228 20:06:56.640874 53274 BpfLoader.cpp:103] Can't find map w/ name: decap_dst
Server listening on 0.0.0.0:50051
I1228 20:07:33.064746 53279 KatranLb.cpp:433] adding new vip: 10.210.9.103:80:6

$ sudo ./katran_server_grpc -balancer_prog ./balancer_kern.o -default_mac aa:bb:cc:dd:ee:ff \
         -lru_size=10000 -hc_forwarding=0 -intf=eth1 -port 50052
I1228 20:07:10.631013 53371 KatranGrpcService.cpp:67] Starting Katran
E1228 20:07:10.631250 53371 BpfLoader.cpp:166] Can't read section size for index: 2
I1228 20:07:10.631265 53371 BpfLoader.cpp:448] Skipping section: 2 of file: ./balancer_kern.o
I1228 20:07:10.647718 53371 BpfLoader.cpp:367] relocation for non existing prog w/ idx 7
E1228 20:07:10.660948 53371 BpfLoader.cpp:103] Can't find map w/ name: lpm_src_v4
E1228 20:07:10.660962 53371 BpfLoader.cpp:103] Can't find map w/ name: decap_dst
Server listening on 0.0.0.0:50052
I1228 20:07:33.918964 53450 KatranLb.cpp:433] adding new vip: 10.210.9.103:80:6

Set vip and rs:

$ ./katran_goclient -A -t 10.210.9.103:80 -rpc_port 50051
2018/12/28 17:31:56 Adding service: 10.210.9.103:80 6
2018/12/28 17:31:56 Vip modified

$ ./katran_goclient -a -t 10.210.9.103:80 -r 10.210.9.102 -rpc_port 50051
2018/12/28 17:48:55 Reals modified

$ ./katran_goclient -A -t 10.210.9.103:80 -rpc_port 50052
2018/12/28 17:31:56 Adding service: 10.210.9.103:80 6
2018/12/28 17:31:56 Vip modified

$ ./katran_goclient -a -t 10.210.9.103:80 -r 10.210.9.102 -rpc_port 50052
2018/12/28 17:48:55 Reals modified

Configured service:

 $ ./katran_goclient -l -rpc_port 50051  
2018/12/28 17:50:16 vips len 1
VIP:         10.210.9.103 Port:     80 Protocol: tcp
Vip's flags: 
 ->10.210.9.102      weight: 1
 
$ ./katran_goclient -l -rpc_port 50052
2018/12/28 17:50:17 vips len 1
VIP:         10.210.9.103 Port:     80 Protocol: tcp
Vip's flags: 
 ->10.210.9.102      weight: 1

Test client:

$ curl http://10.210.9.103/t -v
* About to connect() to 10.210.9.103 port 80 (#0)
*   Trying 10.210.9.103...
* Connection timed out
* Failed connect to 10.210.9.103:80; Connection timed out
* Closing connection 0
curl: (7) Failed connect to 10.210.9.103:80; Connection timed out

Stats of LB:

$ ./katran_goclient -s  -rpc_port 50052
vip: 10.210.9.103:80:6 :        1 pkts/sec       52 bytes/sec
vip: 10.210.9.103:80:6 :        1 pkts/sec       52 bytes/sec

$./katran_goclient -s -lru -rpc_port 50052
summary: 1 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 1 pkts/sec
summary: 1 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 1 pkts/sec

Actually, after i started the rpc server, i got stutters in terminal, seemed lost packets.

Interface bond1 is not working too.

$ sudo ./katran_server_grpc -balancer_prog ./balancer_kern.o -default_mac aa:bb:cc:dd:ee:ff \
       -lru_size=10000 -hc_forwarding=0  -intf=bond1 -port 50051
I1228 20:41:18.373968  6522 KatranGrpcService.cpp:67] Starting Katran
E1228 20:41:18.374246  6522 BpfLoader.cpp:166] Can't read section size for index: 2
I1228 20:41:18.374264  6522 BpfLoader.cpp:448] Skipping section: 2 of file: ./balancer_kern.o
I1228 20:41:18.396837  6522 BpfLoader.cpp:367] relocation for non existing prog w/ idx 7
E1228 20:41:18.409310  6522 BpfLoader.cpp:103] Can't find map w/ name: lpm_src_v4
E1228 20:41:18.409329  6522 BpfLoader.cpp:103] Can't find map w/ name: decap_dst
Server listening on 0.0.0.0:50051
I1228 20:41:31.838093  6527 KatranLb.cpp:433] adding new vip: 10.210.9.103:80:6

the result is:

$ curl 10.210.9.103/t -v
* About to connect() to 10.210.9.103 port 80 (#0)
*   Trying 10.210.9.103...
* Connection refused
* Failed connect to 10.210.9.103:80; Connection refused
* Closing connection 0
curl: (7) Failed connect to 10.210.9.103:80; Connection refused

obviously, forwarding is not working.

But if i add flag XDP_FLAGS_SKB_MODE to call modifyXdpProg and start the two servers, all will work fine.

$ curl http://10.210.9.103/t -v
* About to connect() to 10.210.9.103 port 80 (#0)
*   Trying 10.210.9.103...
* Connected to 10.210.9.103 (10.210.9.103) port 80 (#0)
> GET /t HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.210.9.103
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.14.1
< Date: Fri, 28 Dec 2018 12:30:29 GMT
< Content-Type: application/octet-stream
< Content-Length: 2
< Connection: keep-alive
< 
* Connection #0 to host 10.210.9.103 left intact
OK

tehnerd · 2018-12-28T14:47:25Z

could you please show stats from port 5051 as well.
output of dmest | tail -100
"ip link" after xdp programs loaded
"tcpdump -ni any proto 4" from real server when trying to run curl

whl739 · 2018-12-28T15:21:23Z

In fact, only one nic can receive the request packets.

$ ./katran_goclient -s  -rpc_port 50051
vip: 10.210.9.103:80:6 :        0 pkts/sec        0 bytes/sec
vip: 10.210.9.103:80:6 :        0 pkts/sec        0 bytes/sec

$ ./katran_goclient -s -lru  -rpc_port 50051 
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec

$ dmesg | tail -100                               
tx_buffer_info[next_to_clean]
  time_stamp           <0>
  jiffies              <109e44780>
[166254.552583] ixgbe 0000:02:00.1 eth1: tx hang 462 detected on queue 21, resetting adapter
[166254.557718] ixgbe 0000:02:00.1 eth1: initiating reset due to tx timeout
[166254.562817] ixgbe 0000:02:00.1 eth1: Reset adapter
[166255.358438] ixgbe 0000:02:00.1 eth1: speed changed to 0 for port eth1
[166255.381953] ixgbe 0000:02:00.1 eth1: NIC Link is Up 10 Gbps, Flow Control: None
[166255.462802] bond1: link status up for interface eth1, enabling it in 200 ms
[166255.678860] bond1: link status definitely up for interface eth1, 10000 Mbps full duplex
[166260.590986] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang (XDP)
  Tx Queue             <27>
  TDH, TDT             <0>, <1>
  next_to_use          <1>
  next_to_clean        <0>
tx_buffer_info[next_to_clean]
  time_stamp           <0>
  jiffies              <109e45f40>
[166260.634855] ixgbe 0000:02:00.0 eth0: tx hang 318 detected on queue 27, resetting adapter
[166260.640482] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout
[166260.646200] ixgbe 0000:02:00.0 eth0: Reset adapter
[166261.448693] ixgbe 0000:02:00.0 eth0: speed changed to 0 for port eth0
[166261.472588] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
[166261.550794] bond1: link status up for interface eth0, enabling it in 200 ms
[166261.614974] ixgbe 0000:02:00.1 eth1: Detected Tx Unit Hang (XDP)
  Tx Queue             <47>
  TDH, TDT             <0>, <1>
  next_to_use          <1>
  next_to_clean        <0>
tx_buffer_info[next_to_clean]
  time_stamp           <0>
  jiffies              <109e46340>
[166261.659558] ixgbe 0000:02:00.1 eth1: tx hang 463 detected on queue 47, resetting adapter
[166261.665304] ixgbe 0000:02:00.1 eth1: initiating reset due to tx timeout
[166261.671080] ixgbe 0000:02:00.1 eth1: Reset adapter
[166262.477336] ixgbe 0000:02:00.1 eth1: speed changed to 0 for port eth1
[166262.484868] bond1: link status definitely up for interface eth0, 10000 Mbps full duplex
[166262.501209] ixgbe 0000:02:00.1 eth1: NIC Link is Up 10 Gbps, Flow Control: None
[166262.590807] bond1: link status up for interface eth1, enabling it in 200 ms
[166262.806862] bond1: link status definitely up for interface eth1, 10000 Mbps full duplex
[166275.886984] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang (XDP)
  Tx Queue             <27>
  TDH, TDT             <0>, <1>
  next_to_use          <1>
  next_to_clean        <0>
tx_buffer_info[next_to_clean]
  time_stamp           <0>
  jiffies              <109e49b00>
[166275.932956] ixgbe 0000:02:00.0 eth0: tx hang 319 detected on queue 27, resetting adapter
[166275.938915] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout
[166275.944927] ixgbe 0000:02:00.0 eth0: Reset adapter
[166276.746928] ixgbe 0000:02:00.0 eth0: speed changed to 0 for port eth0
[166276.770550] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
[166276.854810] bond1: link status up for interface eth0, enabling it in 200 ms
[166277.070856] bond1: link status definitely up for interface eth0, 10000 Mbps full duplex
[166283.054981] ixgbe 0000:02:00.1 eth1: Detected Tx Unit Hang (XDP)
  Tx Queue             <21>
  TDH, TDT             <0>, <1>
  next_to_use          <1>
  next_to_clean        <0>
tx_buffer_info[next_to_clean]
  time_stamp           <0>
  jiffies              <109e4b700>
[166283.100848] ixgbe 0000:02:00.1 eth1: tx hang 464 detected on queue 21, resetting adapter
[166283.106798] ixgbe 0000:02:00.1 eth1: initiating reset due to tx timeout
[166283.112803] ixgbe 0000:02:00.1 eth1: Reset adapter
[166283.920289] ixgbe 0000:02:00.1 eth1: speed changed to 0 for port eth1
[166283.943914] ixgbe 0000:02:00.1 eth1: NIC Link is Up 10 Gbps, Flow Control: None
[166284.022819] bond1: link status up for interface eth1, enabling it in 200 ms
[166284.238865] bond1: link status definitely up for interface eth1, 10000 Mbps full duplex
[166290.158984] ixgbe 0000:02:00.1 eth1: Detected Tx Unit Hang (XDP)
  Tx Queue             <47>
  TDH, TDT             <0>, <1>
  next_to_use          <1>
  next_to_clean        <0>
tx_buffer_info[next_to_clean]
  time_stamp           <0>
  jiffies              <109e4d2c0>
[166290.205560] ixgbe 0000:02:00.1 eth1: tx hang 465 detected on queue 47, resetting adapter
[166290.211646] ixgbe 0000:02:00.1 eth1: initiating reset due to tx timeout
[166290.217640] ixgbe 0000:02:00.1 eth1: Reset adapter
[166291.017944] ixgbe 0000:02:00.1 eth1: speed changed to 0 for port eth1
[166291.041900] ixgbe 0000:02:00.1 eth1: NIC Link is Up 10 Gbps, Flow Control: None
[166291.126803] bond1: link status up for interface eth1, enabling it in 200 ms
[166291.182977] ixgbe 0000:02:00.0 eth0: Detected Tx Unit Hang (XDP)
  Tx Queue             <27>
  TDH, TDT             <0>, <1>
  next_to_use          <1>
  next_to_clean        <0>
tx_buffer_info[next_to_clean]
  time_stamp           <0>
  jiffies              <109e4d6c0>
[166291.227494] ixgbe 0000:02:00.0 eth0: tx hang 320 detected on queue 27, resetting adapter
[166291.233223] ixgbe 0000:02:00.0 eth0: initiating reset due to tx timeout
[166291.238981] ixgbe 0000:02:00.0 eth0: Reset adapter
[166292.045660] ixgbe 0000:02:00.0 eth0: speed changed to 0 for port eth0
[166292.052867] bond1: link status definitely up for interface eth1, 10000 Mbps full duplex
[166292.068881] ixgbe 0000:02:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: None
[166292.158797] bond1: link status up for interface eth0, enabling it in 200 ms
[166292.374849] bond1: link status definitely up for interface eth0, 10000 Mbps full duplex

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 xdp qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
    link/ether 60:de:f3:4d:9f:6f brd ff:ff:ff:ff:ff:ff
    prog/xdp id 24 
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 xdp qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
    link/ether 60:de:f3:4d:9f:6f brd ff:ff:ff:ff:ff:ff
    prog/xdp id 25 
4: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
5: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 36:53:6f:4a:b5:03 brd ff:ff:ff:ff:ff:ff
6: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 60:de:f3:4d:9f:6f brd ff:ff:ff:ff:ff:ff

no packets captured on real server.

$ sudo tcpdump -ni any proto 4
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes

According to dmesg, tx hang is the point.

tehnerd · 2018-12-28T15:26:06Z

could you please show the output of

nproc
ethtool -l eth0
ethtool -l eth1

whl739 · 2018-12-28T15:27:57Z

$ nproc
56

$ ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:             0
TX:             0
Other:          1
Combined:       63
Current hardware settings:
RX:             0
TX:             0
Other:          1
Combined:       56

$ ethtool -l eth1
Channel parameters for eth1:
Pre-set maximums:
RX:             0
TX:             0
Other:          1
Combined:       63
Current hardware settings:
RX:             0
TX:             0
Other:          1
Combined:       56

tehnerd · 2018-12-28T15:30:43Z

see https://www.spinics.net/lists/netdev/msg519645.html for tx-hang. tl;dr it is a known issue of ixgbe driver. workaround (for it to be able to work in driver mode) is to set "combined + nproc < maximum_queues"
so could you set combined ("ethtool -L eth0 combined 6" "ethtool -L eth1 combined 6" and rerun the tests)

whl739 · 2018-12-28T15:40:14Z

It is exactly the reason, it works now.
Thank you very much.

tehnerd · 2018-12-28T15:44:25Z

yw. will close this issue for now (as it seems resolved)

Summary: After diving in all the build system I found that the first error mentioned in #219 and #220 ``` 1597 | static_assert(formattable_char, "Mixing character types is disallowed."); ``` Was basically happening while compiling folly, after compiling it by itself I noticed this didn't happened, so I found that there was an issue with the fmt dependencies, removing the one that was downloaded by katran The issue mentioned in: #221 Was because katran was configured to use C++14 by default, and some of the libraries of folly require C++17, updated our requirements. Test Plan: TEST Output: ``` Test project /home/ubuntu/ivanmorett/katran/_build/build/katran/lib/tests Start 1: IpHelpersTests.testV4ParsingBe 1/56 Test #1: IpHelpersTests.testV4ParsingBe ....................... Passed 0.01 sec Start 2: IpHelpersTests.testV4ParsingInt 2/56 Test #2: IpHelpersTests.testV4ParsingInt ...................... Passed 0.01 sec Start 3: IpHelpersTests.testV6ParsingBe 3/56 Test #3: IpHelpersTests.testV6ParsingBe ....................... Passed 0.01 sec Start 4: IpHelpersTests.testV6ParsingInt 4/56 Test #4: IpHelpersTests.testV6ParsingInt ...................... Passed 0.01 sec Start 5: IpHelpersTests.testIncorrectAddr 5/56 Test #5: IpHelpersTests.testIncorrectAddr ..................... Passed 0.01 sec Start 6: CHHelpersTest.testMaglevCHSameWeight 6/56 Test #6: CHHelpersTest.testMaglevCHSameWeight ................. Passed 0.01 sec Start 7: CHHelpersTest.testMaglevV2CHSameWeight 7/56 Test #7: CHHelpersTest.testMaglevV2CHSameWeight ............... Passed 0.01 sec Start 8: CHHelpersTest.testMaglevCHDiffWeight 8/56 Test #8: CHHelpersTest.testMaglevCHDiffWeight ................. Passed 0.01 sec Start 9: CHHelpersTest.testMaglevV2CHDiffWeight 9/56 Test #9: CHHelpersTest.testMaglevV2CHDiffWeight ............... Passed 0.01 sec Start 10: CHHelpersTest.testMaglevWeightsSumLargerThanRing 10/56 Test #10: CHHelpersTest.testMaglevWeightsSumLargerThanRing ..... Passed 0.01 sec Start 11: CHHelpersTest.testMaglevWeightsSumBelowRingSize 11/56 Test #11: CHHelpersTest.testMaglevWeightsSumBelowRingSize ...... Passed 0.01 sec Start 12: KatranLbTest.testChangeMac 12/56 Test #12: KatranLbTest.testChangeMac ........................... Passed 0.01 sec Start 13: KatranLbTest.testIfIndex 13/56 Test #13: KatranLbTest.testIfIndex ............................. Passed 0.01 sec Start 14: KatranLbTest.testVipHelpers 14/56 Test #14: KatranLbTest.testVipHelpers .......................... Passed 0.14 sec Start 15: KatranLbTest.testAddingInvalidVip 15/56 Test #15: KatranLbTest.testAddingInvalidVip .................... Passed 0.01 sec Start 16: KatranLbTest.testRealHelpers 16/56 Test #16: KatranLbTest.testRealHelpers ......................... Passed 0.01 sec Start 17: KatranLbTest.testRealFlags 17/56 Test #17: KatranLbTest.testRealFlags ........................... Passed 0.01 sec Start 18: KatranLbTest.testVipStatsHelper 18/56 Test #18: KatranLbTest.testVipStatsHelper ...................... Passed 0.01 sec Start 19: KatranLbTest.testLruStatsHelper 19/56 Test #19: KatranLbTest.testLruStatsHelper ...................... Passed 0.01 sec Start 20: KatranLbTest.testLruMissStatsHelper 20/56 Test #20: KatranLbTest.testLruMissStatsHelper .................. Passed 0.01 sec Start 21: KatranLbTest.testHcHelpers 21/56 Test #21: KatranLbTest.testHcHelpers ........................... Passed 0.01 sec Start 22: KatranLbTest.getVipFlags 22/56 Test #22: KatranLbTest.getVipFlags ............................. Passed 0.01 sec Start 23: KatranLbTest.getAllVips 23/56 Test #23: KatranLbTest.getAllVips .............................. Passed 0.01 sec Start 24: KatranLbTest.testUpdateRealsHelper 24/56 Test #24: KatranLbTest.testUpdateRealsHelper ................... Passed 0.07 sec Start 25: KatranLbTest.testUpdateQuicRealsHelper 25/56 Test #25: KatranLbTest.testUpdateQuicRealsHelper ............... Passed 0.06 sec Start 26: KatranLbTest.testUpdateQuicReal 26/56 Test #26: KatranLbTest.testUpdateQuicReal ...................... Passed 0.01 sec Start 27: KatranLbTest.getRealsForVip 27/56 Test #27: KatranLbTest.getRealsForVip .......................... Passed 0.01 sec Start 28: KatranLbTest.getHealthcheckersDst 28/56 Test #28: KatranLbTest.getHealthcheckersDst .................... Passed 0.01 sec Start 29: KatranLbTest.invalidAddressHandling 29/56 Test #29: KatranLbTest.invalidAddressHandling .................. Passed 0.01 sec Start 30: KatranLbTest.addInvalidSrcRoutingRule 30/56 Test #30: KatranLbTest.addInvalidSrcRoutingRule ................ Passed 0.01 sec Start 31: KatranLbTest.addValidSrcRoutingRuleV4 31/56 Test #31: KatranLbTest.addValidSrcRoutingRuleV4 ................ Passed 0.01 sec Start 32: KatranLbTest.addValidSrcRoutingRuleV6 32/56 Test #32: KatranLbTest.addValidSrcRoutingRuleV6 ................ Passed 0.01 sec Start 33: KatranLbTest.addMaxSrcRules 33/56 Test #33: KatranLbTest.addMaxSrcRules .......................... Passed 0.01 sec Start 34: KatranLbTest.delSrcRules 34/56 Test #34: KatranLbTest.delSrcRules ............................. Passed 0.01 sec Start 35: KatranLbTest.clearSrcRules 35/56 Test #35: KatranLbTest.clearSrcRules ........................... Passed 0.01 sec Start 36: KatranLbTest.addFewInvalidNets 36/56 Test #36: KatranLbTest.addFewInvalidNets ....................... Passed 0.01 sec Start 37: KatranLbTest.addInvalidDecapDst 37/56 Test #37: KatranLbTest.addInvalidDecapDst ...................... Passed 0.01 sec Start 38: KatranLbTest.addInvalidDecapDstNet 38/56 Test #38: KatranLbTest.addInvalidDecapDstNet ................... Passed 0.01 sec Start 39: KatranLbTest.addValidDecapDst 39/56 Test #39: KatranLbTest.addValidDecapDst ........................ Passed 0.01 sec Start 40: KatranLbTest.delValidDecapDst 40/56 Test #40: KatranLbTest.delValidDecapDst ........................ Passed 0.01 sec Start 41: KatranLbTest.delInvalidDecapDst 41/56 Test #41: KatranLbTest.delInvalidDecapDst ...................... Passed 0.01 sec Start 42: KatranLbTest.addMaxDecapDst 42/56 Test #42: KatranLbTest.addMaxDecapDst .......................... Passed 0.01 sec Start 43: VipTestF.testBatchUpdateReals 43/56 Test #43: VipTestF.testBatchUpdateReals ........................ Passed 0.04 sec Start 44: VipTestF.testBatchUpdateRealsWeight 44/56 Test #44: VipTestF.testBatchUpdateRealsWeight .................. Passed 0.05 sec Start 45: VipTestF.testGetRealsAndWeight 45/56 Test #45: VipTestF.testGetRealsAndWeight ....................... Passed 0.01 sec Start 46: VipTestF.testGetReals 46/56 Test #46: VipTestF.testGetReals ................................ Passed 0.02 sec Start 47: VipTest.testAddRemoveReal 47/56 Test #47: VipTest.testAddRemoveReal ............................ Passed 0.01 sec Start 48: EventPipeCallbackTest.SimpleCallbackTest 48/56 Test #48: EventPipeCallbackTest.SimpleCallbackTest ............. Passed 0.01 sec Start 49: EventPipeCallbackTest.LargeWriteTest 49/56 Test #49: EventPipeCallbackTest.LargeWriteTest ................. Passed 0.15 sec Start 50: TestMonitoringServiceCore.SimpleAcceptSubscription 50/56 Test #50: TestMonitoringServiceCore.SimpleAcceptSubscription ... Passed 0.01 sec Start 51: TestMonitoringServiceCore.SimpleErrors 51/56 Test #51: TestMonitoringServiceCore.SimpleErrors ............... Passed 0.01 sec Start 52: TestMonitoringServiceCore.EventIntersection 52/56 Test #52: TestMonitoringServiceCore.EventIntersection .......... Passed 0.01 sec Start 53: TestMonitoringServiceCore.RacingClients 53/56 Test #53: TestMonitoringServiceCore.RacingClients .............. Passed 0.01 sec Start 54: TestMonitoringServiceCore.SubscribeAndCancel 54/56 Test #54: TestMonitoringServiceCore.SubscribeAndCancel ......... Passed 0.01 sec Start 55: PcapWriterTest.SingleWriter 55/56 Test #55: PcapWriterTest.SingleWriter .......................... Passed 0.02 sec Start 56: PcapWriterTest.MultiWriter 56/56 Test #56: PcapWriterTest.MultiWriter ........................... Passed 0.01 sec 100% tests passed, 0 tests failed out of 56 Total Test time (real) = 1.04 sec + cd ../testing/ + ctest -v ./CMakeFiles ./CTestTestfile.cmake ./Makefile ./base64helpers-tests './base64helpers-tests[1]_include.cmake' './base64helpers-tests[1]_tests.cmake' ./cmake_install.cmake ./katran_tester ./libbase64_helpers.a ./libbpftester.a ./libkatran_test_provision.a ./libkatran_test_util.a ./libpcap_parser.a ctest: /usr/local/lib/libcurl.so.4: no version information available (required by ctest) Test project /home/ubuntu/ivanmorett/katran/_build/build/katran/lib/testing Start 1: Base64Tests.testEncode 1/2 Test #1: Base64Tests.testEncode ........... Passed 0.01 sec Start 2: Base64Tests.testDecode 2/2 Test #2: Base64Tests.testDecode ........... Passed 0.01 sec 100% tests passed, 0 tests failed out of 2 Total Test time (real) = 0.01 sec + popd ~/ivanmorett/katran/_build ``` Differential Revision: D55108012 Pulled By: lima1756

whl739 mentioned this issue Dec 25, 2018

katran with bonding #13

Closed

tehnerd closed this as completed Dec 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

katran xdp issue when running with bonding in driver mode #20

katran xdp issue when running with bonding in driver mode #20

whl739 commented Dec 25, 2018 •

edited

Loading

tehnerd commented Dec 26, 2018

whl739 commented Dec 26, 2018

whl739 commented Dec 28, 2018

tehnerd commented Dec 28, 2018

whl739 commented Dec 28, 2018

tehnerd commented Dec 28, 2018

whl739 commented Dec 28, 2018

tehnerd commented Dec 28, 2018

whl739 commented Dec 28, 2018

tehnerd commented Dec 28, 2018

katran xdp issue when running with bonding in driver mode #20

katran xdp issue when running with bonding in driver mode #20

Comments

whl739 commented Dec 25, 2018 • edited Loading

tehnerd commented Dec 26, 2018

whl739 commented Dec 26, 2018

whl739 commented Dec 28, 2018

tehnerd commented Dec 28, 2018

whl739 commented Dec 28, 2018

tehnerd commented Dec 28, 2018

whl739 commented Dec 28, 2018

tehnerd commented Dec 28, 2018

whl739 commented Dec 28, 2018

tehnerd commented Dec 28, 2018

whl739 commented Dec 25, 2018 •

edited

Loading