BpfAdapter.cpp:472] Error receiving netlink message: Operation not supported [95]

[yangxingwu]

No description provided.

[yangxingwu]

sudo ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/linux/bpfprog/bpf/balancer_kern.o -default_mac 08:00:27:dd:54:d3 -forwarding_cores=0 -healthchecker_prog ./deps/linux/bpfprog/bpf/healthchecking_ipip.o -intf=enp0s3 -ipip_intf=ipip0 -lru_size=100
I1220 07:56:55.643523  2753 KatranGrpcService.cpp:67] Starting Katran
E1220 07:56:55.648823  2753 BpfLoader.cpp:166] Can't read section size for index: 2
I1220 07:56:55.648860  2753 BpfLoader.cpp:448] Skipping section: 2 of file: ./deps/linux/bpfprog/bpf/balancer_kern.o
I1220 07:56:55.690763  2753 BpfLoader.cpp:367] relocation for non existing prog w/ idx 10
I1220 07:56:55.690811  2753 BpfLoader.cpp:367] relocation for non existing prog w/ idx 14
I1220 07:56:55.690815  2753 BpfLoader.cpp:367] relocation for non existing prog w/ idx 16
I1220 07:56:55.690819  2753 BpfLoader.cpp:367] relocation for non existing prog w/ idx 18
I1220 07:56:55.690821  2753 BpfLoader.cpp:367] relocation for non existing prog w/ idx 20
E1220 07:56:55.702682  2753 BpfLoader.cpp:166] Can't read section size for index: 2
I1220 07:56:55.702718  2753 BpfLoader.cpp:448] Skipping section: 2 of file: ./deps/linux/bpfprog/bpf/healthchecking_ipip.o
E1220 07:56:55.704231  2753 BpfLoader.cpp:166] Can't read section size for index: 12
I1220 07:56:55.704246  2753 BpfLoader.cpp:448] Skipping section: 12 of file: ./deps/linux/bpfprog/bpf/healthchecking_ipip.o
I1220 07:56:55.705085  2753 BpfLoader.cpp:367] relocation for non existing prog w/ idx 10
I1220 07:56:55.705112  2753 BpfLoader.cpp:367] relocation for non existing prog w/ idx 14
I1220 07:56:55.705127  2753 BpfLoader.cpp:367] relocation for non existing prog w/ idx 16
I1220 07:56:55.705130  2753 BpfLoader.cpp:367] relocation for non existing prog w/ idx 18
I1220 07:56:55.705132  2753 BpfLoader.cpp:367] relocation for non existing prog w/ idx 20
E1220 07:56:55.709900  2753 BpfLoader.cpp:103] Can't find map w/ name: lpm_src_v4
E1220 07:56:55.709923  2753 BpfLoader.cpp:103] Can't find map w/ name: decap_dst
E1220 07:56:55.710944  2753 BpfAdapter.cpp:472] Error receiving netlink message: Operation not supported [95]
terminate called after throwing an instance of 'std::invalid_argument'
  what():  can't attach main bpf prog to main inteface
*** Aborted at 1545292615 (Unix time, try 'date -d @1545292615') ***
*** Signal 6 (SIGABRT) (0xac1) received by PID 2753 (pthread TID 0x7f20bcd97900) (linux TID 2753) (maybe from PID 2753, UID 0), stack trace: ***
    @ 0000559633d22058 folly::symbolizer::(anonymous namespace)::innerSignalHandler(int, siginfo_t*, void*)
                       /home/vagrant/katran/deps/folly/folly/experimental/symbolizer/SignalHandler.cpp:423
    @ 0000559633d22123 folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       /home/vagrant/katran/deps/folly/folly/experimental/symbolizer/SignalHandler.cpp:438
    @ 00007f20bc97788f (unknown)
    @ 00007f20bbc87e97 gsignal
    @ 00007f20bbc89800 abort
    @ 00007f20bc4648b6 (unknown)
    @ 00007f20bc46aa05 (unknown)
    @ 00007f20bc46aa40 std::terminate()
    @ 00007f20bc46ac73 __cxa_throw
    @ 00005596340584e6 katran::KatranLb::attachBpfProgs()
                       /home/vagrant/katran/katran/lib/KatranLb.cpp:388
    @ 0000559633e847c2 lb::katran::KatranGrpcService::KatranGrpcService(katran::KatranConfig const&)
                       -> /home/vagrant/katran/deps/folly/folly/detail/IPAddress.cpp
    @ 0000559633bc57d0 RunServer(katran::KatranConfig&, int, std::shared_ptr<folly::EventBase>)
                       -> /home/vagrant/katran/deps/folly/folly/detail/IPAddress.cpp
    @ 0000559633bc5f8d main
                       -> /home/vagrant/katran/deps/folly/folly/detail/IPAddress.cpp
    @ 00007f20bbc6ab96 __libc_start_main
    @ 0000559633bc5409 _start
                       -> /home/vagrant/katran/deps/folly/folly/detail/IPAddress.cpp

[tehnerd]

What is the kernel version. Feels like it is to old and doesn’t have xdp support

[tehnerd]

No. Something else is going on. What is the driver of enp0s3 interface?
Could you please send the output of this two commands

1. ip link show
2. ethtool -i enp0s3

[yangxingwu]

the driver is virio_net

~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:48:37:b8:a5:68 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:b7:72:ef brd ff:ff:ff:ff:ff:ff

~$ ethtool -i enp0s8
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:00:08.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

[yangxingwu]

kernel version .

~$ uname -r
4.15.0-42-generic

[tehnerd]

could you please show output of "dmesg| tail -50" right after you try to run katran. looking on viriton_net driver - it could return EINVAL only in this two cases:

        if (vi->mergeable_rx_bufs && !vi->any_header_sg) {
                NL_SET_ERR_MSG_MOD(extack, "XDP expects header/data in single page, any_header_sg required");
                return -EINVAL;
        }

        if (dev->mtu > max_sz) {
                NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP");
                netdev_warn(dev, "XDP requires MTU less than %lu\n", max_sz);
                return -EINVAL;
        }

so want to figure out which on is it

[tehnerd]

oh. wait. my bad; it is EOPNOTSUP , not EIVAL. and this is happens when LRO is enabled on the interface:

  if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS)
            && (virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_TSO4) ||
                virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_TSO6) ||
                virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_ECN) ||
                virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_UFO))) {
                NL_SET_ERR_MSG_MOD(extack, "Can't set XDP while host is implementing LRO, disable LRO first");
                return -EOPNOTSUPP;
        }

could you please show

1. ethtool -k enp0s3
2. disabled lro with "ethtool -K enp0s3 lro off"

[yangxingwu]

~$ sudo ethtool -k enp0s8
Features for enp0s8:
rx-checksumming: off
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp-mangleid-segmentation: off
	tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on [fixed]

large receive offload is disabled

and also the error message NL_SET_ERR_MSG_MOD(extack, "Can't set XDP while host is implementing LRO, disable LRO first"); is not printed

[tehnerd]

You are showing output from enp0s8 but trying to attach to the enp0s3. Also could you please disable all other offloads from the virtio code (ethtool -K enp0s3 tso off lro off ) and try again

[tehnerd]

ok. i've tried virtio on virtualbox (i assume you are using vagrant with it) and indeed it fails on codepath, described above. even with all offloads disabled. most likely some weird interaction between virtio driver @ virtualbox and gues vm. at this point for testing i'd just recomend to move away from virtion to something else (e.g. e100/e1000 etc)

[yangxingwu]

yes, I'm using vagrant and it always failed...

ok, I will move to e1000

[tehnerd]

it future we would add explicit fallback to "generic xdp mode" (which is NIC driver agnostic). for now virtio on virtualbox has an issue and unfortunately you need to use another driver for tests

[tehnerd]

with latest commit there is an option to force run xdp in any mode (e.g. in generic) even if driver claims that it is supporting "driver" mode.