Keepalived cannot handle multiple incoming/outgoing interfaces and unicast

[vincentbernat]

Describe the bug
When using unicast_peer, I would expect Keepalived to use system routes to contact the peer. However, Keepalived only sends and receives using the main interface. Therefore, if the peer is not reachable through this interface or if the remote peer sends VRRP announces through another interface, Keepalived gets confused.

To Reproduce
On host 1:

ip addr add 203.0.113.10/26 dev eth0
ip addr add 203.0.113.70/26 dev eth1
ip addr add 192.0.2.1/32 dev lo
ip route add default nexthop via 203.0.113.1 nexthop via 203.0.113.65

On host 2:

ip addr add 203.0.113.140/26 dev eth0
ip addr add 203.0.113.200/26 dev eth1
ip addr add 192.0.2.2/32 dev lo
ip route add default nexthop via 203.0.113.129 nexthop via 203.0.113.193

It is assumed there is a router in the middle routing the four subnets appropriately.

Configuration for host 1:

vrrp_instance K1v4 {
  state BACKUP
  virtual_router_id 11
  priority 101
  advert_int 1
  interface eth0
  unicast_src_ip 192.0.2.1
  unicast_peer {
    192.0.2.2
  }
  virtual_ipaddress {
    192.0.2.255/32 dev lo
  }
}

Configuration for host 2:

vrrp_instance K2v4 {
  state BACKUP
  virtual_router_id 11
  priority 100
  advert_int 1
  interface eth0
  unicast_src_ip 192.0.2.2
  unicast_peer {
    192.0.2.1
  }
  virtual_ipaddress {
    192.0.2.255/32 dev lo
  }
}

Expected behavior
I would expect to not have to specify interface eth0 since it is not needed with unicast (though I understand why Keepalived actually needs it since it is using raw sockets for both sending and receiving).

Keepalived version

Copyright(C) 2001-2019 Alexandre Cassen, <acassen@gmail.com>

Built with kernel headers for Linux 5.4.19
Running on Linux 5.6.0-1-amd64 #1 SMP Debian 5.6.7-1 (2020-04-29)

configure options: --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --with-kernel-dir=debian/ --enable-snmp --enable-sha1 --enable-snmp-rfcv2 --enable-snmp-rfcv3 --enable-dbus --enable-json --enable-bfd --enable-regex build_alias=x86_64-linux-gnu CFLAGS=-g -O2 -fdebug-prefix-map=/build/keepalived-nKqjfO/keepalived-2.0.19=. -fstack-protector-strong -Wformat -Werror=format-security LDFLAGS=-Wl,-z,relro CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2

Config options:  NFTABLES LVS REGEX VRRP VRRP_AUTH JSON BFD OLD_CHKSUM_COMPAT FIB_ROUTING SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 DBUS

System options:  PIPE2 SIGNALFD INOTIFY_INIT1 VSYSLOG EPOLL_CREATE1 IPV4_DEVCONF IPV6_ADVANCED_API LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_OIFNAME FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS IP_MULTICAST_ALL LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION LIBIPVS_NETLINK IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_VMAC VRRP_IPVLAN IFLA_LINK_NETNSID CN_PROC SOCK_NONBLOCK SOCK_CLOEXEC O_PATH GLOB_BRACE INET6_ADDR_GEN_MODE VRF SO_MARK SCHED_RT SCHED_RESET_ON_FORK

Distro

• Debian unstable on amd64

System Log entries
In my case, both host 1 and host 2 are entering MASTER state because they don't see each other advertisement because they happen to flow on the second interface (eth1).
Here is what I get from eth0 on host 1:

16:43:51.343324 IP 192.0.2.1 > 192.0.2.2: VRRPv2, Advertisement, vrid 11, prio 101, authtype none, intvl 1s, length 20
16:43:52.343514 IP 192.0.2.1 > 192.0.2.2: VRRPv2, Advertisement, vrid 11, prio 101, authtype none, intvl 1s, length 20

And on eth1:

16:44:06.958406 IP 192.0.2.2 > 192.0.2.1: VRRPv2, Advertisement, (ttl 254), vrid 11, prio 100, authtype none, intvl 1s, length 20
16:44:06.958449 IP 192.0.2.1 > 192.0.2.2: ICMP 192.0.2.1 protocol 112 unreachable, length 48
16:44:07.347534 IP 192.0.2.2 > 192.0.2.1: ICMP 192.0.2.2 protocol 112 unreachable, length 48

First is the advertisement from host 2, second is the answer from host 1 because no VRRP handler is registered on eth1 and the second is the same answer from host 2.

[vincentbernat]

For reference, here is the complete setup:

• https://github.com/vincentbernat/network-lab/blob/bfa33aaed642814aa218e2dae219d36167d5dac6/lab-keepalived-unicast/setup
• https://github.com/vincentbernat/network-lab/blob/bfa33aaed642814aa218e2dae219d36167d5dac6/lab-keepalived-unicast/keepalived.K1.conf
• https://github.com/vincentbernat/network-lab/blob/bfa33aaed642814aa218e2dae219d36167d5dac6/lab-keepalived-unicast/keepalived.K2.conf

[pqarmitage]

@vincentbernat I have nearly completed the work on this, but it would be very helpful if you could test what I have done so far.

The only outstanding functionality at the moment, so far as I am aware, is properly handling the unicast src address being deleted from or added to interfaces and appropriately leaving or going to fault state.

I do need to further check the changes for SNMP, DBus and JSON for not having an interface specified for the VRRP instance; I suspect just using an ifindex of 0 is not the right thing to do. If you do have an opportunity to look at the SNMP changes and comment on those, it would be very helpful.

I have implemented what you suggested of not specifying an interface when you want to use kernel routing for sending adverts, so for your scenario you must not specify interface on the vrrp instances.

The attached patch applies to commit 1344729 - it isn't yet updated for the transition to list_head
1569-interim.patch.txt

[vincentbernat]

Oh, I didn't thought you would have made the change! In the meantime, I am just using VXLAN with unicast flooding to workaround the issue. I'll test your patch and look at the SNMP part during this week-end.

[pqarmitage]

Once I understood what the issue was I realised that it was quite pertinant; I just didn't realise how big an impact it would have no longer having an interface for a vrrp instance.

While implementing this change, it occurred to me that using ifindex as part of an OID may be dubious (this applies to the RFC MIBs as well). If an interface is deleted (e.g. a hot-swap interface card or USB device is removed) and then recreated, the ifindex of that interface will change oven though the interface name doesn't change. My question is: is it right that the OID should change, or should the OID remain the same?

[vincentbernat]

Most vendors, ensure the ifindex does not change for an interface by keeping some persistence file. This is the default for Juniper and this has to be enabled on Cisco IOS (and I think this is the default on more modern versions).

Net-SNMP doesn't have this feature so people should be used for the OID to change when the index change. This happens for IF-MIB as well. I am not using SNMP a lot with Linux servers anymore, but I don't remember this was an issue in the past with the tools I have been using.

[vincentbernat]

Your patch works for me. As for Net-SNMP, I am currently unable to compile Keepalived with SNMP support due to #include "parser.h" including files from Perl instead of Keepalived. Is that a known problem? Net-SNMP is a pain with respect on how brain-dead it is to link with. I'll try to propose a non-intrusive fix.

[pqarmitage]

Commit 557dfbe adds the ability to omit the interface for unicast peers.

This does not yet handle going to FAULT state if the unicast_src_ip is deleted from the system while keepalived is running, or leaving FAULT state if the unicast_src_ip is added. At the moment if the unicast_src_ip does not exist when keepalived starts, the vrrp instance will go to FAULT state and never leave it.

[robmbrooks]

@pqarmitage is there are resolution to this issue?

[vincentbernat]

As a workaround, you can use VXLAN to emulate a L2 segment between the keepalived peers.