A restart of FRR does not configure the loopback class E address if it already exists

[olivettiwetti]

Description

In our LAB we observe the following: We configure a loopback address via frr.conf that is to be announced. When we "start" the system (after a reboot, or just start the first time the FRR), the announcing works fine and we see on a peer the ip address.
With a restart of FRR this announcing fails and is to be reset manually, either by removing the IP via vty or via ip command, followed by re-applying the configuration.

Example configuration:

...
access-list localsite seq 10 permit 240.xxx.xxx.215/32

route-map lanroutes permit 10
  match ip address localsite

interface lo
  ! probing IP on loopback
  ip address 240.xxx.xxx.215/32 label lo:1
  ip ospf passive
  
ospf
  ...
  redistribute connected metric-type 1 route-map lanroutes
  redistribute static metric-type 1 route-map lanroutes

With a fresh start of the routing suite, the configuration is good:

[devuan-host1:~]# ip addr sh dev lo label lo:1
    inet 240.xxx.xxx.215/32 brd 240.xxx.xxx.215 scope global lo:1
       valid_lft forever preferred_lft forever

... and also learned by a peer:

[ospf-peer2:~]# vtysh -c 'sh ip route ospf' | grep 10.xxx.xxx.10
O>* 27.xxx.xxx.24/29 [110/70] via 10.xxx.xxx.10, eth4, weight 1, 00:02:40
O>* 192.xxx.xxx.0/24 [110/70] via 10.xxx.xxx.10, eth4, weight 1, 00:02:40
O>* 240.xxx.xxx.215/32 [110/70] via 10.xxx.xxx.10, eth4, weight 1, 00:02:40      # <- the route under discussion

After a restart of our routing suite, the route for 240.xxx.xxx.215/32 is missing, though still configured:

# restart FRR
[devuan-host1:~]# /etc/init.d/frr restart
# loopback still configured
[devuan-host1:~]# ip addr sh dev lo label lo:1
    inet 240.80.48.215/32 brd 240.80.48.215 scope global lo:1
       valid_lft forever preferred_lft forever

# but not announced
[ospf-peer2:~]# vtysh -c 'sh ip route ospf' | grep 10.xxx.xxx.10
O>* 27.xxx.xxx.24/29 [110/70] via 10.xxx.xxx.10, eth4, weight 1, 00:01:45
O>* 192.xxx.xxx.0/24 [110/70] via 10.xxx.xxx.10, eth4, weight 1, 00:01:45

Version

The version we run is a devuan system (Excalibur) with FRR v10.0-2:

[devuan-host1:~]# vtysh

Hello, this is FRRouting (version 10.0).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

devuan-host1# sh version
FRRouting 10.0 (devuan-host1) on Linux(6.8.12-amd64).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
    '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--sbindir=/usr/lib/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--enable-rpki' '--disable-scripting' '--disable-pim6d' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'LIBS= -latomic' 'PYTHON=python3'
devuan-host1#

### How to reproduce

See description:
* configure a loopback address that is to be announced
* restart the routing suite on the system you announce the loopback address

### Expected behavior

Expected is to see that loopback address announced, as the other networks/addresses are announced.

### Actual behavior

After the restart of the suite, the configured loopback address is not announced anymore.

### Additional context

_No response_

### Checklist

- [X] I have searched the open issues for this bug.
- [X] I have not included sensitive information in this report.

[aceelindem]

I don't see how this worked prior to restart - the class E address is not accepted on the interface.

acee-ubuntu(config)# interface lo
acee-ubuntu(config-if)# ip address 240.1.1.215/32
% Configuration failed.

invalid address 240.1.1.215/32

[aceelindem]

Please include on devuan-host1 before and after restart:

show ip ospf interface
show ip ospf database router

[olivettiwetti]

That's a good point, thanks for the input. We are using the documented option "allow-reserved-ranges" which actually allows the use of class E.
But I can confirm: Using a different address, not "class E", this would indeed work "normally". We would like to keep this class E prefix, if possible. It worked before migrating to FRR (we used Quagga before).

[olivettiwetti]

The output of the requested commands:

Command 1 BEFORE and AFTER restart: command-1.txt
Command 2 BEFORE and AFTER restart: command-2.txt

Here just the diff:
Command 1:

12c12
<     Hello due in 2.235s
---
>     Hello due in 0.411s

Command 2:

7c7
<   LS age: 86
---
>   LS age: 38
14,15c14,15
<   LS Seq Number: 80000006
<   Checksum: 0xa071
---
>   LS Seq Number: 80000007
>   Checksum: 0x9e72
27c27
<   LS age: 91
---
>   LS age: 44
34,35c34,35
<   LS Seq Number: 80000025
<   Checksum: 0xd880
---
>   LS Seq Number: 80000027
>   Checksum: 0xd482

[aceelindem]

Why are you only posting diffs? If this is truly only the differences, this would imply that the interface still is announced and the problem is something else (possibly the Class E is not accepted by ospf-peer2.

[olivettiwetti]

I have intentionally written only the diff as text because I want to show that there is no difference. The output of the command of course would be attached in the links I shared. I realised that attaching files as link is suboptimal. I apologise for this.

Here is the output as text.

Command 1: show ip ospf interface:

###BEFORE
[devuan-host1~]# vtysh -c 'sh ip ospf interface'
eth0 is up
  ifindex 2, MTU 1500 bytes, BW 0 Mbit <UP,BROADCAST,RUNNING,MULTICAST>
  Internet Address 10.xxx.xxx.10/24, Broadcast 10.xxx.xxx.255, Area 0.0.0.0
  MTU mismatch detection: enabled
  Router ID 0.0.56.161, Network Type BROADCAST, Cost: 50
  Transmit Delay is 1 sec, State Backup, Priority 1
  Designated Router (ID) 213.xxx.xxx.48 Interface Address 10.xxx.xxx.1/24
  Backup Designated Router (ID) 0.0.56.161, Interface Address 10.xxx.xxx.10
  Multicast group memberships: OSPFAllRouters OSPFDesignatedRouters
  Timer intervals configured, Hello 10s, Dead 40s, Wait 40s, Retransmit 5
    Hello due in 2.235s
  Neighbor Count is 1, Adjacent neighbor count is 1
  Graceful Restart hello delay: 10s

###AFTER
[devuan-host1~]# vtysh -c 'sh ip ospf interface'
eth0 is up
  ifindex 2, MTU 1500 bytes, BW 0 Mbit <UP,BROADCAST,RUNNING,MULTICAST>
  Internet Address 10.xxx.xxx.10/24, Broadcast 10.xxx.xxx.255, Area 0.0.0.0
  MTU mismatch detection: enabled
  Router ID 0.0.56.161, Network Type BROADCAST, Cost: 50
  Transmit Delay is 1 sec, State Backup, Priority 1
  Designated Router (ID) 213.xxx.xxx.48 Interface Address 10.xxx.xxx.1/24
  Backup Designated Router (ID) 0.0.56.161, Interface Address 10.xxx.xxx.10
  Multicast group memberships: OSPFAllRouters OSPFDesignatedRouters
  Timer intervals configured, Hello 10s, Dead 40s, Wait 40s, Retransmit 5
    Hello due in 0.411s
  Neighbor Count is 1, Adjacent neighbor count is 1
  Graceful Restart hello delay: 10s

And command 2: show ip ospf database router

###BEFORE
[devuan-host1~]# vtysh -c 'sh ip ospf database router'

       OSPF Router with ID (0.0.56.161)


                Router Link States (Area 0.0.0.0)

  LS age: 86
  Options: 0x2  : *|-|-|-|-|-|E|-
  LS Flags: 0x3
  Flags: 0x2 : ASBR
  LS Type: router-LSA
  Link State ID: 0.0.56.161
  Advertising Router: 0.0.56.161
  LS Seq Number: 80000006
  Checksum: 0xa071
  Length: 36

   Number of Links: 1

    Link connected to: a Transit Network
     (Link ID) Designated Router address: 10.xxx.xxx.1
     (Link Data) Router Interface address: 10.xxx.xxx.10
      Number of TOS metrics: 0
       TOS 0 Metric: 50


  LS age: 91
  Options: 0x2  : *|-|-|-|-|-|E|-
  LS Flags: 0x6
  Flags: 0x2 : ASBR
  LS Type: router-LSA
  Link State ID: 213.xxx.xxx.48
  Advertising Router: 213.xxx.xxx.48
  LS Seq Number: 80000025
  Checksum: 0xd880
  Length: 108

   Number of Links: 7

    Link connected to: a Transit Network
     (Link ID) Designated Router address: 10.xxx.xxx.1
     (Link Data) Router Interface address: 10.xxx.xxx.1
      Number of TOS metrics: 0
       TOS 0 Metric: 50

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.41
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 180

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.53
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 185

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.57
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 185

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.59
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 190

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.17
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 100

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.18
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 105


###AFTER
[devuan-host1~]# vtysh -c 'sh ip ospf database router'

       OSPF Router with ID (0.0.56.161)


                Router Link States (Area 0.0.0.0)

  LS age: 38
  Options: 0x2  : *|-|-|-|-|-|E|-
  LS Flags: 0x3
  Flags: 0x2 : ASBR
  LS Type: router-LSA
  Link State ID: 0.0.56.161
  Advertising Router: 0.0.56.161
  LS Seq Number: 80000007
  Checksum: 0x9e72
  Length: 36

   Number of Links: 1

    Link connected to: a Transit Network
     (Link ID) Designated Router address: 10.xxx.xxx.1
     (Link Data) Router Interface address: 10.xxx.xxx.10
      Number of TOS metrics: 0
       TOS 0 Metric: 50


  LS age: 44
  Options: 0x2  : *|-|-|-|-|-|E|-
  LS Flags: 0x6
  Flags: 0x2 : ASBR
  LS Type: router-LSA
  Link State ID: 213.xxx.xxx.48
  Advertising Router: 213.xxx.xxx.48
  LS Seq Number: 80000027
  Checksum: 0xd482
  Length: 108

   Number of Links: 7

    Link connected to: a Transit Network
     (Link ID) Designated Router address: 10.xxx.xxx.1
     (Link Data) Router Interface address: 10.xxx.xxx.1
      Number of TOS metrics: 0
       TOS 0 Metric: 50

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.41
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 180

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.53
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 185

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.57
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 185

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.59
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 190

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.17
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 100

    Link connected to: Stub Network
     (Link ID) Net: 213.xxx.xxx.18
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metric: 105

[olivettiwetti]

Maybe you are interested in this output?

devuan-host1# conf t
devuan-host1(config)# allow-reserved-ranges
devuan-host1(config)# interface lo
devuan-host1(config-if)# ip address 240.1.1.215/32 label lo:2
devuan-host1(config-if)# end
devuan-host1# show ip ospf interface lo
lo is up
  ifindex 1, MTU 65536 bytes, BW 0 Mbit <UP,LOOPBACK,RUNNING>
  OSPF not enabled on this interface

devuan-host1# exit

[devuan-host1:~]# ip addr sh dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 240.xxx.xxx.215/32 brd 240.xxx.xxx.215 scope global lo:1
       valid_lft forever preferred_lft forever
    inet 240.1.1.215/32 brd 240.1.1.215 scope global lo:2          # <- now also available on the system itself
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo
       valid_lft forever preferred_lft forever

The address itself is configured.

Let me find some debug lines. I guess to reproduce you only succeed when using the mentioned option "allow-reserved-ranges" to allow the class E configuration on that interface.

[olivettiwetti]

I found something interesting:
Listening on the netlink socket (nlmon), I saw the message: ipv4: Address already assigned. This made me curious about some other states:
When starting frr, the address is not here, so it's created and I see it in the output:

devuan-host1# show interface lo
Interface lo is up, line protocol is up
  Link ups:       0    last: (never)
  Link downs:     0    last: (never)
  vrf: default
  index 1 metric 0 mtu 65536 speed 0 txqlen 1000
  flags: <UP,LOOPBACK,RUNNING>
  Type: Loopback
  inet 240.xxx.xxx.215/32 lo:1
  Interface Type Other
  Interface Slave Type None
  protodown: off

Now restarting the daemon, it tries to re-create it? For whatever reason, it is not configured anymore on the interface:

devuan-host1# show interface lo
Interface lo is up, line protocol is up
  Link ups:       0    last: (never)
  Link downs:     0    last: (never)
  vrf: default
  index 1 metric 0 mtu 65536 speed 0 txqlen 1000
  flags: <UP,LOOPBACK,RUNNING>
  Type: Loopback
  Interface Type Other
  Interface Slave Type None
  protodown: off

Although the "running config" would say it so:

[devuan-host1:~]# vtysh -c 'sh run' | grep -A2 "interface lo"
interface lo
 ip address 240.xxx.xxx.215/32 label lo:1
 ip ospf passive

Now trying to "reconfigure" it via vtysh doesn't help, it is just not "configured" anymore on the interface lo. I also don't see a netlink socket message, I have to unconfigure it as mentioned, either via ip command or via vtysh.

To be added: yes, it behaves different, when using the ip 200.xxx.xxx.215/32, then it would be still configured on the interface.

I keep digging, still don't see it.

[aceelindem]

I found something interesting: Listening on the netlink socket (nlmon), I saw the message: ipv4: Address already assigned. This made me curious about some other states: When starting frr, the address is not here, so it's created and I see it in the output:

devuan-host1# show interface lo
Interface lo is up, line protocol is up
  Link ups:       0    last: (never)
  Link downs:     0    last: (never)
  vrf: default
  index 1 metric 0 mtu 65536 speed 0 txqlen 1000
  flags: <UP,LOOPBACK,RUNNING>
  Type: Loopback
  inet 240.xxx.xxx.215/32 lo:1
  Interface Type Other
  Interface Slave Type None
  protodown: off

Now restarting the daemon, it tries to re-create it? For whatever reason, it is not configured anymore on the interface:

devuan-host1# show interface lo
Interface lo is up, line protocol is up
  Link ups:       0    last: (never)
  Link downs:     0    last: (never)
  vrf: default
  index 1 metric 0 mtu 65536 speed 0 txqlen 1000
  flags: <UP,LOOPBACK,RUNNING>
  Type: Loopback
  Interface Type Other
  Interface Slave Type None
  protodown: off

Although the "running config" would say it so:

[devuan-host1:~]# vtysh -c 'sh run' | grep -A2 "interface lo"
interface lo
 ip address 240.xxx.xxx.215/32 label lo:1
 ip ospf passive

Now trying to "reconfigure" it via vtysh doesn't help, it is just not "configured" anymore on the interface lo. I also don't see a netlink socket message, I have to unconfigure it as mentioned, either via ip command or via vtysh.

To be added: yes, it behaves different, when using the ip 200.xxx.xxx.215/32, then it would be still configured on the interface.

I keep digging, still don't see it.

I think you have found the root cause of the problem. If the interface address is not in FRR. It will not be advertised in OSPF.

[olivettiwetti]

With having enabled dplane and kernel msgdump debug informations, the log is this:

devuan-host-1 zebra[3683776]: [JVY1P-93VFY] dplane provider 'Kernel': processing
devuan-host-1 zebra[3683776]: [MVWHC-JAWC6] Dplane intf ADDR_INSTALL, idx 1, addr 240.xxx.xxx.215/32
devuan-host-1 zebra[3671450]: [TJ327-ET8HE] netlink_send_msg: >> netlink message dump [sent]
devuan-host-1 zebra[3671450]: [JAS4D-NCWGP] nlmsghdr [len=52 type=(20) NEWADDR flags=(0x0001) {REQUEST} seq=31 pid=3104268297]
devuan-host-1 zebra[3671450]: [KD75T-V88HS]   ifa [family=(2) AF_INET prefixlen=32 flags=0x0000 {} scope=0 index=1]
devuan-host-1 zebra[3671450]: [KFBSR-XYJV1]     rta [len=8 (payload=4) type=(2) LOCAL]
devuan-host-1 zebra[3671450]: [M8QV4-KY9C0]       240.xxx.xxx.215
devuan-host-1 zebra[3671450]: [KFBSR-XYJV1]     rta [len=8 (payload=4) type=(4) BROADCAST]
devuan-host-1 zebra[3671450]: [M8QV4-KY9C0]       240.xxx.xxx.215
devuan-host-1 zebra[3671450]: [KFBSR-XYJV1]     rta [len=9 (payload=5) type=(3) LABEL]
devuan-host-1 zebra[3671450]: [V74GD-NYS6Y]       lo:1
devuan-host-1 zebra[3671450]: [V8KNF-8EXH8] netlink_recv_msg: << netlink message dump [recv]
devuan-host-1 zebra[3671450]: [JAS4D-NCWGP] nlmsghdr [len=36 type=(2) ERROR flags=(0x0100) {DUMP,(ROOT|REPLACE|CAPPED)} seq=31 pid=3104268297]
devuan-host-1 zebra[3671450]: [KWP1C-6CSXF]   nlmsgerr [error=(-17) File exists]
devuan-host-1 zebra[3671450]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: File exists, type=RTM_NEWADDR(20), seq=31, pid=3104268297

This netlink message is only sent with this 240-address. It is not sent, when using the 200-address.

I then went again through all the code and still could't pinpoint the line that is triggering the re-add of the address in case it's this class E address. Do you have me a hint, where else I should look into then in the dplane?
Is a flag different on this class E address?
I'll keep digging.

[olivettiwetti]

I changed the title, it's not OSPF related.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.

[frrbot]

This issue will be automatically closed in the specified period unless there is further activity.

[olivettiwetti]

Please do not close. I have a vague suspicion:

Not sure, if the following check inside ipv4_unicast_valid() is working as expected: I removed the following part and the IP remained configured.

--- a/lib/prefix.c      2023-12-01 08:51:27.000000000 +0100
+++ b/lib/prefix.c      2025-01-30 00:28:25.858185164 +0100
@@ -1411,13 +1411,6 @@
        if (IPV4_CLASS_D(ip))
                return false;

-       if (IPV4_CLASS_E(ip)) {
-               if (cmd_allow_reserved_ranges_get())
-                       return true;
-               else
-                       return false;
-       }
-
        return true;
 }

Could it be, that the cmd_allow_reserved_ranges_get() is not yet available in this moment?

[frrbot]

This issue will no longer be automatically closed.

[olivettiwetti]

Indeed. I added a log line instead of the true/false:

--- a/lib/prefix.c
+++ b/lib/prefix.c
@@ -1413,9 +1413,9 @@

    if (IPV4_CLASS_E(ip)) {
        if (cmd_allow_reserved_ranges_get())
-           return true;
+           flog_warn(EC_LIB_DEVELOPMENT, "olivettiwetti: CLASS E true");
        else
-           return false;
+           flog_warn(EC_LIB_DEVELOPMENT, "olivettiwetti: CLASS E false");
    }

    return true;

Though I have in my config file the allow-reserved-ranges, which should result in true, I received:

2025/01/30 00:52:52 ZEBRA: [WZRMY-ZF6W9][EC 100663307] olivettiwetti: CLASS E false

[olivettiwetti]

I have only now realized that the issue #17942 got linked. My bad.
I only had the message from frrbot in mind, the periodic cleaner that this issue is automatically closed on inactivity. And than I restarted investing again.
Thank you for bringing this up in a discussion 🙏.