Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Destructive Chaos Testing: remote forwarder healing test #1695

Closed
Mixaster995 opened this issue Jun 8, 2021 · 2 comments
Closed

Destructive Chaos Testing: remote forwarder healing test #1695

Mixaster995 opened this issue Jun 8, 2021 · 2 comments
Assignees
@Mixaster995
Labels
ASAP The issue should be completed as soon as possible bug Something isn't working Planning Issue related to SOW

Comments

@Mixaster995
Copy link
Contributor

Mixaster995 commented Jun 8, 2021 •
edited by huly-github-staging bot
Loading

Issue

In remote forwarder healing scenario, when we kill remote forwarder and then, after it's restored, trying to ping from nsc to nse again it is not working.

part of #1174
@Mixaster995 Mixaster995 self-assigned this Jun 8, 2021
@denis-tingaikin denis-tingaikin added ASAP The issue should be completed as soon as possible bug Something isn't working Planning Issue related to SOW labels Jun 8, 2021
@Mixaster995
Copy link
Contributor Author

  • Logs from all pods after first incorrect ping: logs.zip

  • Additional info from forwarders

  1. Local:
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
4: veth8ae528ad@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 7e:e9:b8:4b:c8:01 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 10.244.1.1/32 scope global veth8ae528ad
       valid_lft forever preferred_lft forever
5: veth1efc7c41@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 0a:64:f9:6c:7b:3b brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet 10.244.1.1/32 scope global veth1efc7c41
       valid_lft forever preferred_lft forever
6: veth5005d5fe@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether e2:bc:88:cb:3a:0e brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet 10.244.1.1/32 scope global veth5005d5fe
       valid_lft forever preferred_lft forever
7: vetha3e5b348@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 52:f7:d5:61:e5:58 brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet 10.244.1.1/32 scope global vetha3e5b348
       valid_lft forever preferred_lft forever
8: veth6c20b8e1@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether fa:73:e5:33:a8:a1 brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet 10.244.1.1/32 scope global veth6c20b8e1
       valid_lft forever preferred_lft forever
90: eth0@if91: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:1d:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.29.0.4/16 brd 172.29.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::4/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe1d:4/64 scope link 
       valid_lft forever preferred_lft forever

# ip route 
default via 172.29.0.1 dev eth0 
10.244.0.0/24 via 172.29.0.2 dev eth0 
10.244.1.2 dev veth8ae528ad scope host 
10.244.1.3 dev veth1efc7c41 scope host 
10.244.1.4 dev veth5005d5fe scope host 
10.244.1.5 dev vetha3e5b348 scope host 
10.244.1.6 dev veth6c20b8e1 scope host 
10.244.2.0/24 via 172.29.0.3 dev eth0 
172.29.0.0/16 dev eth0 proto kernel scope link src 172.29.0.4

# ip link 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/tunnel6 :: brd ::
4: veth8ae528ad@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 7e:e9:b8:4b:c8:01 brd ff:ff:ff:ff:ff:ff link-netnsid 1
5: veth1efc7c41@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 0a:64:f9:6c:7b:3b brd ff:ff:ff:ff:ff:ff link-netnsid 2
6: veth5005d5fe@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether e2:bc:88:cb:3a:0e brd ff:ff:ff:ff:ff:ff link-netnsid 3
7: vetha3e5b348@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 52:f7:d5:61:e5:58 brd ff:ff:ff:ff:ff:ff link-netnsid 4
8: veth6c20b8e1@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether fa:73:e5:33:a8:a1 brd ff:ff:ff:ff:ff:ff link-netnsid 5
90: eth0@if91: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:1d:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0

# vppctl show int addr
host-eth0 (up):
  L3 172.29.0.4/16
local0 (dn):

# vppctl show hardware   
              Name                Idx   Link  Hardware
host-eth0                          1     up   host-eth0
  Link speed: unknown
  Ethernet address 02:42:ac:1d:00:04
  Linux PACKET socket interface
local0                             0    down  local0
  Link speed: unknown
  local

# vppctl show ip neighbor
    Time                       IP                    Flags      Ethernet              Interface       
      1.2591               172.29.0.1                  S    02:42:a0:b4:47:d9 host-eth0
      1.2599               172.29.0.2                  S    02:42:ac:1d:00:02 host-eth0
      1.2596               172.29.0.3                  S    02:42:ac:1d:00:03 host-eth0

# ip neighbor | grep 172.29.0.1    
172.29.0.1 dev eth0 lladdr 02:42:a0:b4:47:d9 STALE
# ip neighbor | grep 172.29.0.2 
172.29.0.2 dev eth0 lladdr 02:42:ac:1d:00:02 REACHABLE
# ip neighbor | grep 172.29.0.3 
172.29.0.3 dev eth0 lladdr 02:42:ac:1d:00:03 REACHABLE

# vppctl show ip fib
ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] epoch:0 flags:none locks:[adjacency:1, default-route:1, nat-hi:2, ]
0.0.0.0/0
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:15 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.1 host-eth0: mtu:1500 next:4 0242a0b447d90242ac1d00040800
0.0.0.0/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to:[0:0]]
    [0] [@0]: dpo-drop ip4
10.244.0.0/24
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:16 buckets:1 uRPF:17 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.2 host-eth0: mtu:1500 next:4 0242ac1d00020242ac1d00040800
10.244.2.0/24
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:17 buckets:1 uRPF:18 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.3 host-eth0: mtu:1500 next:4 0242ac1d00030242ac1d00040800
172.29.0.0/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:12 to:[0:0]]
    [0] [@0]: dpo-drop ip4
172.29.0.1/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:7 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.1 host-eth0: mtu:1500 next:4 0242a0b447d90242ac1d00040800
172.29.0.2/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:9 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.2 host-eth0: mtu:1500 next:4 0242ac1d00020242ac1d00040800
172.29.0.3/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:8 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.3 host-eth0: mtu:1500 next:4 0242ac1d00030242ac1d00040800
172.29.0.0/16
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:11 to:[0:0]]
    [0] [@4]: ipv4-glean: host-eth0: mtu:1500 next:1 ffffffffffff0242ac1d00040806
172.29.0.4/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:15 buckets:1 uRPF:16 to:[49:5834]]
    [0] [@2]: dpo-receive: 172.29.0.4 on host-eth0
172.29.255.255/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:14 buckets:1 uRPF:14 to:[0:0]]
    [0] [@0]: dpo-drop ip4
224.0.0.0/4
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to:[0:0]]
    [0] [@0]: dpo-drop ip4
240.0.0.0/4
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to:[0:0]]
    [0] [@0]: dpo-drop ip4
255.255.255.255/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to:[0:0]]
    [0] [@0]: dpo-drop ip4
  1. Remote:
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
4: veth200f2ef7@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether d2:96:2c:7d:68:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 10.244.2.1/32 scope global veth200f2ef7
       valid_lft forever preferred_lft forever
5: vethe4a1f4cc@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 7a:58:0f:78:85:e8 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet 10.244.2.1/32 scope global vethe4a1f4cc
       valid_lft forever preferred_lft forever
6: vethf9b67f5b@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 32:0d:a0:38:61:47 brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet 10.244.2.1/32 scope global vethf9b67f5b
       valid_lft forever preferred_lft forever
7: veth80b04848@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether f2:b2:b0:ec:fd:6c brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet 10.244.2.1/32 scope global veth80b04848
       valid_lft forever preferred_lft forever
8: veth5b3df5e2@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 5e:ef:d2:fc:22:47 brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet 10.244.2.1/32 scope global veth5b3df5e2
       valid_lft forever preferred_lft forever
88: eth0@if89: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:1d:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.29.0.3/16 brd 172.29.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::3/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe1d:3/64 scope link 
       valid_lft forever preferred_lft forever

# ip route
default via 172.29.0.1 dev eth0 
10.244.0.0/24 via 172.29.0.2 dev eth0 
10.244.1.0/24 via 172.29.0.4 dev eth0 
10.244.2.2 dev veth200f2ef7 scope host 
10.244.2.3 dev vethe4a1f4cc scope host 
10.244.2.4 dev vethf9b67f5b scope host 
10.244.2.5 dev veth80b04848 scope host 
10.244.2.6 dev veth5b3df5e2 scope host 
172.29.0.0/16 dev eth0 proto kernel scope link src 172.29.0.3 

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/tunnel6 :: brd ::
4: veth200f2ef7@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether d2:96:2c:7d:68:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 1
5: vethe4a1f4cc@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 7a:58:0f:78:85:e8 brd ff:ff:ff:ff:ff:ff link-netnsid 2
6: vethf9b67f5b@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 32:0d:a0:38:61:47 brd ff:ff:ff:ff:ff:ff link-netnsid 3
7: veth80b04848@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether f2:b2:b0:ec:fd:6c brd ff:ff:ff:ff:ff:ff link-netnsid 4
8: veth5b3df5e2@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 5e:ef:d2:fc:22:47 brd ff:ff:ff:ff:ff:ff link-netnsid 5
88: eth0@if89: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:1d:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0

# vppctl show int addr
host-eth0 (up):
  L3 172.29.0.3/16
local0 (dn):
tap0 (up):

# vppctl show hardware
              Name                Idx   Link  Hardware
host-eth0                          1     up   host-eth0
  Link speed: unknown
  Ethernet address 02:42:ac:1d:00:03
  Linux PACKET socket interface
local0                             0    down  local0
  Link speed: unknown
  local
tap0                               2     up   tap0
  Link speed: unknown
  Ethernet address 02:fe:85:c7:60:97
  VIRTIO interface
     instance 0

# vppctl show ip neighbor
    Time                       IP                    Flags      Ethernet              Interface       
   3427.0697               172.29.0.1                  D    02:42:a0:b4:47:d9 host-eth0
      1.0529               172.29.0.2                  S    02:42:ac:1d:00:02 host-eth0
      1.0533               172.29.0.4                  S    02:42:ac:1d:00:04 host-eth0

# ip neighbor | grep 172.29.0.1    
172.29.0.1 dev eth0 lladdr 02:42:a0:b4:47:d9 REACHABLE
# ip neighbor | grep 172.29.0.2 
172.29.0.2 dev eth0 lladdr 02:42:ac:1d:00:02 REACHABLE
# ip neighbor | grep 172.29.0.4 
172.29.0.4 dev eth0 lladdr 02:42:ac:1d:00:04 REACHABLE

# vppctl show ip fib
ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] epoch:0 flags:none locks:[adjacency:1, default-route:1, nat-hi:2, ]
0.0.0.0/0
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:18 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.1 host-eth0: mtu:1500 next:4 0242a0b447d90242ac1d00030800
0.0.0.0/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to:[0:0]]
    [0] [@0]: dpo-drop ip4
10.244.0.0/24
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:15 buckets:1 uRPF:16 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.2 host-eth0: mtu:1500 next:4 0242ac1d00020242ac1d00030800
10.244.1.0/24
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:17 buckets:1 uRPF:17 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.4 host-eth0: mtu:1500 next:4 0242ac1d00040242ac1d00030800
172.29.0.0/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:11 to:[0:0]]
    [0] [@0]: dpo-drop ip4
172.29.0.1/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:16 buckets:1 uRPF:14 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.1 host-eth0: mtu:1500 next:4 0242a0b447d90242ac1d00030800
172.29.0.2/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:7 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.2 host-eth0: mtu:1500 next:4 0242ac1d00020242ac1d00030800
172.29.0.0/16
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:10 to:[0:0]]
    [0] [@4]: ipv4-glean: host-eth0: mtu:1500 next:1 ffffffffffff0242ac1d00030806
172.29.0.3/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:14 buckets:1 uRPF:15 to:[5:622]]
    [0] [@2]: dpo-receive: 172.29.0.3 on host-eth0
172.29.0.4/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:8 to:[0:0]]
    [0] [@5]: ipv4 via 172.29.0.4 host-eth0: mtu:1500 next:4 0242ac1d00040242ac1d00030800
172.29.255.255/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:13 to:[0:0]]
    [0] [@0]: dpo-drop ip4
224.0.0.0/4
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to:[0:0]]
    [0] [@0]: dpo-drop ip4
240.0.0.0/4
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to:[0:0]]
    [0] [@0]: dpo-drop ip4
255.255.255.255/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to:[0:0]]
    [0] [@0]: dpo-drop ip4

@Mixaster995 Mixaster995 mentioned this issue Jun 11, 2021
Closed
@denis-tingaikin
Copy link
Member

I think this can be closed because we did a lot of with in heal.
@glazychev-art Be free to reopen this if the issue is still actual.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Mixaster995 Mixaster995

Labels
ASAP The issue should be completed as soon as possible bug Something isn't working Planning Issue related to SOW
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Mixaster995 @denis-tingaikin