cpu/sam0_common: Ethernet driver stops working after a while

[benpicco]

Description

The sam0 ethernet driver will stop working after a while of use.

Steps to reproduce the issue

I don't have a good way of reproducing the issue yet.
Basically I just leave the same54-xpro connected to an ethernet network for a while using examples/gnrc_networking or examples/gnrc_border_router.
After some time, it will cease to receive any packets (send still works) on the Ethernet interface.

Expected results

Packet reception is reliable

Actual results

Ethernet can not receive packets after some time.
Sending packets still works.

Versions

current RIOT master

[dylad]

@benpicco
How many time did you wait before this issue to pop up ?

[haukepetersen]

I am not sure its related, but I saw a similar behavior last weekend when building an Ethernet based node: after booting the node Ethernet works as expected, but after a while (anywhere between 5min and 30min) the node stops responding. In my cases its a bluepull connected to a enc28j60 ethernet module via SPI. But so far I do not have any shell access to the node directly, so I can't really tell why it stops responding. Once I find some time I will dig into this... Maybe it is some generic GNRC/netdev problem we are facing here?!

[dylad]

I can confirm there is a weird behavior, I'll investigate and report back if the problem is related to sam0 or not.

[benpicco]

It's actually quite quick to reproduce with the same54-xpro as border router with 'spurious' traffic.
e.g. I connect the REB215-XPRO extension to it and run

USEMODULE=at86rf215 BOARD=same54-xpro UPLINK=ethernet make -C examples/gnrc_border_router flash

I have a node that gets an IP and can ping an address on the Internet:

2021-04-23 17:33:19,415 # Iface  7  HWaddr: 0B:B1  Channel: 26  Page: 0  NID: 0x23  PHY: O-QPSK 
2021-04-23 17:33:19,415 #           
2021-04-23 17:33:19,421 #           Long HWaddr: FC:C2:3D:00:00:00:0B:B1 
2021-04-23 17:33:19,427 #            TX-Power: 0dBm  State: IDLE  max. Retrans.: 3  CSMA Retries: 4 
2021-04-23 17:33:19,434 #           AUTOACK  ACK_REQ  CSMA  L2-PDU:102  MTU:1280  HL:64  RTR  
2021-04-23 17:33:19,437 #           RTR_ADV  6LO  IPHC  
2021-04-23 17:33:19,440 #           Source address length: 8
2021-04-23 17:33:19,442 #           Link type: wireless
2021-04-23 17:33:19,449 #           inet6 addr: fe80::fec2:3d00:0:bb1  scope: link  VAL
2021-04-23 17:33:19,457 #           inet6 addr: 2001:16b8:4572:51f8:fec2:3d00:0:bb1  scope: global  VAL
2021-04-23 17:33:19,460 #           inet6 group: ff02::2
2021-04-23 17:33:19,463 #           inet6 group: ff02::1
2021-04-23 17:33:19,466 #           inet6 group: ff02::1:ff00:bb1

2021-04-23 17:33:33,017 #  ping 2600::
2021-04-23 17:33:33,164 # 12 bytes from 2600::: icmp_seq=0 ttl=48 rssi=-21 dBm time=141.056 ms
2021-04-23 17:33:34,163 # 12 bytes from 2600::: icmp_seq=1 ttl=48 rssi=-21 dBm time=139.984 ms
2021-04-23 17:33:35,162 # 12 bytes from 2600::: icmp_seq=2 ttl=48 rssi=-21 dBm time=137.808 ms
2021-04-23 17:33:35,162 # 
2021-04-23 17:33:35,165 # --- 2600:: PING statistics ---
2021-04-23 17:33:35,168 # 3 packets transmitted, 3 packets received, 0% packet loss

Now I flood ping this node, and withing a few seconds, the border router stops routing:

sudo ping -f 2001:16b8:4572:51f8:fec2:3d00:0:bb1
PING 2001:16b8:4572:51f8:fec2:3d00:0:bb1(2001:16b8:4572:51f8:fec2:3d00:0:bb1) 56 data bytes
............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C
--- 2001:16b8:4572:51f8:fec2:3d00:0:bb1 ping statistics ---
663 packets transmitted, 187 received, 71.7949% packet loss, time 10217ms
rtt min/avg/max/mdev = 13.658/34.274/70.565/10.589 ms, pipe 5, ipg/ewma 15.433/47.509 ms

2021-04-23 17:33:37,393 #  ping 2600:: -c 100
2021-04-23 17:33:37,540 # 12 bytes from 2600::: icmp_seq=0 ttl=48 rssi=-21 dBm time=139.664 ms
2021-04-23 17:33:38,539 # 12 bytes from 2600::: icmp_seq=1 ttl=48 rssi=-21 dBm time=138.832 ms
2021-04-23 17:33:44,557 # 12 bytes from 2600::: icmp_seq=7 ttl=48 rssi=-21 dBm time=155.888 ms
2021-04-23 17:35:17,422 # 
2021-04-23 17:35:17,423 # --- 2600:: PING statistics ---
2021-04-23 17:35:17,431 # 100 packets transmitted, 3 packets received, 97% packet loss

Border router is dead too

2021-04-23 17:38:26,828 # Iface  7  HWaddr: FC:C2:3D:0D:2D:1F 
2021-04-23 17:38:26,833 #           L2-PDU:1500  MTU:1492  HL:255  RTR  
2021-04-23 17:38:26,836 #           Source address length: 6
2021-04-23 17:38:26,838 #           Link type: wired
2021-04-23 17:38:26,844 #           inet6 addr: fe80::fec2:3dff:fe0d:2d1f  scope: link  VAL
2021-04-23 17:38:26,851 #           inet6 addr: 2001:16b8:4572:5100:fec2:3dff:fe0d:2d1f  scope: global  VAL
2021-04-23 17:38:26,854 #           inet6 group: ff02::2
2021-04-23 17:38:26,857 #           inet6 group: ff02::1
2021-04-23 17:38:26,860 #           inet6 group: ff02::1:ff0d:2d1f
2021-04-23 17:38:26,861 #           
2021-04-23 17:38:26,867 # Iface  6  HWaddr: 6A:E4  Channel: 26  Page: 0  NID: 0x23  PHY: O-QPSK 
2021-04-23 17:38:26,869 #           
2021-04-23 17:38:26,873 #           Long HWaddr: E6:EA:AF:F8:AF:5D:EA:E4 
2021-04-23 17:38:26,879 #            TX-Power: 0dBm  State: IDLE  max. Retrans.: 3  CSMA Retries: 4 
2021-04-23 17:38:26,886 #           AUTOACK  ACK_REQ  CSMA  L2-PDU:102  MTU:1280  HL:64  RTR  
2021-04-23 17:38:26,888 #           RTR_ADV  6LO  IPHC  
2021-04-23 17:38:26,891 #           Source address length: 8
2021-04-23 17:38:26,894 #           Link type: wireless
2021-04-23 17:38:26,900 #           inet6 addr: fe80::e4ea:aff8:af5d:eae4  scope: link  VAL
2021-04-23 17:38:26,907 #           inet6 addr: 2001:16b8:4572:51f8:e4ea:aff8:af5d:eae4  scope: global  VAL
2021-04-23 17:38:26,910 #           inet6 group: ff02::2
2021-04-23 17:38:26,912 #           inet6 group: ff02::1
2021-04-23 17:38:26,916 #           inet6 group: ff02::1:ff5d:eae4
2021-04-23 17:38:26,917 #           
2021-04-23 17:38:26,923 # Iface  5  HWaddr: 6A:E5  Channel: 0  Page: 2  NID: 0x23  PHY: O-QPSK 
2021-04-23 17:38:26,924 #           
2021-04-23 17:38:26,928 #           Long HWaddr: E6:EA:AF:F8:AF:5D:EA:E5 
2021-04-23 17:38:26,935 #            TX-Power: 0dBm  State: IDLE  max. Retrans.: 3  CSMA Retries: 4 
2021-04-23 17:38:26,941 #           AUTOACK  ACK_REQ  CSMA  L2-PDU:102  MTU:1280  HL:64  RTR  
2021-04-23 17:38:26,944 #           RTR_ADV  6LO  IPHC  
2021-04-23 17:38:26,947 #           Source address length: 8
2021-04-23 17:38:26,950 #           Link type: wireless
2021-04-23 17:38:26,956 #           inet6 addr: fe80::e4ea:aff8:af5d:eae5  scope: link  VAL
2021-04-23 17:38:26,958 #           inet6 group: ff02::2
2021-04-23 17:38:26,961 #           inet6 group: ff02::1
2021-04-23 17:38:26,965 #           inet6 group: ff02::1:ff5d:eae5
2021-04-23 17:38:26,965 #           
> ping 2600::
2021-04-23 17:38:30,306 #  ping 2600::
2021-04-23 17:38:33,306 # 
2021-04-23 17:38:33,308 # --- 2600:: PING statistics ---
2021-04-23 17:38:33,314 # 3 packets transmitted, 0 packets received, 100% packet loss