bpi-r3 branch 6.3-rc missing SFP support / transmit queue timeout #103

gustavobsch · 2023-04-08T14:41:07Z

Hi Frank,

Testing the 6.3-rc branch I noticed the SFP's stopped working, any fix for this?

Thanks

frank-w · 2023-04-08T14:49:56Z

which SFP do you use? which issue do you have with this? does sfp appear in dmesg? what does ethtool say?

there are several patches for sfp-support (mostly quirks for 2.5G rj45 sfp)

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/log/drivers/net/phy/sfp.c

network-stack is currently broken...but there are already fixes for it

#throughput for r3-gmac (wan sfp)
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/drivers/net/ethernet/mediatek/mtk_eth_soc.c?id=193250ace270fecd586dd2d0dfbd9cbd2ade977f

fixes for "net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues" which i have reverted in 6.3-rc, so these should not be needed
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/drivers/net/ethernet/mediatek/mtk_eth_soc.c?id=07b3af42d8d528374d4f42d688bae86eeb30831a

#mt7623 switch-throughput
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/drivers/net/ethernet/mediatek/mtk_eth_soc.c?id=e669ce46740a9815953bb4452a6bc5a7fdc21a50

gustavobsch · 2023-04-12T16:46:54Z

I pulled the 6.3-rc branch. compiled and booted from it and noticed the rj45 sfp stopped working. As I didn't have time to play with it and 6.2-rc has been working fine, I just went back to 6.2.

frank-w · 2023-04-12T16:58:10Z

which SFP do you use exactly?

c686900 2023-03-14 net: sfp: support 2.5g copper SFP v3

should be make 2g5 sfp from aliexpress work...if you have the tplink then daniel prepared a quirk-patch for it

gustavobsch · 2023-04-12T17:07:43Z

It's the tplink from aliexpress model TL-SM410U

I will give the 6.3-rc another try

frank-w · 2023-04-12T17:16:49Z

Ok,the tp-link needs patch from daniel i have not yet in my tree...

gustavobsch · 2023-04-12T17:19:04Z

Oh ok, got confused, do you have plans to add Daniels patch to your 6.3 tree?

frank-w · 2023-04-12T17:27:32Z

have added the tp-link quirk, but daniel reports that it is not clean working and they search the issue

https://www.spinics.net/lists/netdev/msg892902.html

btw. there is 6.2-main...better than rc to use

but i wonder why 6.2-rc works for you....

maybe you can try to revert the sfp-patch above...possibly it breaks your sfps

gustavobsch · 2023-04-12T23:18:42Z

Thanks for the tp-link quirk patch, the SFP is working now in 6.3-rc.

I will test the 6.2-main now, I noticed the fiber SFP is not detecting link.

gustavobsch · 2023-04-12T23:32:40Z

have added the tp-link quirk, but daniel reports that it is not clean working and they search the issue

https://www.spinics.net/lists/netdev/msg892902.html

btw. there is 6.2-main...better than rc to use

but i wonder why 6.2-rc works for you....

maybe you can try to revert the sfp-patch above...possibly it breaks your sfps

I couldn't find the 6.2-main branch

frank-w · 2023-04-13T07:53:42Z

which fibre sfp? mine (H!Fibre in left cage) is working in 6.3-rc

sorry about 6.2-main, had not pushed it...done now

gustavobsch · 2023-04-13T07:59:34Z

Thanks, I'll pull 6.3-main.

I tested two, I can see them but link is not detected

Brocade 57-1000013-01
Cisco GLC-SX-MM=

frank-w · 2023-04-13T08:03:31Z

6.2-main...6.3 is not final (sry typo from me...rc branches are till the kernel is released - when only rc are available)

which slot do you use?
what does ethtool and dmesg tell about them?

ethtool -m eth1/lan4

dmesg |grep -i 'eth\|sfp'

what happens when you do the autoneg workaround?

ethtool -s eth1 autoneg off
ethtool -s lan4 autoneg off

gustavobsch · 2023-04-13T15:24:36Z

Ok, my bad, last time I tried to make the SFP work I played too much with it and let auto negotiation disabled on the other side.. so after enabling it I can detect link and tcpdump traffic. Looks fine so far on 6.3-rc. vlan is not working, if I remember correctly left side SFP does not support vlan.

frank-w · 2023-04-13T15:59:47Z

Yes,there was a patch series from felix fietkau which cannot be applied...thought patch from vladimir fixed it but maybe it was another issue...felix patch disabled hw offloading on the gmac1

frank-w · 2023-04-14T07:12:22Z

This should be the patch fixing the vlan-issue on gmac

1382688

frank-w · 2023-04-16T09:22:58Z

i have pushed the vlan-fix to 6.3-pwm-branch you can cherry-pick to any other branch you want...it works so far on my bananapi-r3 and i send it as RFC/RFT to ML

https://patchwork.kernel.org/project/linux-mediatek/patch/20230416091038.54479-1-linux@fw-web.de/

gustavobsch · 2023-04-16T13:09:17Z

I just build 6.3-pwm and booted from it and I can see the vlan tags in the packets. Amazing, thanks!

frank-w · 2023-04-16T13:15:37Z

I hope there will be no sideeffects...i guess the patch disables rx vlan offload completely but better than broken vlan support

gustavobsch · 2023-04-16T21:34:41Z

I've ran into this kernel panic twice now. I need to reboot to clear it. You think it's related to the a

[20795.193278] ------------[ cut here ]------------
[20795.197920] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 3 timed out
[20795.204916] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x26c/0x274
[20795.213182] Modules linked in: pps_ldisc tun nft_masq nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables libcrc32c nfnetlink pps_gpio mt7915e mt76_connac_lib mt76 mac80211 libarc4 cfg80211 fuse ip_tables x_tables
[20795.234045] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-rc6-bpi-r3-pwm #3
[20795.241167] Hardware name: Bananapi BPI-R3 (DT)
[20795.245686] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[20795.252633] pc : dev_watchdog+0x26c/0x274
[20795.256638] lr : dev_watchdog+0x26c/0x274
[20795.260637] sp : ffffffc00800be00
[20795.263938] x29: ffffffc00800be00 x28: ffffffc009426000 x27: ffffffc00800bee8
[20795.271061] x26: ffffffc009106008 x25: 0000000000000000 x24: ffffffc0094288c8
[20795.278183] x23: 0000000000000100 x22: ffffffc009426000 x21: 0000000000000003
[20795.285306] x20: ffffff8001e31000 x19: ffffff8001e31488 x18: ffffffffffffffff
[20795.292427] x17: ffffffc076a73000 x16: ffffffc008008000 x15: 076e076107720774
[20795.299550] x14: 0720073a07290768 x13: 0000000000000255 x12: 00000000ffffffea
[20795.306673] x11: 00000000ffffefff x10: 00000000ffffefff x9 : ffffffc00949a438
[20795.313796] x8 : 0000000000017fe8 x7 : c0000000ffffefff x6 : 0000000000000001
[20795.320921] x5 : 0000000000000002 x4 : 0000000000000000 x3 : 0000000000000002
[20795.328046] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffff8000134380
[20795.335168] Call trace:
[20795.337604]  dev_watchdog+0x26c/0x274
[20795.341262]  call_timer_fn+0x34/0x150
[20795.344917]  __run_timers.part.0+0x2f4/0x358
[20795.349177]  run_timer_softirq+0x3c/0x74
[20795.353089]  _stext+0x118/0x344
[20795.356220]  ____do_softirq+0x10/0x1c
[20795.359872]  call_on_irq_stack+0x24/0x4c
[20795.363782]  do_softirq_own_stack+0x1c/0x2c
[20795.367953]  __irq_exit_rcu+0xe0/0xfc
[20795.371606]  irq_exit_rcu+0x10/0x1c
[20795.375083]  el1_interrupt+0x38/0x54
[20795.378648]  el1h_64_irq_handler+0x18/0x24
[20795.382733]  el1h_64_irq+0x68/0x6c
[20795.386123]  default_idle_call+0x58/0x108
[20795.390121]  do_idle+0xa8/0x100
[20795.393253]  cpu_startup_entry+0x24/0x2c
[20795.397166]  secondary_start_kernel+0x128/0x14c
[20795.401686]  __secondary_switched+0xb8/0xbc
[20795.405859] ---[ end trace 0000000000000000 ]---

[ 7674.040708] ------------[ cut here ]------------
[ 7674.045340] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 8 timed out
[ 7674.052351] WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x26c/0x274
[ 7674.060614] Modules linked in: pps_ldisc tun nft_masq nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables libcrc32c nfnetlink pps_gpio mt7915e mt76_connac_lib mt76 mac80211 libarc4 cfg80211 fuse ip_tables x_tables
[ 7674.081476] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 6.3.0-rc6-bpi-r3-pwm #3
[ 7674.088595] Hardware name: Bananapi BPI-R3 (DT)
[ 7674.093112] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 7674.100058] pc : dev_watchdog+0x26c/0x274
[ 7674.104061] lr : dev_watchdog+0x26c/0x274
[ 7674.108060] sp : ffffffc0096f3e00
[ 7674.111360] x29: ffffffc0096f3e00 x28: ffffffc009426000 x27: ffffffc0096f3ee8
[ 7674.118483] x26: ffffffc009106008 x25: 0000000000000000 x24: ffffffc0094288c8
[ 7674.125605] x23: 0000000000000100 x22: ffffffc009426000 x21: 0000000000000008
[ 7674.132726] x20: ffffff8000324000 x19: ffffff8000324488 x18: ffffffffffffffff
[ 7674.139847] x17: ffffffc076aab000 x16: ffffffc0096f0000 x15: ffffffc0896f3a47
[ 7674.146968] x14: 0000000000000000 x13: 00000000000001e8 x12: 00000000ffffffea
[ 7674.154089] x11: 00000000ffffefff x10: 00000000ffffefff x9 : ffffffc00949a438
[ 7674.161209] x8 : 0000000000017fe8 x7 : c0000000ffffefff x6 : 0000000000000001
[ 7674.168330] x5 : 0000000000000000 x4 : ffffff807fbb2ac8 x3 : ffffff807fbbe7b0
[ 7674.175450] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffff8000135e80
[ 7674.182572] Call trace:
[ 7674.185007]  dev_watchdog+0x26c/0x274
[ 7674.188660]  call_timer_fn+0x34/0x150
[ 7674.192315]  __run_timers.part.0+0x2f4/0x358
[ 7674.196574]  run_timer_softirq+0x3c/0x74
[ 7674.200485]  _stext+0x118/0x344
[ 7674.203616]  ____do_softirq+0x10/0x1c
[ 7674.207266]  call_on_irq_stack+0x24/0x4c
[ 7674.211178]  do_softirq_own_stack+0x1c/0x2c
[ 7674.215348]  __irq_exit_rcu+0xe0/0xfc
[ 7674.219001]  irq_exit_rcu+0x10/0x1c
[ 7674.222478]  el1_interrupt+0x38/0x54
[ 7674.226043]  el1h_64_irq_handler+0x18/0x24
[ 7674.230127]  el1h_64_irq+0x68/0x6c
[ 7674.233517]  default_idle_call+0x58/0x108
[ 7674.237514]  do_idle+0xa8/0x100
[ 7674.240647]  cpu_startup_entry+0x24/0x2c
[ 7674.244558]  secondary_start_kernel+0x128/0x14c
[ 7674.249078]  __secondary_switched+0xb8/0xbc
[ 7674.253252] ---[ end trace 0000000000000000 ]---

frank-w · 2023-04-17T04:57:10Z

I think this is related to felix patch adding the transmi queues...but i have no idea how to fix...there is a issue on openwrt github repo

frank-w · 2023-04-17T15:10:23Z

Do you know a way to trigger this bug?

This is the issue on openwrt repo where i sent an update: openwrt/openwrt#12143 (comment)

If we know how to trigger it we have a chance to debug it to root cause

frank-w · 2023-04-17T15:14:30Z

Let issue open for tracing the transmit error

gustavobsch · 2023-04-17T15:16:44Z

I don't, but it hasn't crashed today and yesterday when it crashed I was copying big files through SFP1/SFP2.

I'll run iperf3 test and see if it crashes again

frank-w · 2023-04-17T15:19:06Z

I have not seen it with iperf3 yet...thats my standard-test in both directions,but i guess the problem is when there are multiple streams which are spreaded over different tx queues over time

frank-w · 2023-04-19T10:31:43Z

Have you seen this with 6.2 too? If not you can try the vlan-patch there and look if this is the cause but imho this cannot be the cause as this is not in tx path

gustavobsch · 2023-04-19T10:53:47Z

I havent tested with 6.2, can I just checkout latest 6.2-main and build it?

I ran into the issue again yesterday and I think the issue has something to do with the right SFP, where I have the copper tplink SFP.

I ran into it three times when traffic is flowing as shown below.

              ------------------>
       eth1                           lan4
------------------------------------------
|   fiber/vlan   |   copper/untagged     |
------------------------------------------

frank-w · 2023-04-19T12:46:09Z

Is this reproducable for you (e.g. you can trigger it this way)?

Is the other end also 2.5g or do you run at lower speed there?

Could you try with another sfp (maybe only 1g) in the lan-sfp cage or towards one of the rj45 ports?

Basicly you can use every branch to build :) vlanfix is only applied to the last one (6.3-pwm) but for testing the sfp itself other branches can work too,but maybe need patches for supporting the tplink-sfp

gustavobsch · 2023-04-19T12:56:45Z

I can trigger it but unsure exactly how.. it just happens. I noticed it doesn't happen when I just stop using the right sfp2/lan4

Both ends on both SFP's are 2.5g

I tried a copper 1g SFP on the right sfp2/lan4 and it wont take it seems like it only allows 2.5g modules


root@bpi-r3:~# dmesg
[ 2265.225821] sfp sfp-2: module AVAGO            ABCU-5700RZ      rev      sn AN1001A2CX       dc 100107  
[ 2286.381969] mt7530 mdio-bus:1f lan4: configuring for inband/2500base-x link mode
[ 2286.454562] mt7530 mdio-bus:1f lan4: validation with support 00,00000000,00000000,00000000 failed: -EINVAL
[ 2286.464391] sfp sfp-2: sfp_add_phy failed: -EINVAL
[ 2315.659816] mt7530 mdio-bus:1f lan4: configuring for inband/2500base-x link mode
[ 2315.798868] mt7530 mdio-bus:1f lan4: validation with support 00,00000000,00000000,00000000 failed: -EINVAL
[ 2315.808653] sfp sfp-2: sfp_add_phy failed: -EINVAL
root@bpi-r3:~# ethtool lan4
Settings for lan4:
	Supported ports: [ MII ]
	Supported link modes:   2500baseX/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: No
	Supported FEC modes: Not reported
	Advertised link modes:  2500baseX/Full
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: No
	Advertised FEC modes: Not reported
	Speed: Unknown!
	Duplex: Unknown! (255)
	Auto-negotiation: off
	Port: MII
	PHYAD: 0
	Transceiver: internal
	Supports Wake-on: d
	Wake-on: d
	Link detected: no

frank-w · 2023-04-20T17:13:38Z

My fiber sfp works in both cages and this is 1g only ..maybe your 1g module needs a quirk too...daniel reported from some sfps having checksum errors or having invalid data (e.g. supported modes) in eeprom

frank-w · 2023-04-22T11:42:50Z

Can you trigger the issue when only working over lan4/sfp2 (writing/reading to/from r3).

gustavobsch · 2023-04-27T13:31:52Z

Can you trigger the issue when only working over lan4/sfp2 (writing/reading to/from r3).

I took a while to reply because I was testing. The good news is that I haven't ran into this issue again. I have both SFP's populated with the same modules and same traffic going through them.

The only thing that changed is back then I was doing a lot of NTP/GPS/PPS testing, maybe the kernel panic had something to do with it? I can't think of anything else.

gustavobsch · 2023-04-28T12:37:37Z

I loaded 6.3.0-bpi-r3-main 24 hours ago and no crashes. When I initially ran into this issue it could crash after couple hours.

frank-w · 2023-05-03T16:04:00Z

Have you same connections as before (sfps,rj45 ports,vlan)? Have not seen any fix for these problems

gustavobsch · 2023-05-03T17:57:50Z

Something definitely changed, but I can't tell what.
The SFP's and the cages where they are inserted hasn't changed.
It hasn't crashed since I reported the initial crash so I'm happy

gustavobsch · 2024-04-23T21:19:06Z

Hi Frank,

Can you add the tp-link quirk patch to 6.6.25-bpi-r3-main?

Thanks
Gustavo

frank-w · 2024-04-24T04:43:29Z

Can you make a pull request or point to the patch?

gustavobsch · 2024-04-24T14:58:49Z

I'm actually not sure that patch has the fix. The issue this time is the SFP not detecting link.


[246390.041031] sfp sfp-2: module TP-LINK          TL-SM410U        rev 1.0  sn 121B0Y9000456    dc 211108  
[246390.050640] mt7530-mdio mdio-bus:1f lan4: unsupported SFP module: no common interface modes
root@bpi-r3:~# ethtool lan4
Settings for lan4:
	Supported ports: [ MII ]
	Supported link modes:   2500baseX/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  2500baseX/Full
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: Unknown!
	Duplex: Unknown! (255)
	Auto-negotiation: on
	Port: MII
	PHYAD: 0
	Transceiver: internal
	Supports Wake-on: d
	Wake-on: d
	Link detected: no

frank-w · 2024-04-24T15:40:55Z

This looks like the main problem for all 2g5 sfp...look what i have done in 6.9-net-next...i guess you need at least erics phylink patch and the tplink quirk let phylink know about 2g5...the phy-mapping i had done for the oem imho does not work for tplink

gustavobsch closed this as completed Apr 12, 2023

frank-w reopened this Apr 17, 2023

frank-w changed the title ~~bpi-r3 branch 6.3-rc missing SFP support~~ bpi-r3 branch 6.3-rc missing SFP support / transmit queue timeout Apr 17, 2023

bpi-r3 branch 6.3-rc missing SFP support / transmit queue timeout #103

bpi-r3 branch 6.3-rc missing SFP support / transmit queue timeout #103

Comments

gustavobsch commented Apr 8, 2023

frank-w commented Apr 8, 2023 • edited Loading

gustavobsch commented Apr 12, 2023

frank-w commented Apr 12, 2023

gustavobsch commented Apr 12, 2023

frank-w commented Apr 12, 2023

gustavobsch commented Apr 12, 2023

frank-w commented Apr 12, 2023 • edited Loading

gustavobsch commented Apr 12, 2023

gustavobsch commented Apr 12, 2023

frank-w commented Apr 13, 2023 • edited Loading

gustavobsch commented Apr 13, 2023

frank-w commented Apr 13, 2023 • edited Loading

gustavobsch commented Apr 13, 2023

frank-w commented Apr 13, 2023

frank-w commented Apr 14, 2023

frank-w commented Apr 16, 2023

gustavobsch commented Apr 16, 2023 • edited Loading

frank-w commented Apr 16, 2023

gustavobsch commented Apr 16, 2023 • edited Loading

frank-w commented Apr 17, 2023

frank-w commented Apr 17, 2023 • edited Loading

frank-w commented Apr 17, 2023

gustavobsch commented Apr 17, 2023

frank-w commented Apr 17, 2023

frank-w commented Apr 19, 2023

gustavobsch commented Apr 19, 2023 • edited Loading

frank-w commented Apr 19, 2023 • edited Loading

gustavobsch commented Apr 19, 2023

frank-w commented Apr 20, 2023

frank-w commented Apr 22, 2023

gustavobsch commented Apr 27, 2023

gustavobsch commented Apr 28, 2023

frank-w commented May 3, 2023

gustavobsch commented May 3, 2023

gustavobsch commented Apr 23, 2024

frank-w commented Apr 24, 2024

gustavobsch commented Apr 24, 2024 • edited Loading

frank-w commented Apr 24, 2024 • edited Loading

frank-w commented Apr 8, 2023 •

edited

Loading

frank-w commented Apr 12, 2023 •

edited

Loading

frank-w commented Apr 13, 2023 •

edited

Loading

frank-w commented Apr 13, 2023 •

edited

Loading

gustavobsch commented Apr 16, 2023 •

edited

Loading

gustavobsch commented Apr 16, 2023 •

edited

Loading

frank-w commented Apr 17, 2023 •

edited

Loading

gustavobsch commented Apr 19, 2023 •

edited

Loading

frank-w commented Apr 19, 2023 •

edited

Loading

gustavobsch commented Apr 24, 2024 •

edited

Loading

frank-w commented Apr 24, 2024 •

edited

Loading