Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kernel panic #3401

Open
js290 opened this issue Jan 12, 2020 · 30 comments
Open

kernel panic #3401

js290 opened this issue Jan 12, 2020 · 30 comments

Comments

@js290
Copy link

js290 commented Jan 12, 2020

Is this the right place for my bug report?
This repository contains the Linux kernel used on the Raspberry Pi. If you believe that the issue you are seeing is kernel-related, this is the right place. If not, we have other repositories for the GPU firmware at github.com/raspberrypi/firmware and Raspberry Pi userland applications at github.com/raspberrypi/userland. If you have problems with the Raspbian distribution packages, report them in the github.com/RPi-Distro/repo. If you simply have a question, then the Raspberry Pi forums are the best place to ask it.

Describe the bug
Add a clear and concise description of what you think the bug is.
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167099] ------------[ cut here ]------------
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167177] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:466 dev_watchdog+0x26c/0x288
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167190] NETDEV WATCHDOG: eth0 (dm9601): transmit queue 0 timed out
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167196] Modules linked in: sha256_generic cfg80211 rfkill 8021q garp stp llc nft_counter xt_tcpudp xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink sr9700 dm9601 raspberrypi_hwmon hwmon bcm2835_codec(C) bcm2835_v4l2(C) v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig v4l2_common videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media vc_sm_cma(C) uio_pdrv_genirq uio fixed ip_tables x_tables ipv6
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167417] CPU: 0 PID: 0 Comm: swapper Tainted: G C 4.19.75+ #1270
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167424] Hardware name: BCM2835
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167472] [] (unwind_backtrace) from [] (show_stack+0x20/0x24)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167510] [] (show_stack) from [] (dump_stack+0x20/0x28)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167546] [] (dump_stack) from [] (__warn+0xf4/0x11c)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167570] [] (__warn) from [] (warn_slowpath_fmt+0x54/0x74)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167599] [] (warn_slowpath_fmt) from [] (dev_watchdog+0x26c/0x288)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167646] [] (dev_watchdog) from [] (call_timer_fn+0x3c/0x120)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167674] [] (call_timer_fn) from [] (expire_timers+0x9c/0xd8)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167700] [] (expire_timers) from [] (run_timer_softirq+0xa4/0x1ac)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167726] [] (run_timer_softirq) from [] (__do_softirq+0x12c/0x344)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167761] [] (__do_softirq) from [] (irq_exit+0xe4/0x100)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167792] [] (irq_exit) from [] (__handle_domain_irq+0x60/0xb8)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167815] [] (__handle_domain_irq) from [] (bcm2835_handle_irq+0x28/0x48)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167834] [] (bcm2835_handle_irq) from [] (__irq_svc+0x5c/0x7c)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167845] Exception stack(0xc09e7ef0 to 0xc09e7f38)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167859] 7ee0: c00115e4 00000000 40000093 40000093
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167878] 7f00: c09e6000 c09e9090 c0a76fb6 c09e9020 c0a84c80 dbfffa40 c09c0a30 c09e7f4c
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167896] 7f20: c09e941c c09e7f40 00000000 c00115e8 40000013 ffffffff
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167927] [] (__irq_svc) from [] (arch_cpu_idle+0x30/0x40)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167957] [] (arch_cpu_idle) from [] (default_idle_call+0x34/0x48)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.167994] [] (default_idle_call) from [] (do_idle+0x94/0xf4)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.168027] [] (do_idle) from [] (cpu_startup_entry+0x1c/0x20)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.168063] [] (cpu_startup_entry) from [] (rest_init+0x7c/0x9c)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.168088] [] (rest_init) from [] (start_kernel+0x3e8/0x49c)
Jan 12 00:18:06 raspberrypi kernel: [ 8378.168103] ---[ end trace 5d6e43a630ac3840 ]---

To reproduce
List the steps required to reproduce the issue.

Expected behaviour
Add a clear and concise description of what you expected to happen.

Actual behaviour
Add a clear and concise description of what actually happened.

System
System Information

Raspberry Pi Zero Rev 1.3
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"

Raspberry Pi reference 2019-09-26
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc3fc13cf3a2f8b464e129be, stage2

Linux raspberrypi 4.19.75+ #1270 Tue Sep 24 18:38:54 BST 2019 armv6l GNU/Linux
Revision : 900093
Serial : 000000001a48b5ea
Model : Raspberry Pi Zero Rev 1.3
Throttled flag : throttled=0x0
Camera : supported=0 detected=0

Videocore information

Sep 24 2019 17:37:19
Copyright (c) 2012 Broadcom
version 6820edeee4ef3891b95fc01cf02a7abd7ca52f17 (clean) (release) (start)

alloc failures: 0
compactions: 0
legacy block fails: 0

Filesystem information

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 1866100 1380732 372524 79% /
devtmpfs 217212 0 217212 0% /dev
tmpfs 221540 332 221208 1% /dev/shm
tmpfs 221540 3124 218416 2% /run
tmpfs 5120 0 5120 0% /run/lock
tmpfs 221540 0 221540 0% /sys/fs/cgroup
/dev/mmcblk0p1 258095 53702 204394 21% /boot
tmpfs 44308 0 44308 0% /run/user/999
tmpfs 44308 0 44308 0% /run/user/1000

Filename Type Size Used Priority
/var/swap file 102396 0 -2

Package version information

raspberrypi-ui-mods:
Installed: (none)
raspberrypi-sys-mods:
Installed: 20191105
openbox:
Installed: (none)
lxpanel:
Installed: (none)
pcmanfm:
Installed: (none)
rpd-plym-splash:
Installed: (none)

Networking Information

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet x.x.x.x netmask x.x.x.x broadcast x.x.x.x
ether m.m.m.m txqueuelen 1000 (Ethernet)
RX packets 3716 bytes 413699 (404.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3169 bytes 539152 (526.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet x.x.x.x netmask x.x.x.x
loop txqueuelen 1000 (Local Loopback)
RX packets 87 bytes 7763 (7.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 87 bytes 7763 (7.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

USB Information

/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 2: Dev 3, If 0, Class=, Driver=dm9601, 12M

config.txt

aphy_params_current=547
arm_freq=1000
audio_pwm_mode=514
config_hdmi_boost=5
core_freq=400
disable_auto_turbo=1
disable_commandline_tags=2
display_hdmi_rotate=-1
display_lcd_rotate=-1
dphy_params_current=547
enable_uart=1
force_eeprom_read=1
force_pwm_open=1
framebuffer_ignore_alpha=1
framebuffer_swap=1
gpu_freq=300
hdmi_safe=1
ignore_lcd=1
init_uart_clock=0x2dc6c00
over_voltage_avs=0x249f0
overscan_bottom=16
overscan_left=24
overscan_right=24
overscan_top=16
pause_burst_frames=1
program_serial_random=1
sdram_freq=450
hdmi_force_cec_address:0=65535
hdmi_force_cec_address:1=65535
hdmi_force_hotplug:0=1
hdmi_force_hotplug:1=1
hdmi_group:0=2
hdmi_group:1=2
hdmi_mode:0=4
hdmi_mode:1=4
hdmi_pixel_freq_limit:0=0x9a7ec80
hdmi_pixel_freq_limit:1=0x9a7ec80
device_tree=-
hdmi_cvt:0=
hdmi_cvt:1=
hdmi_edid_filename:0=
hdmi_edid_filename:1=
hdmi_timings:0=
hdmi_timings:1=

cmdline.txt

coherent_pool=1M bcm2708_fb.fbwidth=592 bcm2708_fb.fbheight=448 bcm2708_fb.fbswap=1 smsc95xx.macaddr=B8:27:EB:48:B5:EA vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000 console=ttyAMA0,115200 console=tty1 root=PARTUUID=6c586e13-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

raspi-gpio settings

BANK0 (GPIO 0 to 27):
GPIO 0: level=1 fsel=0 func=INPUT
GPIO 1: level=1 fsel=0 func=INPUT
GPIO 2: level=1 fsel=0 func=INPUT
GPIO 3: level=1 fsel=0 func=INPUT
GPIO 4: level=1 fsel=0 func=INPUT
GPIO 5: level=1 fsel=0 func=INPUT
GPIO 6: level=1 fsel=0 func=INPUT
GPIO 7: level=1 fsel=0 func=INPUT
GPIO 8: level=1 fsel=0 func=INPUT
GPIO 9: level=0 fsel=0 func=INPUT
GPIO 10: level=0 fsel=0 func=INPUT
GPIO 11: level=0 fsel=0 func=INPUT
GPIO 12: level=0 fsel=0 func=INPUT
GPIO 13: level=0 fsel=0 func=INPUT
GPIO 14: level=1 fsel=4 alt=0 func=TXD0
GPIO 15: level=1 fsel=4 alt=0 func=RXD0
GPIO 16: level=0 fsel=0 func=INPUT
GPIO 17: level=0 fsel=0 func=INPUT
GPIO 18: level=0 fsel=0 func=INPUT
GPIO 19: level=0 fsel=0 func=INPUT
GPIO 20: level=0 fsel=0 func=INPUT
GPIO 21: level=0 fsel=0 func=INPUT
GPIO 22: level=0 fsel=0 func=INPUT
GPIO 23: level=0 fsel=0 func=INPUT
GPIO 24: level=0 fsel=0 func=INPUT
GPIO 25: level=0 fsel=0 func=INPUT
GPIO 26: level=0 fsel=0 func=INPUT
GPIO 27: level=0 fsel=0 func=INPUT
BANK1 (GPIO 28 to 45):
GPIO 28: level=1 fsel=0 func=INPUT
GPIO 29: level=1 fsel=0 func=INPUT
GPIO 30: level=0 fsel=0 func=INPUT
GPIO 31: level=0 fsel=0 func=INPUT
GPIO 32: level=0 fsel=1 func=OUTPUT
GPIO 33: level=0 fsel=0 func=INPUT
GPIO 34: level=1 fsel=0 func=INPUT
GPIO 35: level=1 fsel=0 func=INPUT
GPIO 36: level=1 fsel=0 func=INPUT
GPIO 37: level=0 fsel=0 func=INPUT
GPIO 38: level=0 fsel=0 func=INPUT
GPIO 39: level=0 fsel=0 func=INPUT
GPIO 40: level=0 fsel=0 func=INPUT
GPIO 41: level=0 fsel=1 func=OUTPUT
GPIO 42: level=0 fsel=0 func=INPUT
GPIO 43: level=0 fsel=0 func=INPUT
GPIO 44: level=0 fsel=0 func=INPUT
GPIO 45: level=0 fsel=0 func=INPUT
BANK2 (GPIO 46 to 53):
GPIO 46: level=1 fsel=0 func=INPUT
GPIO 47: level=0 fsel=1 func=OUTPUT
GPIO 48: level=0 fsel=4 alt=0 func=SD0_CLK
GPIO 49: level=1 fsel=4 alt=0 func=SD0_CMD
GPIO 50: level=1 fsel=4 alt=0 func=SD0_DAT0
GPIO 51: level=1 fsel=4 alt=0 func=SD0_DAT1
GPIO 52: level=1 fsel=4 alt=0 func=SD0_DAT2
GPIO 53: level=1 fsel=4 alt=0 func=SD0_DAT3

vcdbg log messages

001192.433: brfs: File read: /mfs/sd/config.txt
001193.216: brfs: File read: 1828 bytes
001203.765: HDMI:EDID error reading EDID block 0 attempt 0
001205.067: HDMI:EDID error reading EDID block 0 attempt 1
001206.360: HDMI:EDID error reading EDID block 0 attempt 2
001207.654: HDMI:EDID error reading EDID block 0 attempt 3
001208.947: HDMI:EDID error reading EDID block 0 attempt 4
001210.243: HDMI:EDID error reading EDID block 0 attempt 5
001211.535: HDMI:EDID error reading EDID block 0 attempt 6
001212.830: HDMI:EDID error reading EDID block 0 attempt 7
001214.124: HDMI:EDID error reading EDID block 0 attempt 8
001215.419: HDMI:EDID error reading EDID block 0 attempt 9
001216.475: HDMI:EDID giving up on reading EDID block 0
001216.833: HDMI:EDID error reading EDID block 0 attempt 0
001218.132: HDMI:EDID error reading EDID block 0 attempt 1
001219.432: HDMI:EDID error reading EDID block 0 attempt 2
001220.731: HDMI:EDID error reading EDID block 0 attempt 3
001222.030: HDMI:EDID error reading EDID block 0 attempt 4
001223.329: HDMI:EDID error reading EDID block 0 attempt 5
001224.627: HDMI:EDID error reading EDID block 0 attempt 6
001225.925: HDMI:EDID error reading EDID block 0 attempt 7
001227.223: HDMI:EDID error reading EDID block 0 attempt 8
001228.522: HDMI:EDID error reading EDID block 0 attempt 9
001229.584: HDMI:EDID giving up on reading EDID block 0
001243.617: brfs: File read: /mfs/sd/config.txt
001244.418: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
001254.541: gpioman: gpioman_get_pin_num: pin WL_LPO_CLK not defined
001254.608: gpioman: gpioman_get_pin_num: pin BT_ON not defined
001254.666: gpioman: gpioman_get_pin_num: pin WL_ON not defined
001276.964: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
001277.056: *** Restart logging
001277.122: brfs: File read: 1828 bytes
001277.840: hdmi: HDMI:EDID error reading EDID block 0 attempt 0
001279.150: hdmi: HDMI:EDID error reading EDID block 0 attempt 1
001280.452: hdmi: HDMI:EDID error reading EDID block 0 attempt 2
001281.755: hdmi: HDMI:EDID error reading EDID block 0 attempt 3
001283.058: hdmi: HDMI:EDID error reading EDID block 0 attempt 4
001284.361: hdmi: HDMI:EDID error reading EDID block 0 attempt 5
001285.664: hdmi: HDMI:EDID error reading EDID block 0 attempt 6
001286.967: hdmi: HDMI:EDID error reading EDID block 0 attempt 7
001288.270: hdmi: HDMI:EDID error reading EDID block 0 attempt 8
001289.573: hdmi: HDMI:EDID error reading EDID block 0 attempt 9
001290.640: hdmi: HDMI:EDID giving up on reading EDID block 0
001291.015: hdmi: HDMI:EDID error reading EDID block 0 attempt 0
001292.316: hdmi: HDMI:EDID error reading EDID block 0 attempt 1
001293.615: hdmi: HDMI:EDID error reading EDID block 0 attempt 2
001294.914: hdmi: HDMI:EDID error reading EDID block 0 attempt 3
001296.213: hdmi: HDMI:EDID error reading EDID block 0 attempt 4
001297.512: hdmi: HDMI:EDID error reading EDID block 0 attempt 5
001298.811: hdmi: HDMI:EDID error reading EDID block 0 attempt 6
001300.110: hdmi: HDMI:EDID error reading EDID block 0 attempt 7
001301.409: hdmi: HDMI:EDID error reading EDID block 0 attempt 8
001302.708: hdmi: HDMI:EDID error reading EDID block 0 attempt 9
001303.770: hdmi: HDMI:EDID giving up on reading EDID block 0
001307.358: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
001308.707: brfs: File read: /mfs/sd/cmdline.txt
001308.785: Read command line from file 'cmdline.txt':
001308.843: 'console=serial0,115200 console=tty1 root=PARTUUID=6c586e13-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait'
001313.523: brfs: File read: 121 bytes
001317.172: brfs: File read: /mfs/sd/bcm2708-rpi-zero.dtb
001317.234: Loading 'bcm2708-rpi-zero.dtb' to 0x100 size 0x5c77
001406.350: brfs: File read: 23671 bytes
001408.890: brfs: File read: /mfs/sd/config.txt
001409.385: brfs: File read: 1828 bytes
001413.632: brfs: File read: /mfs/sd/overlays/disable-wifi.dtbo
001425.545: Loaded overlay 'disable-wifi'
001443.198: brfs: File read: 387 bytes
001444.425: brfs: File read: /mfs/sd/overlays/disable-bt.dtbo
001455.141: dterror: can't find symbol 'uart0_pins'
001455.156: Failed to resolve overlay 'disable-bt'
003531.408: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
003554.007: brfs: File read: 954 bytes
003897.831: brfs: File read: /mfs/sd/kernel.img
003897.898: Loading 'kernel.img' to 0x8000 size 0x4cbd38
003897.966: Device tree loaded to 0x1bfe9f00 (size 0x6038)
003899.424: gpioman: gpioman_get_pin_num: pin SDCARD_CONTROL_POWER not defined
005753.985: vchiq_core: vchiq_init_state: slot_zero = 0x5b580000, is_master = 1
005759.205: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
005764.555: TV service:host side not connected, dropping notification 0x00000002, 0x00000002, 0x00000004
019472.546: brfs: File read: 5029176 bytes

dmesg log

[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.19.75+ (dom@buildbot) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1270 Tue Sep 24 18:38:54 BST 2019
[ 0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[ 0.000000] OF: fdt: Machine model: Raspberry Pi Zero Rev 1.3
[ 0.000000] Memory policy: Data cache writeback
[ 0.000000] cma: Reserved 8 MiB at 0x1b400000
[ 0.000000] On node 0 totalpages: 114688
[ 0.000000] Normal zone: 1008 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 114688 pages, LIFO batch:31
[ 0.000000] random: get_random_bytes called from start_kernel+0x8c/0x49c with crng_init=0
[ 0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[ 0.000000] pcpu-alloc: [0] 0
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 113680
[ 0.000000] Kernel command line: coherent_pool=1M bcm2708_fb.fbwidth=592 bcm2708_fb.fbheight=448 bcm2708_fb.fbswap=1 smsc95xx.macaddr=m.m.m.m vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000 console=ttyAMA0,115200 console=tty1 root=PARTUUID=6c586e13-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
[ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 0.000000] Memory: 434424K/458752K available (6973K kernel code, 635K rwdata, 2080K rodata, 464K init, 797K bss, 16136K reserved, 8192K cma-reserved)
[ 0.000000] Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xdc800000 - 0xff800000 ( 560 MB)
lowmem : 0xc0000000 - 0xdc000000 ( 448 MB)
modules : 0xbf000000 - 0xc0000000 ( 16 MB)
.text : 0x(ptrval) - 0x(ptrval) (6975 kB)
.init : 0x(ptrval) - 0x(ptrval) ( 464 kB)
.data : 0x(ptrval) - 0x(ptrval) ( 636 kB)
.bss : 0x(ptrval) - 0x(ptrval) ( 798 kB)
[ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] ftrace: allocating 25197 entries in 74 pages
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000031] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[ 0.000094] clocksource: timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[ 0.000218] bcm2835: system timer (irq = 27)
[ 0.000637] Console: colour dummy device 80x30
[ 0.001274] console [tty1] enabled
[ 0.001355] Calibrating delay loop... 697.95 BogoMIPS (lpj=3489792)
[ 0.060395] pid_max: default: 32768 minimum: 301
[ 0.060965] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.061025] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.062466] CPU: Testing write buffer coherency: ok
[ 0.063859] Setting up static identity map for 0x8200 - 0x8238
[ 0.065350] devtmpfs: initialized
[ 0.075733] VFP support v0.3: implementor 41 architecture 1 part 20 variant b rev 5
[ 0.076196] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.076275] futex hash table entries: 256 (order: -1, 3072 bytes)
[ 0.077667] pinctrl core: initialized pinctrl subsystem
[ 0.079276] NET: Registered protocol family 16
[ 0.082504] DMA: preallocated 1024 KiB pool for atomic coherent allocations
[ 0.089531] hw-breakpoint: found 6 breakpoint and 1 watchpoint registers.
[ 0.089608] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.089791] Serial: AMBA PL011 UART driver
[ 0.093105] bcm2835-mbox 2000b880.mailbox: mailbox enabled
[ 0.136162] bcm2835-dma 20007000.dma: DMA legacy API manager at (ptrval), dmachans=0x1
[ 0.138779] SCSI subsystem initialized
[ 0.139108] usbcore: registered new interface driver usbfs
[ 0.139255] usbcore: registered new interface driver hub
[ 0.139517] usbcore: registered new device driver usb
[ 0.150904] raspberrypi-firmware soc:firmware: Attached to firmware from 2019-09-24 17:37, variant start
[ 0.160927] raspberrypi-firmware soc:firmware: Firmware hash is 6820edeee4ef3891b95fc01cf02a7abd7ca52f17
[ 0.173283] clocksource: Switched to clocksource timer
[ 0.277799] VFS: Disk quotas dquot_6.6.0
[ 0.277985] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 0.278355] FS-Cache: Loaded
[ 0.278729] CacheFiles: Loaded
[ 0.298192] NET: Registered protocol family 2
[ 0.299580] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes)
[ 0.299706] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.299821] TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.299944] TCP: Hash tables configured (established 4096 bind 4096)
[ 0.300135] UDP hash table entries: 256 (order: 0, 4096 bytes)
[ 0.300205] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[ 0.300645] NET: Registered protocol family 1
[ 0.301808] RPC: Registered named UNIX socket transport module.
[ 0.301873] RPC: Registered udp transport module.
[ 0.301906] RPC: Registered tcp transport module.
[ 0.301934] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.304146] hw perfevents: no irqs for PMU, sampling events not supported
[ 0.304271] hw perfevents: enabled with armv6_1176 PMU driver, 3 counters available
[ 0.308399] Initialise system trusted keyrings
[ 0.308869] workingset: timestamp_bits=14 max_order=17 bucket_order=3
[ 0.323989] FS-Cache: Netfs 'nfs' registered for caching
[ 0.325375] NFS: Registering the id_resolver key type
[ 0.325474] Key type id_resolver registered
[ 0.325510] Key type id_legacy registered
[ 0.325563] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 0.331173] Key type asymmetric registered
[ 0.331245] Asymmetric key parser 'x509' registered
[ 0.331383] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[ 0.331909] io scheduler noop registered
[ 0.331967] io scheduler deadline registered (default)
[ 0.332618] io scheduler cfq registered
[ 0.332672] io scheduler mq-deadline registered (default)
[ 0.332708] io scheduler kyber registered
[ 0.335526] bcm2708_fb soc:fb: FB found 1 display(s)
[ 0.344694] Console: switching to colour frame buffer device 74x28
[ 0.352217] bcm2708_fb soc:fb: Registered framebuffer for display 0, size 592x448
[ 0.361700] bcm2835-rng 20104000.rng: hwrng registered
[ 0.365285] vc-mem: phys_addr:0x00000000 mem_base=0x1ec00000 mem_size:0x20000000(512 MiB)
[ 0.372169] vc-sm: Videocore shared memory driver
[ 0.375994] gpiomem-bcm2835 20200000.gpiomem: Initialised: Registers at 0x20200000
[ 0.402477] brd: module loaded
[ 0.419185] loop: module loaded
[ 0.423053] Loading iSCSI transport class v2.0-870.
[ 0.427167] usbcore: registered new interface driver smsc95xx
[ 0.430180] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[ 0.461345] dwc_otg 20980000.usb: base=(ptrval)
[ 0.664740] Core Release: 2.80a
[ 0.667597] Setting default values for core params
[ 0.670420] Finished setting default values for core params
[ 0.873794] Using Buffer DMA mode
[ 0.876737] Periodic Transfer Interrupt Enhancement - disabled
[ 0.879694] Multiprocessor Interrupt Enhancement - disabled
[ 0.882679] OTG VER PARAM: 0, OTG VER FLAG: 0
[ 0.885781] Dedicated Tx FIFOs mode
[ 0.889576] WARN::dwc_otg_hcd_init:1045: FIQ DMA bounce buffers: virt = db514000 dma = 0x5b514000 len=9024
[ 0.895626] FIQ FSM acceleration enabled for :
Non-periodic Split Transactions
Periodic Split Transactions
High-Speed Isochronous Endpoints
Interrupt/Control Split Transaction hack enabled
[ 0.909423] dwc_otg: Microframe scheduler enabled
[ 0.909678] WARN::hcd_init_root@raspberrypi:/var/log# fiq:457: FIQ on core 0
[ 0.912405] WARN::hcd_init_fiq:458: FIQ ASM at c04fdd70 length 36
[ 0.915121] WARN::hcd_init_fiq:497: MPHI regs_base at dc810000
[ 0.917929] dwc_otg 20980000.usb: DWC OTG Controller
[ 0.920750] dwc_otg 20980000.usb: new USB bus registered, assigned bus number 1
[ 0.926196] dwc_otg 20980000.usb: irq 56, io mem 0x00000000
[ 0.929046] Init: Port Power? op_state=1
[ 0.931801] Init: Power Port (0)
[ 0.934978] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19
[ 0.940472] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 0.946151] usb usb1: Product: DWC OTG Controller
[ 0.949047] usb usb1: Manufacturer: Linux 4.19.75+ dwc_otg_hcd
[ 0.951966] usb usb1: SerialNumber: 20980000.usb
[ 0.955999] hub 1-0:1.0: USB hub found
[ 0.958949] hub 1-0:1.0: 1 port detected
[ 0.962802] dwc_otg: FIQ enabled
[ 0.962816] dwc_otg: NAK holdoff enabled
[ 0.962823] dwc_otg: FIQ split-transaction FSM enabled
[ 0.962843] Module dwc_common_port init
[ 0.963653] usbcore: registered new interface driver usb-storage
[ 0.967047] mousedev: PS/2 mouse device common for all mice
[ 0.971354] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[ 0.974935] bcm2835-cpufreq: min=700000 max=1000000
[ 0.978698] sdhci: Secure Digital Host Controller Interface driver
[ 0.981718] sdhci: Copyright(c) Pierre Ossman
[ 0.985467] sdhost-bcm2835 20202000.mmc: could not get clk, deferring probe
[ 0.991704] sdhci-pltfm: SDHCI platform and OF driver helper
[ 0.995623] ledtrig-cpu: registered to indicate activity on CPUs
[ 0.998922] hidraw: raw HID events driver (C) Jiri Kosina
[ 1.002263] usbcore: registered new interface driver usbhid
[ 1.005420] usbhid: USB HID core driver
[ 1.009663] vchiq: vchiq_init_state: slot_zero = (ptrval), is_master = 0
[ 1.018078] [vc_sm_connected_init]: start
[ 1.028729] [vc_sm_connected_init]: end - returning 0
[ 1.033774] Initializing XFRM netlink socket
[ 1.036858] NET: Registered protocol family 17
[ 1.039906] Key type dns_resolver registered
[ 1.044720] registered taskstats version 1
[ 1.047686] Loading compiled-in X.509 certificates
[ 1.060707] uart-pl011 20201000.serial: cts_event_workaround enabled
[ 1.064072] 20201000.serial: ttyAMA0 at MMIO 0x20201000 (irq = 81, base_baud = 0) is a PL011 rev2
[ 1.962529] console [ttyAMA0] enabled
[ 1.972771] sdhost: log_buf @ (ptrval) (5b513000)
[ 2.026485] mmc0: sdhost-bcm2835 loaded - DMA enabled (>1)
[ 2.036739] of_cfs_init
[ 2.042244] of_cfs_init: OK
[ 2.069193] Waiting for root device PARTUUID=6c586e13-02...
[ 2.078287] random: fast init done
[ 2.085019] Indeed it is in host mode hprt0 = 00021501
[ 2.187167] mmc0: host does not support reading read-only switch, assuming write-enable
[ 2.203443] mmc0: new high speed SDHC card at address 0001
[ 2.213971] mmcblk0: mmc0:0001 00000 14.6 GiB
[ 2.224773] mmcblk0: p1 p2
[ 2.239828] EXT4-fs (mmcblk0p2): INFO: recovery required on readonly filesystem
[ 2.253026] EXT4-fs (mmcblk0p2): write access will be enabled during recovery
[ 2.293449] usb 1-1: new high-speed USB device number 2 using dwc_otg
[ 2.305430] Indeed it is in host mode hprt0 = 00001101
[ 2.504031] EXT4-fs (mmcblk0p2): orphan cleanup on readonly fs
[ 2.513658] EXT4-fs (mmcblk0p2): 1 orphan inode deleted
[ 2.522080] EXT4-fs (mmcblk0p2): recovery complete
[ 2.554151] usb 1-1: New USB device found, idVendor=1a40, idProduct=0101, bcdDevice= 1.00
[ 2.568683] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[ 2.582150] usb 1-1: Product: USB2.0 HUB
[ 2.591088] hub 1-1:1.0: USB hub found
[ 2.599258] hub 1-1:1.0: 4 ports detected
[ 2.618301] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 2.632982] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
[ 2.657192] devtmpfs: mounted
[ 2.668185] Freeing unused kernel memory: 464K
[ 2.676006] This architecture does not have kernel memory protection.
[ 2.685796] Run /sbin/init as init process
[ 3.023419] usb 1-1.2: new full-speed USB device number 3 using dwc_otg
[ 3.266117] usb 1-1.2: New USB device found, idVendor=0fe6, idProduct=9700, bcdDevice= 1.01
[ 3.280804] usb 1-1.2: New USB device strings: Mfr=0, Product=2, SerialNumber=0
[ 3.294406] usb 1-1.2: Product: USB 2.0 10/100M Ethernet Adaptor
[ 3.537511] systemd[1]: System time before build time, advancing clock.
[ 3.724577] NET: Registered protocol family 10
[ 3.734713] Segment Routing with IPv6
[ 3.804405] systemd[1]: systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[ 3.840361] systemd[1]: Detected architecture arm.
[ 3.871131] systemd[1]: Set hostname to .
[ 3.884774] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument
[ 5.413987] uart-pl011 20201000.serial: no DMA platform data
[ 5.819931] systemd[1]: File /lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[ 5.847419] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[ 6.562489] systemd[1]: /lib/systemd/system/lighttpd.service:6: PIDFile= references path below legacy directory /var/run/, updating /var/run/lighttpd.pid → /run/lighttpd.pid; please update the unit file accordingly.
[ 6.784658] random: systemd: uninitialized urandom read (16 bytes read)
[ 6.820442] random: systemd: uninitialized urandom read (16 bytes read)
[ 6.839903] systemd[1]: Created slice system-serial\x2dgetty.slice.
[ 6.855220] random: systemd: uninitialized urandom read (16 bytes read)
[ 6.865697] systemd[1]: Reached target Swap.
[ 6.881233] systemd[1]: Created slice system-getty.slice.
[ 8.817544] systemd[1]: Starting File System Check on Root Device...
[ 8.900983] systemd[1]: Condition check resulted in FUSE Control File System being skipped.
[ 9.101741] systemd[1]: Starting Apply Kernel Variables...
[ 9.291070] systemd[1]: Mounting Kernel Configuration File System...
[ 9.717440] systemd[1]: Mounted Kernel Configuration File System.
[ 9.911395] systemd[1]: Started File System Check Daemon to report status.
[ 10.139092] systemd[1]: Started Apply Kernel Variables.
[ 10.314910] systemd[1]: Started Journal Service.
[ 11.539049] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 12.124588] systemd-journald[83]: Received request to flush runtime journal from PID 1
[ 14.246681] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[ 14.274264] bcm2835_vc_sm_cma_probe: Videocore shared memory driver
[ 14.274291] [vc_sm_connected_init]: start
[ 14.296261] media: Linux media interface: v0.10
[ 14.299190] [vc_sm_connected_init]: installed successfully
[ 14.518271] videodev: Linux video capture interface: v2.00
[ 14.627519] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[ 14.704098] bcm2835_v4l2: module is from the staging directory, the quality is unknown, you have been warned.
[ 14.728331] bcm2835_codec: module is from the staging directory, the quality is unknown, you have been warned.
[ 14.841631] bcm2835-codec bcm2835-codec: Device registered as /dev/video10
[ 14.841724] bcm2835-codec bcm2835-codec: Loaded V4L2 decode
[ 14.852073] bcm2835-codec bcm2835-codec: Device registered as /dev/video11
[ 14.852136] bcm2835-codec bcm2835-codec: Loaded V4L2 encode
[ 14.862817] bcm2835-codec bcm2835-codec: Device registered as /dev/video12
[ 14.862876] bcm2835-codec bcm2835-codec: Loaded V4L2 isp
[ 17.316185] dm9601 1-1.2:1.0 eth0: register 'dm9601' at usb-20980000.usb-1.2, Davicom DM96xx USB 10/100 Ethernet, m.m.m.m
[ 17.316646] usbcore: registered new interface driver dm9601
[ 17.331312] usbcore: registered new interface driver sr9700
[ 22.425432] random: crng init done
[ 22.425456] random: 7 urandom warning(s) missed due to ratelimiting
[ 24.557142] 8021q: 802.1Q VLAN Support v1.8
[ 25.574583] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[ 26.476602] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[ 26.949592] Adding 102396k swap on /var/swap. Priority:-2 extents:1 across:102396k SSFS
[ 26.967067] dm9601 1-1.2:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xFFFF

Logs
If applicable, add the relevant output from dmesg or similar.

Additional context
Add any other relevant context for the problem.

@JamesH65
Copy link
Contributor

Can you please list the steps required to replicate the issue, and describe the setup of your system.

@js290
Copy link
Author

js290 commented Jan 13, 2020

I don't know how to replicate. The pi was unresponsive, and after reboot I found the shared stack trace in the logs. My Pi is running Pi Hole, but it has never crashed before.

@dwrobel
Copy link

dwrobel commented Jan 18, 2020

Can you please list the steps required to replicate the issue, and describe the setup of your system.

I have a similar callstack from 4.19.83:

[173036.430458] ------------[ cut here ]------------
[173036.435289] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:465 dev_watchdog+0x290/0x294
[173036.443781] NETDEV WATCHDOG: eth1 (ax88179_178a): transmit queue 0 timed out
[173036.451059] Modules linked in: tun xt_nat ipt_MASQUERADE nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_REJECT nf_reject_ipv6 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod brcmfmac brcmutil cfg80211 ax88179_178a bcm2835_codec(C) v4l2_mem2mem rfkill bcm2835_v4l2(C) bcm2835_mmal_vchiq(C) videobuf2_dma_contig v4l2_common videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common raspberrypi_hwmon videodev vc_sm_cma(C) media gpio_fan
[173036.526417]  hwmon uio_pdrv_genirq uio fixed zram sch_fq_codel zsmalloc fuse
[173036.535602] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        4.19.83-2.rpi.fc31.armv7hl #1
[173036.548687] Hardware name: BCM2835
[173036.554174] [<80112330>] (unwind_backtrace) from [<8010d594>] (show_stack+0x20/0x24)
[173036.566084] [<8010d594>] (show_stack) from [<80908224>] (dump_stack+0xc8/0x10c)
[173036.577768] [<80908224>] (dump_stack) from [<80125df0>] (__warn.part.0+0xc4/0xe0)
[173036.589713] [<80125df0>] (__warn.part.0) from [<80125e84>] (warn_slowpath_fmt+0x78/0x9c)
[173036.602407] [<80125e84>] (warn_slowpath_fmt) from [<807d11f8>] (dev_watchdog+0x290/0x294)
[173036.615328] [<807d11f8>] (dev_watchdog) from [<8019f308>] (call_timer_fn+0x3c/0x198)
[173036.627892] [<8019f308>] (call_timer_fn) from [<8019f550>] (expire_timers+0xec/0x14c)
[173036.640667] [<8019f550>] (expire_timers) from [<8019f668>] (run_timer_softirq+0xb8/0x1f0)
[173036.653979] [<8019f668>] (run_timer_softirq) from [<801023f0>] (__do_softirq+0x180/0x3e0)
[173036.667479] [<801023f0>] (__do_softirq) from [<8012bdc4>] (irq_exit+0xf8/0x134)
[173036.680390] [<8012bdc4>] (irq_exit) from [<801862c0>] (__handle_domain_irq+0x70/0xc0)
[173036.693957] [<801862c0>] (__handle_domain_irq) from [<801021a4>] (bcm2836_arm_irqchip_handle_irq+0x64/0xa4)
[173036.709527] [<801021a4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
[173036.724305] Exception stack(0x80e01ef0 to 0x80e01f38)
[173036.732344] 1ee0:                                     00000000 395bc000 400e0093 400e0093
[173036.746346] 1f00: 80e00000 00000000 80e04df4 80e04e3c 80e97522 80b8a9ec 10c5387d 80e01f4c
[173036.760295] 1f20: ffffe000 80e01f40 80e00000 80109bcc 400e0013 ffffffff
[173036.769846] [<801019bc>] (__irq_svc) from [<80109bcc>] (arch_cpu_idle+0x34/0x4c)
[173036.782957] [<80109bcc>] (arch_cpu_idle) from [<80923c7c>] (default_idle_call+0x34/0x48)
[173036.796767] [<80923c7c>] (default_idle_call) from [<80158614>] (do_idle+0xec/0x168)
[173036.810119] [<80158614>] (do_idle) from [<8015895c>] (cpu_startup_entry+0x28/0x2c)
[173036.823355] [<8015895c>] (cpu_startup_entry) from [<8091d3fc>] (rest_init+0xb8/0xbc)
[173036.836802] [<8091d3fc>] (rest_init) from [<80d00fec>] (start_kernel+0x490/0x4c0)
[173036.850016] ---[ end trace 8a54cc574983428a ]---

It's from the headless system (no hdmi, keyboard, etc. connected) which works as a router (has two USB ethernet cards, both are: 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet).

It usually happened after a week of stable work.

Later on, I upgraded to:

# uname -a
Linux localhost.localdomain 4.19.93-1.rpi.fc31.armv7hl #1 SMP Tue Jan 7 19:57:20 CET 2020 armv7l armv7l armv7l GNU/Linux

and at the time of writing:

# uptime
 17:31:35 up 10 days, 18:24,  1 user,  load average: 0.00, 0.01, 0.00

I haven't observed it so far.

@ablacksheep
Copy link
Contributor

as @dwrobel said, it might be related

It's from the headless system (no hdmi, keyboard, etc. connected) which works as a router (has two USB ethernet cards, both are: 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet).

It usually happened after a week of stable work.

I have a very similar setup with 2x 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet. One card stopped responding after aprox 10 days with kernel:
Linux ky015 4.19.93-v7l+ #1290 SMP Fri Jan 10 16:45:11 GMT 2020 armv7l GNU/Linux

kernel: [987818.955728] ------------[ cut here ]------------
kernel: [987818.955779] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:466 dev_watchdog+0x294/0x298
kernel: [987818.955796] NETDEV WATCHDOG: enx00249b1e558a (ax88179_178a): transmit queue 0 timed out
kernel: [987818.955809] Modules linked in: nf_conntrack_netlink xt_multiport tun bridge 8021q garp stp llc rtc_ds1307 ax88179_178a brcmfmac brcmutil sha256_generic cfg80211 rfkill vc4 bcm2835_codec(C) drm_kms_helper v4l2_mem2mem v3d gpu_sched bcm2835_v4l2(C) bcm2835_mmal_vchiq(C) videobuf2_dma_contig v4l2_common videobuf2_vmalloc videobuf2_memops nf_log_ipv6 videobuf2_v4l2 videobuf2_common ip6t_REJECT nf_reject_ipv6 videodev media vc_sm_cma(C) drm drm_panel_orientation_quirks xt_hl raspberrypi_hwmon snd_soc_core hwmon ip6_tables snd_compress snd_pcm_dmaengine ip6t_rt syscopyarea sysfillrect sysimgblt fb_sys_fops snd_bcm2835(C) snd_pcm snd_timer snd i2c_bcm2835 rpivid_mem nf_log_ipv4 nf_log_common ipt_REJECT nf_reject_ipv4 uio_pdrv_genirq uio fixed xt_LOG nft_limit xt_limit xt_addrtype nft_chain_nat_ipv4 xt_nat
kernel: [987818.956016]  xt_tcpudp ipt_MASQUERADE nf_nat_ipv4 xt_conntrack nft_compat nft_counter nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables i2c_dev nfnetlink ip_tables x_tables ipv6
kernel: [987818.956090] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        4.19.88-v7l+ #1284
kernel: [987818.956099] Hardware name: BCM2835
kernel: [987818.956127] [<c0212e04>] (unwind_backtrace) from [<c020d5e0>] (show_stack+0x20/0x24)
kernel: [987818.956147] [<c020d5e0>] (show_stack) from [<c09afa48>] (dump_stack+0xe0/0x124)
kernel: [987818.956169] [<c09afa48>] (dump_stack) from [<c0222544>] (__warn+0x104/0x11c)
kernel: [987818.956187] [<c0222544>] (__warn) from [<c02225b4>] (warn_slowpath_fmt+0x58/0x74)
kernel: [987818.956205] [<c02225b4>] (warn_slowpath_fmt) from [<c08d5a7c>] (dev_watchdog+0x294/0x298)
kernel: [987818.956225] [<c08d5a7c>] (dev_watchdog) from [<c029b848>] (call_timer_fn+0x3c/0x198)
kernel: [987818.956243] [<c029b848>] (call_timer_fn) from [<c029ba90>] (expire_timers+0xec/0x14c)
kernel: [987818.956259] [<c029ba90>] (expire_timers) from [<c029bba8>] (run_timer_softirq+0xb8/0x1ec)
kernel: [987818.956275] [<c029bba8>] (run_timer_softirq) from [<c0202410>] (__do_softirq+0x190/0x3f0)
kernel: [987818.956292] [<c0202410>] (__do_softirq) from [<c02286cc>] (irq_exit+0xfc/0x120)
kernel: [987818.956310] [<c02286cc>] (irq_exit) from [<c0281d9c>] (__handle_domain_irq+0x70/0xc4)
kernel: [987818.956325] [<c0281d9c>] (__handle_domain_irq) from [<c0202244>] (gic_handle_irq+0x4c/0x88)
kernel: [987818.956340] [<c0202244>] (gic_handle_irq) from [<c02019bc>] (__irq_svc+0x5c/0x7c)
kernel: [987818.956350] Exception stack(0xc1001ed8 to 0xc1001f20)
kernel: [987818.956360] 1ec0:                                                       c0209ae4 00000000
kernel: [987818.956372] 1ee0: 40000093 40000093 c1004dbc c1000000 c1004e04 00000001 c109681a c10abf40
kernel: [987818.956384] 1f00: c0e67a38 c1001f34 c10051c4 c1001f28 00000000 c0209ae8 40000013 ffffffff
kernel: [987818.956406] [<c02019bc>] (__irq_svc) from [<c0209ae8>] (arch_cpu_idle+0x34/0x4c)
kernel: [987818.956425] [<c0209ae8>] (arch_cpu_idle) from [<c09ccd0c>] (default_idle_call+0x34/0x48)
kernel: [987818.956441] [<c09ccd0c>] (default_idle_call) from [<c0254a14>] (do_idle+0xec/0x16c)
kernel: [987818.956456] [<c0254a14>] (do_idle) from [<c0254d54>] (cpu_startup_entry+0x28/0x2c)
kernel: [987818.956472] [<c0254d54>] (cpu_startup_entry) from [<c09c63a0>] (rest_init+0xbc/0xc0)
kernel: [987818.956492] [<c09c63a0>] (rest_init) from [<c0e00fe0>] (start_kernel+0x4b4/0x4e4)
kernel: [987818.956509] ---[ end trace e4e030a1002f670f ]---

@ablacksheep
Copy link
Contributor

13:46:09 up 18 days, 14:03, 1 user, load average: 0,00, 0,01, 0,00

Same port crashed again, not sure if it is a pattern after 2 crashes, but so far it is the one plugged in the usb2 port that crash. The one pluggen in the usb3 seems to work fine. Might not be related, but i have to mention it. I can also add that the two interfaces have different traffic patterns, the one not crashing has continues traffic, and the port that crash has traffic spikes from a few times a day to some days between. Between the traffic spikes, there is basically no traffic.

Bus 002 Device 002: ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
------------[ cut here ]------------
[1605418.628803] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:466 dev_watchdog+0x294/0x298
[1605418.628829] NETDEV WATCHDOG: enx00249b1e558a (ax88179_178a): transmit queue 0 timed out
[1605418.628843] Modules linked in: xt_recent nf_conntrack_netlink xt_multiport tun bridge 8021q garp stp llc rtc_ds1307 ax88179_178a brcmfmac brcmutil nf_log_ipv6 sha256_generic ip6t_REJECT nf_reject_ipv6 cfg80211 rfkill xt_hl vc4 ip6_tables ip6t_rt v3d raspberrypi_hwmon drm_kms_helper hwmon gpu_sched drm drm_panel_orientation_quirks snd_soc_core snd_bcm2835(C) snd_compress bcm2835_codec(C) snd_pcm_dmaengine v4l2_mem2mem bcm2835_v4l2(C) snd_pcm syscopyarea bcm2835_mmal_vchiq(C) v4l2_common videobuf2_vmalloc snd_timer videobuf2_dma_contig nf_log_ipv4 nf_log_common sysfillrect snd sysimgblt vc_sm_cma(C) videobuf2_memops ipt_REJECT fb_sys_fops nf_reject_ipv4 videobuf2_v4l2 videobuf2_common xt_LOG videodev media i2c_bcm2835 rpivid_mem nft_limit uio_pdrv_genirq uio fixed xt_limit xt_addrtype nft_chain_nat_ipv4
[1605418.629058]  xt_nat xt_tcpudp ipt_MASQUERADE nf_nat_ipv4 xt_conntrack nft_compat nft_counter nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink i2c_dev ip_tables x_tables ipv6
[1605418.629146] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        4.19.93-v7l+ #1290
[1605418.629154] Hardware name: BCM2835
[1605418.629187] [<c0212e04>] (unwind_backtrace) from [<c020d5e0>] (show_stack+0x20/0x24)
[1605418.629210] [<c020d5e0>] (show_stack) from [<c09b05e8>] (dump_stack+0xe0/0x124)
[1605418.629237] [<c09b05e8>] (dump_stack) from [<c0222544>] (__warn+0x104/0x11c)
[1605418.629255] [<c0222544>] (__warn) from [<c02225b4>] (warn_slowpath_fmt+0x58/0x74)
[1605418.629273] [<c02225b4>] (warn_slowpath_fmt) from [<c08d664c>] (dev_watchdog+0x294/0x298)
[1605418.629294] [<c08d664c>] (dev_watchdog) from [<c029b8c4>] (call_timer_fn+0x3c/0x198)
[1605418.629315] [<c029b8c4>] (call_timer_fn) from [<c029bb0c>] (expire_timers+0xec/0x14c)
[1605418.629333] [<c029bb0c>] (expire_timers) from [<c029bc24>] (run_timer_softirq+0xb8/0x1ec)
[1605418.629351] [<c029bc24>] (run_timer_softirq) from [<c0202410>] (__do_softirq+0x190/0x3f0)
[1605418.629369] [<c0202410>] (__do_softirq) from [<c02286cc>] (irq_exit+0xfc/0x120)
[1605418.629387] [<c02286cc>] (irq_exit) from [<c0281e18>] (__handle_domain_irq+0x70/0xc4)
[1605418.629402] [<c0281e18>] (__handle_domain_irq) from [<c0202244>] (gic_handle_irq+0x4c/0x88)
[1605418.629420] [<c0202244>] (gic_handle_irq) from [<c02019bc>] (__irq_svc+0x5c/0x7c)
[1605418.629429] Exception stack(0xc1001ed8 to 0xc1001f20)
[1605418.629447] 1ec0:                                                       c0209ae4 00000000
[1605418.629462] 1ee0: 40000093 40000093 c1004dbc c1000000 c1004e04 00000001 c109685a c10abf80
[1605418.629475] 1f00: c0e67a38 c1001f34 c10051c4 c1001f28 00000000 c0209ae8 40000013 ffffffff
[1605418.629495] [<c02019bc>] (__irq_svc) from [<c0209ae8>] (arch_cpu_idle+0x34/0x4c)
[1605418.629513] [<c0209ae8>] (arch_cpu_idle) from [<c09cd884>] (default_idle_call+0x34/0x48)
[1605418.629532] [<c09cd884>] (default_idle_call) from [<c0254a5c>] (do_idle+0xec/0x16c)
[1605418.629548] [<c0254a5c>] (do_idle) from [<c0254d9c>] (cpu_startup_entry+0x28/0x2c)
[1605418.629565] [<c0254d9c>] (cpu_startup_entry) from [<c09c6f18>] (rest_init+0xbc/0xc0)
[1605418.629584] [<c09c6f18>] (rest_init) from [<c0e00fe4>] (start_kernel+0x4b8/0x4e8)
[1605418.629602] ---[ end trace 7ab266a8281fe475 ]---

@ablacksheep
Copy link
Contributor

Same again, so incr counter.

Same port crashed again, not sure if it is a pattern after 2 crashes, but so far it is the one plugged in the usb2 port that crash. The one pluggen in the usb3 seems to work fine. Might not be related, but i have to mention it. I can also add that the two interfaces have different traffic patterns, the one not crashing has continues traffic, and the port that crash has traffic spikes from a few times a day to some days between. Between the traffic spikes, there is basically no traffic.

Last known traffic on interface was Feb 14 19:34:55.
Crash happend when i tried to pass traffic today at Feb 18.
15:49:29 up 3 days, 20:16, 1 user, load average: 0,00, 0,00, 0,00

Some wild guessing.. Could this be an usb power related issue?
Something like the device deadlocks when power control is trying to suspend it?

kernel: [332213.777361] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:466 dev_watchdog+0x294/0x298
kernel: [332213.777383] NETDEV WATCHDOG: enx00249b1e558a (ax88179_178a): transmit queue 0 timed out
kernel: [332213.777402] Modules linked in: nf_conntrack_netlink xt_multiport tun bridge 8021q garp stp llc rtc_ds1307 ax88179_178a nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 vc4 xt_hl brcmfmac ip6_tables drm_kms_helper v3d brcmutil gpu_sched ip6t_rt drm raspberrypi_hwmon hwmon drm_panel_orientation_quirks sha256_generic snd_soc_core snd_compress snd_pcm_dmaengine cfg80211 snd_bcm2835(C) snd_pcm bcm2835_codec(C) nf_log_ipv4 snd_timer nf_log_common syscopyarea sysfillrect rfkill sysimgblt v4l2_mem2mem fb_sys_fops snd videobuf2_dma_contig bcm2835_v4l2(C) ipt_REJECT nf_reject_ipv4 bcm2835_mmal_vchiq(C) v4l2_common i2c_bcm2835 videobuf2_vmalloc xt_LOG videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media vc_sm_cma(C) rpivid_mem nft_limit uio_pdrv_genirq uio xt_limit xt_addrtype nft_chain_nat_ipv4 xt_nat xt_tcpudp
kernel: [332213.777617]  ipt_MASQUERADE nf_nat_ipv4 xt_conntrack nft_compat nft_counter nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink i2c_dev ip_tables x_tables ipv6
kernel: [332213.777696] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        4.19.97-v7l+ #1294
kernel: [332213.777706] Hardware name: BCM2835
kernel: [332213.777738] [<c0212e04>] (unwind_backtrace) from [<c020d5e0>] (show_stack+0x20/0x24)
kernel: [332213.777761] [<c020d5e0>] (show_stack) from [<c09b15c8>] (dump_stack+0xe0/0x124)
kernel: [332213.777782] [<c09b15c8>] (dump_stack) from [<c0222544>] (__warn+0x104/0x11c)
kernel: [332213.777800] [<c0222544>] (__warn) from [<c02225b4>] (warn_slowpath_fmt+0x58/0x74)
kernel: [332213.777822] [<c02225b4>] (warn_slowpath_fmt) from [<c08d7624>] (dev_watchdog+0x294/0x298)
kernel: [332213.777844] [<c08d7624>] (dev_watchdog) from [<c029b8c4>] (call_timer_fn+0x3c/0x198)
kernel: [332213.777863] [<c029b8c4>] (call_timer_fn) from [<c029bb0c>] (expire_timers+0xec/0x14c)
kernel: [332213.777880] [<c029bb0c>] (expire_timers) from [<c029bc24>] (run_timer_softirq+0xb8/0x1ec)
kernel: [332213.777900] [<c029bc24>] (run_timer_softirq) from [<c0202410>] (__do_softirq+0x190/0x3f0)
kernel: [332213.777919] [<c0202410>] (__do_softirq) from [<c02286cc>] (irq_exit+0xfc/0x120)
kernel: [332213.777937] [<c02286cc>] (irq_exit) from [<c0281e18>] (__handle_domain_irq+0x70/0xc4)
kernel: [332213.777952] [<c0281e18>] (__handle_domain_irq) from [<c0202244>] (gic_handle_irq+0x4c/0x88)
kernel: [332213.777971] [<c0202244>] (gic_handle_irq) from [<c02019bc>] (__irq_svc+0x5c/0x7c)
kernel: [332213.777981] Exception stack(0xc1001ed8 to 0xc1001f20)
kernel: [332213.777993] 1ec0:                                                       c0209ae4 00000000
kernel: [332213.778007] 1ee0: 40000093 40000093 c1004dbc c1000000 c1004e04 00000001 c109689a c10abfc0
kernel: [332213.778023] 1f00: c0e67a38 c1001f34 c10051c4 c1001f28 00000000 c0209ae8 40000013 ffffffff
kernel: [332213.778043] [<c02019bc>] (__irq_svc) from [<c0209ae8>] (arch_cpu_idle+0x34/0x4c)
kernel: [332213.778061] [<c0209ae8>] (arch_cpu_idle) from [<c09ce864>] (default_idle_call+0x34/0x48)
kernel: [332213.778081] [<c09ce864>] (default_idle_call) from [<c0254a5c>] (do_idle+0xec/0x16c)
kernel: [332213.778098] [<c0254a5c>] (do_idle) from [<c0254d9c>] (cpu_startup_entry+0x28/0x2c)
kernel: [332213.778117] [<c0254d9c>] (cpu_startup_entry) from [<c09c7ef8>] (rest_init+0xbc/0xc0)
kernel: [332213.778139] [<c09c7ef8>] (rest_init) from [<c0e00fe4>] (start_kernel+0x4b8/0x4e8)
kernel: [332213.778156] ---[ end trace 0500650fdc7f74d3 ]---```

@ablacksheep
Copy link
Contributor

@JamesH65
Created a close to identical testrig, it failed pretty quick. Same pattern here usb2 interfaced one crashed. Both interfaces has close to no traffic in this test rig(i suggest they have some initial dhcp client traffic), as i just connect the two usb interfaces together with a cable without sending any additional traffic over them.

However, when i did add the interfaces to different network namespaces, and sendt loads of traffic over them, then they stayed up healty.

syslog-testrig2.txt

To reproduce:
Raspberry Pi 4 Model B 4GB RAM
Get two adapters, plug one in the usb3 and one in the usb2 port. Connect a cable, so link is up. Wait.
Remote access to the testrig can be provided to the right person, if that person feels for helping debugging it. :)

Info:

Raspberry Pi reference 2019-09-26
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc3fc13cf3a2f8b464e129be, stage2

Jan 22 2020 17:23:15 
Copyright (c) 2012 Broadcom
version acb03c7d18aa08f25f6419d696d0cc39761d3e62 (clean) (release) (start)

Linux raspberrypi 4.19.97-v7l+ #1293 SMP Wed Jan 22 17:16:14 GMT 2020 armv7l GNU/Linux

@TheTiEr
Copy link

TheTiEr commented Feb 26, 2020

I get a similar issue with a Raspberry Pi 3 B+

It runs as a headless system only connected to a webcam and an ehternet cable. It is powered via an POE Shield.

Feb 26 09:40:21 beta2-pi kernel: [20113.151890] ------------[ cut here ]------------
Feb 26 09:40:21 beta2-pi kernel: [20113.151964] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:466 dev_watchdog+0x294/0x298
Feb 26 09:40:21 beta2-pi kernel: [20113.151994] NETDEV WATCHDOG: eth0 (lan78xx): transmit queue 0 timed out
Feb 26 09:40:21 beta2-pi kernel: [20113.152004] Modules linked in: bnep hci_uart btbcm serdev bluetooth ecdh_generic 8021q garp stp llc evdev brcmfmac brcmutil snd_usb_audio snd_hwdep sha256_generic snd_usbmidi_lib snd_rawmidi uvcvideo snd_seq_device cfg80211 rfkill snd_bcm2835(C) raspberrypi_hwmon snd_pcm bcm2835_codec(C) v4l2_mem2mem snd_timer bcm2835_v4l2(C) snd videobuf2_dma_contig bcm2835_mmal_vchiq(C) v4l2_common videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev vc_sm_cma(C) media rpi_poe_fan hwmon uio_pdrv_genirq uio fixed ip_tables x_tables ipv6
Feb 26 09:40:21 beta2-pi kernel: [20113.152208] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G C 4.19.102-v7+ #1295
Feb 26 09:40:21 beta2-pi kernel: [20113.152213] Hardware name: BCM2835
Feb 26 09:40:21 beta2-pi kernel: [20113.152242] [<801120c0>] (unwind_backtrace) from [<8010d5f4>] (show_stack+0x20/0x24)
Feb 26 09:40:21 beta2-pi kernel: [20113.152260] [<8010d5f4>] (show_stack) from [<80849c68>] (dump_stack+0xe0/0x124)
Feb 26 09:40:21 beta2-pi kernel: [20113.152279] [<80849c68>] (dump_stack) from [<80120c9c>] (__warn+0x104/0x11c)
Feb 26 09:40:21 beta2-pi kernel: [20113.152294] [<80120c9c>] (__warn) from [<80120d0c>] (warn_slowpath_fmt+0x58/0x74)
Feb 26 09:40:21 beta2-pi kernel: [20113.152311] [<80120d0c>] (warn_slowpath_fmt) from [<8076f218>] (dev_watchdog+0x294/0x298)
Feb 26 09:40:21 beta2-pi kernel: [20113.152329] [<8076f218>] (dev_watchdog) from [<80199044>] (call_timer_fn+0x3c/0x198)
Feb 26 09:40:21 beta2-pi kernel: [20113.152345] [<80199044>] (call_timer_fn) from [<8019928c>] (expire_timers+0xec/0x14c)
Feb 26 09:40:21 beta2-pi kernel: [20113.152359] [<8019928c>] (expire_timers) from [<801993a4>] (run_timer_softirq+0xb8/0x1ec)
Feb 26 09:40:21 beta2-pi kernel: [20113.152375] [<801993a4>] (run_timer_softirq) from [<80102410>] (__do_softirq+0x190/0x3f0)
Feb 26 09:40:21 beta2-pi kernel: [20113.152392] [<80102410>] (__do_softirq) from [<80126e24>] (irq_exit+0xfc/0x120)
Feb 26 09:40:21 beta2-pi kernel: [20113.152409] [<80126e24>] (irq_exit) from [<80180270>] (__handle_domain_irq+0x70/0xc4)
Feb 26 09:40:21 beta2-pi kernel: [20113.152424] [<80180270>] (__handle_domain_irq) from [<801021b4>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
Feb 26 09:40:21 beta2-pi kernel: [20113.152439] [<801021b4>] (bcm2836_arm_irqchip_handle_irq) from [<801019bc>] (__irq_svc+0x5c/0x7c)
Feb 26 09:40:21 beta2-pi kernel: [20113.152446] Exception stack(0x80d01ee8 to 0x80d01f30)
Feb 26 09:40:21 beta2-pi kernel: [20113.152458] 1ee0: 80109ae4 00000000 40000093 40000093 80d04d70 80d00000
Feb 26 09:40:21 beta2-pi kernel: [20113.152471] 1f00: 80d04db8 00000001 80d8efbe babff9c0 80c64a38 80d01f44 80d0517c 80d01f38
Feb 26 09:40:21 beta2-pi kernel: [20113.152480] 1f20: 00000000 80109ae8 40000013 ffffffff
Feb 26 09:40:21 beta2-pi kernel: [20113.152497] [<801019bc>] (__irq_svc) from [<80109ae8>] (arch_cpu_idle+0x34/0x4c)
Feb 26 09:40:21 beta2-pi kernel: [20113.152515] [<80109ae8>] (arch_cpu_idle) from [<80866d54>] (default_idle_call+0x34/0x48)
Feb 26 09:40:21 beta2-pi kernel: [20113.152533] [<80866d54>] (default_idle_call) from [<80152e68>] (do_idle+0xec/0x16c)
Feb 26 09:40:21 beta2-pi kernel: [20113.152552] [<80152e68>] (do_idle) from [<801531a8>] (cpu_startup_entry+0x28/0x2c)
Feb 26 09:40:21 beta2-pi kernel: [20113.152570] [<801531a8>] (cpu_startup_entry) from [<80860400>] (rest_init+0xbc/0xc0)
Feb 26 09:40:21 beta2-pi kernel: [20113.152589] [<80860400>] (rest_init) from [<80c00fb0>] (start_kernel+0x484/0x4b4)
Feb 26 09:40:21 beta2-pi kernel: [20113.152620] ---[ end trace 6a98d99e7596fecd ]---
Feb 26 09:40:21 beta2-pi kernel: [20113.153022] WARN::dwc_otg_hcd_urb_dequeue:639: Timed out waiting for FSM NP transfer to complete on 2
Feb 26 09:40:21 beta2-pi kernel: [20113.154499] lan78xx 1-1.1.1:1.0 eth0: kevent 0 may have been dropped
Feb 26 09:40:21 beta2-pi kernel: [20113.154638] lan78xx 1-1.1.1:1.0 eth0: kevent 0 may have been dropped
Feb 26 09:40:21 beta2-pi kernel: [20113.154773] lan78xx 1-1.1.1:1.0 eth0: kevent 0 may have been dropped
Feb 26 09:40:21 beta2-pi kernel: [20113.154914] lan78xx 1-1.1.1:1.0 eth0: kevent 0 may have been dropped
Feb 26 09:40:21 beta2-pi kernel: [20113.155057] lan78xx 1-1.1.1:1.0 eth0: kevent 0 may have been dropped
Feb 26 09:40:21 beta2-pi kernel: [20113.155195] lan78xx 1-1.1.1:1.0 eth0: kevent 0 may have been dropped
Feb 26 09:40:21 beta2-pi kernel: [20113.155320] lan78xx 1-1.1.1:1.0 eth0: kevent 0 may have been dropped
Feb 26 09:40:21 beta2-pi kernel: [20113.155772] lan78xx 1-1.1.1:1.0 eth0: kevent 0 may have been dropped

It becomes completly unresponsive over the network. The last message is repeated until a reboot by plugging it out and in again.

uname -a:

Linux beta2-pi 4.19.102-v7+ #1295 SMP Thu Feb 6 15:43:59 GMT 2020 armv7l GNU/Linux

EDIT: Changed Debug messages to quote to improve readability

@ablacksheep
Copy link
Contributor

I get a similar issue with a Raspberry Pi 3 B+

It runs as a headless system only connected to a webcam and an ehternet cable. It is powered via an POE Shield.

I suspect it is related.

It seems to have the usb2 bus in common. As far as i remember the pi3b+ ethernet is provided via onboard chip connected to the USB 2.0 bus and so far holds true for my p4 testing above. it is only the adapter connected to the usb2 bus that crashes.

@ablacksheep
Copy link
Contributor

Updated test-rig yesterday(25.02.2020), apt update && apt full-upgrade and rpi-update. Crashed some hours later.. adapter plugged in usb2 port.

Feb 26 04:50:17 raspberrypi kernel: [19287.339544] ------------[ cut here ]------------
Feb 26 04:50:17 raspberrypi kernel: [19287.339592] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:466 dev_watchdog+0x294/0x298
Feb 26 04:50:17 raspberrypi kernel: [19287.339619] NETDEV WATCHDOG: enx00249b5b748d (ax88179_178a): transmit queue 0 timed out
Feb 26 04:50:17 raspberrypi kernel: [19287.339634] Modules linked in: xt_multiport bnep hci_uart btbcm serdev bluetooth ecdh_generic 8021q garp stp llc sg brcmfmac ax88179_178a vc4 brcmutil drm_kms_helper sha256_generic snd_soc_core snd_compress cfg80211 v3d snd_pcm_dmaengine gpu_sched syscopyarea bcm2835_codec(C) sysfillrect rfkill bcm2835_v4l2(C) snd_bcm2835(C) v4l2_mem2mem drm bcm2835_mmal_vchiq(C) snd_pcm sysimgblt videobuf2_dma_contig v4l2_common videobuf2_vmalloc drm_panel_orientation_quirks videobuf2_memops snd_timer fb_sys_fops videobuf2_v4l2 raspberrypi_hwmon videobuf2_common hwmon snd videodev media vc_sm_cma(C) rpivid_mem uio_pdrv_genirq uio nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 xt_hl ip6_tables ip6t_rt nf_log_ipv4 nf_log_common ipt_REJECT nf_reject_ipv4 xt_LOG nft_limit xt_limit xt_addrtype xt_tcpudp xt_conntrack nft_compat
Feb 26 04:50:17 raspberrypi kernel: [19287.339856]  nft_counter nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink ip_tables x_tables ipv6
Feb 26 04:50:17 raspberrypi kernel: [19287.339921] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        4.19.105-v7l+ #1296
Feb 26 04:50:17 raspberrypi kernel: [19287.339933] Hardware name: BCM2835
Feb 26 04:50:17 raspberrypi kernel: [19287.339961] [<c0212e04>] (unwind_backtrace) from [<c020d5e0>] (show_stack+0x20/0x24)
Feb 26 04:50:17 raspberrypi kernel: [19287.339981] [<c020d5e0>] (show_stack) from [<c09b63a8>] (dump_stack+0xe0/0x124)
Feb 26 04:50:17 raspberrypi kernel: [19287.340004] [<c09b63a8>] (dump_stack) from [<c0222548>] (__warn+0x104/0x11c)
Feb 26 04:50:17 raspberrypi kernel: [19287.340021] [<c0222548>] (__warn) from [<c02225b8>] (warn_slowpath_fmt+0x58/0x74)
Feb 26 04:50:17 raspberrypi kernel: [19287.340040] [<c02225b8>] (warn_slowpath_fmt) from [<c08dc098>] (dev_watchdog+0x294/0x298)
Feb 26 04:50:17 raspberrypi kernel: [19287.340065] [<c08dc098>] (dev_watchdog) from [<c029b95c>] (call_timer_fn+0x3c/0x198)
Feb 26 04:50:17 raspberrypi kernel: [19287.340083] [<c029b95c>] (call_timer_fn) from [<c029bba4>] (expire_timers+0xec/0x14c)
Feb 26 04:50:17 raspberrypi kernel: [19287.340098] [<c029bba4>] (expire_timers) from [<c029bcbc>] (run_timer_softirq+0xb8/0x1ec)
Feb 26 04:50:17 raspberrypi kernel: [19287.340118] [<c029bcbc>] (run_timer_softirq) from [<c0202410>] (__do_softirq+0x190/0x3f0)
Feb 26 04:50:17 raspberrypi kernel: [19287.340139] [<c0202410>] (__do_softirq) from [<c02286d0>] (irq_exit+0xfc/0x120)
Feb 26 04:50:17 raspberrypi kernel: [19287.340157] [<c02286d0>] (irq_exit) from [<c0281e54>] (__handle_domain_irq+0x70/0xc4)
Feb 26 04:50:17 raspberrypi kernel: [19287.340173] [<c0281e54>] (__handle_domain_irq) from [<c0202244>] (gic_handle_irq+0x4c/0x88)
Feb 26 04:50:17 raspberrypi kernel: [19287.340187] [<c0202244>] (gic_handle_irq) from [<c02019bc>] (__irq_svc+0x5c/0x7c)
Feb 26 04:50:17 raspberrypi kernel: [19287.340204] Exception stack(0xc1001ed8 to 0xc1001f20)
Feb 26 04:50:17 raspberrypi kernel: [19287.340219] 1ec0:                                                       c0209ae4 00000000
Feb 26 04:50:17 raspberrypi kernel: [19287.340232] 1ee0: 40000093 40000093 c1004dbc c1000000 c1004e04 00000001 c10968da c10abfc0
Feb 26 04:50:17 raspberrypi kernel: [19287.340244] 1f00: c0e67a38 c1001f34 c10051c4 c1001f28 00000000 c0209ae8 40000013 ffffffff
Feb 26 04:50:17 raspberrypi kernel: [19287.340260] [<c02019bc>] (__irq_svc) from [<c0209ae8>] (arch_cpu_idle+0x34/0x4c)
Feb 26 04:50:17 raspberrypi kernel: [19287.340279] [<c0209ae8>] (arch_cpu_idle) from [<c09d3644>] (default_idle_call+0x34/0x48)
Feb 26 04:50:17 raspberrypi kernel: [19287.340300] [<c09d3644>] (default_idle_call) from [<c0254a48>] (do_idle+0xec/0x16c)
Feb 26 04:50:17 raspberrypi kernel: [19287.340316] [<c0254a48>] (do_idle) from [<c0254d88>] (cpu_startup_entry+0x28/0x2c)
Feb 26 04:50:17 raspberrypi kernel: [19287.340333] [<c0254d88>] (cpu_startup_entry) from [<c09cccd8>] (rest_init+0xbc/0xc0)
Feb 26 04:50:17 raspberrypi kernel: [19287.340352] [<c09cccd8>] (rest_init) from [<c0e00fe4>] (start_kernel+0x4b8/0x4e8)
Feb 26 04:50:17 raspberrypi kernel: [19287.340369] ---[ end trace ca8f5cbe348fd389 ]---

@ablacksheep
Copy link
Contributor

ablacksheep commented Mar 22, 2020

Tested driver #2999 mentioned in #2942, so far it has given 17 days uptime with no fault. One thing i noticed is that they disable hub_initiated_lpm. They prob also does more stuff, but I havn't had time to look into that.

For the upstream driver it might look like this is a pattern,
it seems like "transmit queue 0 timed out" typical happens when when usbcore or the devices initiate lpm and the device lock up because of failed or incorrect pre requirements before lpm can be achieved. I tested by incr time before lpm and by disabled lpm on usb2, and the timeout crashes stopped happening. However, without the crashes the device still locked up. It looked like something still activating the lpm state of the device, maybe the device itself?

If someone is annoyed by the issue with "ID 0b95:1790 ASIX Electronics Corp. AX88179" and the Pi4¿ It can simply be worked around by forcing usb2 device not to enter lpm by running an endless ping with standard 1 second interval. (make sure you ping something that leaves the correct interface.) So far I got 22 days uptime on a system with this workaround.

@ablacksheep
Copy link
Contributor

10:45:53 up 15 days, 13:19, 1 user, load average: 0,02, 0,05, 0,00
With no bad behavior on test rig after moving both adapters to the usb3 ports running original kernel shipped driver module. I will test moving 1 or both the modules back to usb2 ports. I expect it to crash fairly quick.

@ablacksheep
Copy link
Contributor

ablacksheep commented May 13, 2020

I expect it to crash fairly quick.

I messed up the testrig, will have it back running during the weekend.

@ablacksheep
Copy link
Contributor

Tested with a newer kernel, still crashing. But I discovered that the crash of the device(AX88179) in the usb2.port03 seemed to be dependent on which of the usb3 ports the other device(AX88179) was plugged into. The device plugged in usb2 crashed(fast aprox6h.) when other device was plugged in usb3.port00. Moving the device in the usb3 back to usb3.port01 to check if device in usb2 still crashes, and observed pattern of not crashing in usb3.port01 was just a coincidence.

Linux raspberrypi 4.19.122-v7l-24.05.2020+ #1 SMP Sun May 24 13:13:51 BST 2020 armv7l GNU/Linux
Jun  3 10:30:58 raspberrypi kernel: [    0.000000] Booting Linux on physical CPU 0x0
Jun  3 15:50:43 raspberrypi kernel: [19177.332749] ------------[ cut here ]------------
Jun  3 15:50:43 raspberrypi kernel: [19177.332793] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:466 dev_watchdog+0x298/0x29c
Jun  3 15:50:43 raspberrypi kernel: [19177.332815] NETDEV WATCHDOG: enx00249b2c7dfb (ax88179_178a): transmit queue 0 timed out
Jun  3 15:50:43 raspberrypi kernel: [19177.332832] Modules linked in: xt_multiport bnep hci_uart btbcm serdev bluetooth ecdh_generic 8021q garp stp llc ax88179_178a brcmfmac brcmutil vc4 drm_kms_helper v3d gpu_sched bcm2835_codec(C) cfg80211 rfkill v4l2_mem2mem bcm2835_v4l2(C) videobuf2_dma_contig bcm2835_mmal_vchiq(C) v4l2_common raspberrypi_hwmon videobuf2_vmalloc hwmon videobuf2_memops videobuf2_v4l2 drm videobuf2_common drm_panel_orientation_quirks videodev snd_soc_core media snd_bcm2835(C) vc_sm_cma(C) snd_compress snd_pcm_dmaengine snd_pcm syscopyarea sysfillrect snd_timer sysimgblt fb_sys_fops snd ip6t_REJECT nf_reject_ipv6 rpivid_mem uio_pdrv_genirq uio xt_hl ip6_tables ip6t_rt ipt_REJECT nf_reject_ipv4 nft_limit xt_limit xt_addrtype xt_tcpudp xt_conntrack nft_compat nft_counter nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp
Jun  3 15:50:43 raspberrypi kernel: [19177.333054]  nf_nat nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink ip_tables x_tables ipv6
Jun  3 15:50:43 raspberrypi kernel: [19177.333111] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        4.19.122-v7l-24.05.2020+ #1
Jun  3 15:50:43 raspberrypi kernel: [19177.333120] Hardware name: BCM2835
Jun  3 15:50:43 raspberrypi kernel: [19177.333151] [<c0212d5c>] (unwind_backtrace) from [<c020d370>] (show_stack+0x20/0x24)
Jun  3 15:50:43 raspberrypi kernel: [19177.333171] [<c020d370>] (show_stack) from [<c09c0034>] (dump_stack+0xd8/0x11c)
Jun  3 15:50:43 raspberrypi kernel: [19177.333192] [<c09c0034>] (dump_stack) from [<c02224d8>] (__warn.part.3+0xcc/0xe8)
Jun  3 15:50:43 raspberrypi kernel: [19177.333210] [<c02224d8>] (__warn.part.3) from [<c022256c>] (warn_slowpath_fmt+0x78/0x94)
Jun  3 15:50:43 raspberrypi kernel: [19177.333227] [<c022256c>] (warn_slowpath_fmt) from [<c08e6e88>] (dev_watchdog+0x298/0x29c)
Jun  3 15:50:43 raspberrypi kernel: [19177.333250] [<c08e6e88>] (dev_watchdog) from [<c029c048>] (call_timer_fn+0x3c/0x1a4)
Jun  3 15:50:43 raspberrypi kernel: [19177.333269] [<c029c048>] (call_timer_fn) from [<c029c2a8>] (expire_timers+0xf8/0x168)
Jun  3 15:50:43 raspberrypi kernel: [19177.333286] [<c029c2a8>] (expire_timers) from [<c029c3cc>] (run_timer_softirq+0xb4/0x1e4)
Jun  3 15:50:43 raspberrypi kernel: [19177.333302] [<c029c3cc>] (run_timer_softirq) from [<c02023f8>] (__do_softirq+0x188/0x410)
Jun  3 15:50:43 raspberrypi kernel: [19177.333324] [<c02023f8>] (__do_softirq) from [<c0228520>] (irq_exit+0xf8/0x134)
Jun  3 15:50:43 raspberrypi kernel: [19177.333346] [<c0228520>] (irq_exit) from [<c02825dc>] (__handle_domain_irq+0x70/0xc4)
Jun  3 15:50:43 raspberrypi kernel: [19177.333364] [<c02825dc>] (__handle_domain_irq) from [<c0202234>] (gic_handle_irq+0x4c/0x88)
Jun  3 15:50:43 raspberrypi kernel: [19177.333380] [<c0202234>] (gic_handle_irq) from [<c02019bc>] (__irq_svc+0x5c/0x7c)
Jun  3 15:50:43 raspberrypi kernel: [19177.333389] Exception stack(0xc1001ed0 to 0xc1001f18)
Jun  3 15:50:43 raspberrypi kernel: [19177.333400] 1ec0:                                     c0209ab4 00000000 40000093 40000093
Jun  3 15:50:43 raspberrypi kernel: [19177.333412] 1ee0: ffffe000 c1004db8 c1004e00 00000001 00000001 c109729a c0b94234 c1001f2c
Jun  3 15:50:43 raspberrypi kernel: [19177.333427] 1f00: c1000000 c1001f20 00000000 c0209ab8 40000013 ffffffff
Jun  3 15:50:43 raspberrypi kernel: [19177.333447] [<c02019bc>] (__irq_svc) from [<c0209ab8>] (arch_cpu_idle+0x34/0x4c)
Jun  3 15:50:43 raspberrypi kernel: [19177.333471] [<c0209ab8>] (arch_cpu_idle) from [<c09dbcf8>] (default_idle_call+0x40/0x48)
Jun  3 15:50:43 raspberrypi kernel: [19177.333493] [<c09dbcf8>] (default_idle_call) from [<c02549c0>] (do_idle+0x124/0x164)
Jun  3 15:50:43 raspberrypi kernel: [19177.333509] [<c02549c0>] (do_idle) from [<c0254cc0>] (cpu_startup_entry+0x28/0x2c)
Jun  3 15:50:43 raspberrypi kernel: [19177.333523] [<c0254cc0>] (cpu_startup_entry) from [<c09d537c>] (rest_init+0xb8/0xbc)
Jun  3 15:50:43 raspberrypi kernel: [19177.333541] [<c09d537c>] (rest_init) from [<c0e01000>] (start_kernel+0x4b0/0x4e0)
Jun  3 15:50:43 raspberrypi kernel: [19177.333559] ---[ end trace 3831669ada16d78b ]---

@ablacksheep
Copy link
Contributor

Moving the device in the usb3 back to usb3.port01 to check if device in usb2 still crashes, and observed pattern of not crashing in usb3.port01 was just a coincidence.

It was just a was a coincidence. It keeps crashing. :)

@shaohaoz
Copy link

I've had similar issue on RPI3B+. Any updates?

@pelwell
Copy link
Contributor

pelwell commented Jun 11, 2020

What do you mean by "similar issue"? What external hardware are you using, and what does the kernel log show?

@ablacksheep
Copy link
Contributor

Linux raspberrypi 5.4.51-v7l+ #1327 SMP Thu Jul 23 11:04:39 BST 2020 armv7l GNU/Linux


[   11.591708] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   11.642833] ax88179_178a 1-1.4:1.0 enx00249b2c7dfb: ax88179 - Link status is: 1
[   11.649467] IPv6: ADDRCONF(NETDEV_CHANGE): enx00249b2c7dfb: link becomes ready
[   11.675800] ax88179_178a 2-1:1.0 enx00249b5b748d: ax88179 - Link status is: 1
[   11.682343] IPv6: ADDRCONF(NETDEV_CHANGE): enx00249b5b748d: link becomes ready

[15255.045089] ax88179_178a 2-1:1.0 enx00249b5b748d: ax88179 - Link status is: 0
[19342.353100] ------------[ cut here ]------------
[19342.353148] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x310/0x314
[19342.353170] NETDEV WATCHDOG: enx00249b2c7dfb (ax88179_178a): transmit queue 0 timed out
[19342.353187] Modules linked in: xt_multiport bnep hci_uart btbcm bluetooth ecdh_generic ecc 8021q garp stp llc brcmfmac brcmutil ax88179_178a sha256_generic libsha256 vc4 cfg80211 cec rfkill v3d gpu_sched drm_kms_helper raspberrypi_hwmon drm bcm2835_codec(C) bcm2835_isp(C) v4l2_mem2mem bcm2835_v4l2(C) videobuf2_dma_contig drm_panel_orientation_quirks bcm2835_mmal_vchiq(C) videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_soc_core videobuf2_common snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm videodev mc snd_timer vc_sm_cma(C) snd syscopyarea sysfillrect rpivid_mem sysimgblt fb_sys_fops ip6t_REJECT nf_reject_ipv6 uio_pdrv_genirq uio xt_hl ip6_tables ip6t_rt ipt_REJECT nf_reject_ipv4 nft_limit xt_limit xt_addrtype xt_tcpudp xt_conntrack nft_compat nft_counter nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack nf_defrag_ipv4 nf_tables nfnetlink ip_tables x_tables ipv6 nf_defrag_ipv6
[19342.353518] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.4.51-v7l+ #1327
[19342.353526] Hardware name: BCM2711
[19342.353534] Backtrace: 
[19342.353565] [<c020d46c>] (dump_backtrace) from [<c020d768>] (show_stack+0x20/0x24)
[19342.353579]  r6:c1200000 r5:00000000 r4:c129c8f8 r3:3b427126
[19342.353600] [<c020d748>] (show_stack) from [<c0a39a24>] (dump_stack+0xe0/0x124)
[19342.353618] [<c0a39944>] (dump_stack) from [<c0221c50>] (__warn+0xec/0x104)
[19342.353631]  r8:000001c0 r7:00000009 r6:c0e375c8 r5:00000000 r4:c1201cd4 r3:3b427126
[19342.353648] [<c0221b64>] (__warn) from [<c0221cec>] (warn_slowpath_fmt+0x84/0xc0)
[19342.353660]  r9:c0e375c8 r8:000001c0 r7:c09476cc r6:00000009 r5:c0e375e0 r4:c1204f88
[19342.353675] [<c0221c6c>] (warn_slowpath_fmt) from [<c09476cc>] (dev_watchdog+0x310/0x314)
[19342.353687]  r9:001d0e50 r8:d6aa3800 r7:00000000 r6:c1203d00 r5:d6aa3aa8 r4:00000000
[19342.353704] [<c09473bc>] (dev_watchdog) from [<c02a05bc>] (call_timer_fn+0x40/0x180)
[19342.353716]  r8:c09473bc r7:00000100 r6:c1200000 r5:d6aa3aa8 r4:eff0c440
[19342.353732] [<c02a057c>] (call_timer_fn) from [<c02a170c>] (run_timer_softirq+0x288/0x654)
[19342.353744]  r9:00000000 r8:d6aa3aa8 r7:c1200000 r6:001d0e50 r5:c1201dac r4:eff0c440
[19342.353758] [<c02a1484>] (run_timer_softirq) from [<c020249c>] (__do_softirq+0x1a4/0x418)
[19342.353770]  r10:00000004 r9:00000282 r8:ef848800 r7:00000100 r6:c1200000 r5:00000001
[19342.353780]  r4:c1203084
[19342.353799] [<c02022f8>] (__do_softirq) from [<c0227d54>] (irq_exit+0x100/0x110)
[19342.353812]  r10:0000014e r9:c1200000 r8:ef848800 r7:00000001 r6:00000000 r5:00000000
[19342.353822]  r4:c10a82e4
[19342.353840] [<c0227c54>] (irq_exit) from [<c028302c>] (__handle_domain_irq+0x70/0xc4)
[19342.353862] [<c0282fbc>] (__handle_domain_irq) from [<c02022b8>] (gic_handle_irq+0x4c/0x88)
[19342.353874]  r8:f0815000 r7:f0814000 r6:c1201ed0 r5:f081400c r4:c1205a14 r3:c1201ed0
[19342.353888] [<c020226c>] (gic_handle_irq) from [<c0201a3c>] (__irq_svc+0x5c/0x7c)
[19342.353898] Exception stack(0xc1201ed0 to 0xc1201f18)
[19342.353909] 1ec0:                                     c0209b94 00000000 60000093 c021c160
[19342.353921] 1ee0: c1204fb4 c1200000 c1204ffc 00000001 c12a32ed c106ca48 0000014e c1201f2c
[19342.353936] 1f00: c12053ac c1201f20 00000000 c0209b98 60000013 ffffffff
[19342.353951]  r8:c12a32ed r7:c1201f04 r6:ffffffff r5:60000013 r4:c0209b98 r3:3b427126
[19342.353973] [<c0209b64>] (arch_cpu_idle) from [<c0a5aa4c>] (default_idle_call+0x34/0x48)
[19342.353989] [<c0a5aa18>] (default_idle_call) from [<c0255b00>] (do_idle+0xec/0x170)
[19342.354002] [<c0255a14>] (do_idle) from [<c0255e60>] (cpu_startup_entry+0x28/0x2c)
[19342.354015]  r8:ffffffff r7:c12b5680 r6:c106ca48 r5:c12b56d8 r4:000000ce r3:c1201f48
[19342.354031] [<c0255e38>] (cpu_startup_entry) from [<c0a53e54>] (rest_init+0xbc/0xc0)
[19342.354047] [<c0a53d98>] (rest_init) from [<c1000bec>] (arch_call_rest_init+0x18/0x1c)
[19342.354056]  r4:c1204f88 r3:c109d270
[19342.354068] [<c1000bd4>] (arch_call_rest_init) from [<c10010d4>] (start_kernel+0x4e4/0x518)
[19342.354081] [<c1000bf0>] (start_kernel) from [<00000000>] (0x0)
[19342.354097] ---[ end trace e35d0439ee9b8b60 ]---

@paralin
Copy link

paralin commented Aug 2, 2020

I have the same handle_IPI / cpu_startup_entry failure on linux kernel 5.4.51 3311531 and firmware bd816dbac723e04f8f8b06bb2e16d767cda7692c

Config (against a pi4)

dtparam=i2c_arm=on
dtparam=spi=on
dtparam=audio=on

[pi4]
dtoverlay=vc4-kms-v3d-pi4
max_framebuffers=2

[all]
enable_uart=1
disable_overscan=1
hdmi_drive=2
gpu_mem=128


@syphr42
Copy link

syphr42 commented Aug 29, 2020

I've got a similar issue on Raspberry Pi 3B+. On Stretch everything is fine. I installed fresh with the same config on Buster (using Ansible for configuration) and it seemed like the network would randomly go down. I managed to find a way to reproduce it by running sudo apt-get update where apt calls go through a proxy located on the local network. This causes the stack trace every time. If I comment out the proxy, everything is fine with the apt update.

Other scenarios cause the same stack trace, but I have not been able to reproduce them consistently. While using tcpdump over SSH this error happens sometimes. Sometimes while tailing a log file over SSH it happens. Sometimes an apt update on another node on the network causes it. Other than the scenario above, nothing seems to consistently cause it.

The PI is setup as a router with the onboard NIC plus a TrendNET USB ethernet adapter. The onboard is on the LAN side and the USB ethernet adapter is on the WAN side. 802.1Q VLAN tagging on the LAN side with virtual interfaces as well.

$ uname -r
5.4.51-v7+
kernel: [ 4777.047109] ------------[ cut here ]------------
kernel: [ 4777.047169] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x310/0x314
kernel: [ 4777.047184] NETDEV WATCHDOG: wan0 (ax88179_178a): transmit queue 0 timed out
kernel: [ 4777.047196] Modules linked in: cmac bnep hci_uart btbcm bluetooth ecdh_generic ecc nf_log_ipv6 ip6_tables nft_chain_nat xt_nat xt_MASQUERADE nf_nat xt_HL nf_log_ipv4 nf_log_common nft_limit nft_counter xt_LOG xt_limit xt_mac xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv4 nft_compat nf_tables nfnetlink ax88179_178a evdev brcmfmac brcmutil raspberrypi_hwmon sha256_generic libsha256 cfg80211 rfkill bcm2835_codec(C) bcm2835_isp(C) bcm2835_v4l2(C) v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_bcm2835(C) videobuf2_common vc_sm_cma(C) snd_pcm snd_timer videodev snd mc 8021q garp stp llc uio_pdrv_genirq uio fixed ip_tables x_tables ipv6 nf_defrag_ipv6
kernel: [ 4777.047470] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.4.51-v7+ #1333
kernel: [ 4777.047475] Hardware name: BCM2835
kernel: [ 4777.047479] Backtrace: 
kernel: [ 4777.047498] [<8010d480>] (dump_backtrace) from [<8010d778>] (show_stack+0x20/0x24)
kernel: [ 4777.047508]  r6:80d00000 r5:00000000 r4:80d95cb8 r3:96b3c785
kernel: [ 4777.047522] [<8010d758>] (show_stack) from [<808cdbe4>] (dump_stack+0xe0/0x124)
kernel: [ 4777.047536] [<808cdb04>] (dump_stack) from [<8012002c>] (__warn+0xec/0x104)
kernel: [ 4777.047548]  r8:000001c0 r7:00000009 r6:80b0f6a4 r5:00000000 r4:80d01cec r3:96b3c785
kernel: [ 4777.047560] [<8011ff40>] (__warn) from [<801200c8>] (warn_slowpath_fmt+0x84/0xc0)
kernel: [ 4777.047572]  r9:80b0f6a4 r8:000001c0 r7:807db124 r6:00000009 r5:80b0f6bc r4:80d04f48
kernel: [ 4777.047587] [<80120048>] (warn_slowpath_fmt) from [<807db124>] (dev_watchdog+0x310/0x314)
kernel: [ 4777.047598]  r9:0006d4d8 r8:b3eb8800 r7:00000000 r6:80d03d00 r5:b3eb8aa8 r4:00000000
kernel: [ 4777.047615] [<807dae14>] (dev_watchdog) from [<8019be6c>] (call_timer_fn+0x40/0x180)
kernel: [ 4777.047626]  r8:807dae14 r7:00000100 r6:80d00000 r5:b3eb8aa8 r4:b6b25440
kernel: [ 4777.047639] [<8019be2c>] (call_timer_fn) from [<8019cfbc>] (run_timer_softirq+0x288/0x654)
kernel: [ 4777.047650]  r9:00000000 r8:b3eb8aa8 r7:80d00000 r6:0006d4d8 r5:80d01dc4 r4:b6b25440
kernel: [ 4777.047666] [<8019cd34>] (run_timer_softirq) from [<8010249c>] (__do_softirq+0x1a4/0x418)
kernel: [ 4777.047678]  r10:00000004 r9:00000082 r8:b645e000 r7:00000100 r6:80d00000 r5:00000001
kernel: [ 4777.047682]  r4:80d03084
kernel: [ 4777.047697] [<801022f8>] (__do_softirq) from [<80126130>] (irq_exit+0x100/0x110)
kernel: [ 4777.047708]  r10:b73ff900 r9:80d00000 r8:b645e000 r7:00000001 r6:00000000 r5:00000000
kernel: [ 4777.047713]  r4:80ca42e4
kernel: [ 4777.047726] [<80126030>] (irq_exit) from [<80181200>] (__handle_domain_irq+0x70/0xc4)
kernel: [ 4777.047738] [<80181190>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
kernel: [ 4777.047750]  r8:80d9b18d r7:80d01f0c r6:ffffffff r5:60000013 r4:00000000 r3:80d01ed8
kernel: [ 4777.047761] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
kernel: [ 4777.047775] Exception stack(0x80d01ed8 to 0x80d01f20)
kernel: [ 4777.047785] 1ec0:                                                       80109b94 00000000
kernel: [ 4777.047796] 1ee0: 60000093 8011a520 80d04f68 80d00000 80d04fb0 00000001 80d9b18d 80c69a40
kernel: [ 4777.047807] 1f00: b73ff900 80d01f34 80d0535c 80d01f28 00000000 80109b98 60000013 ffffffff
kernel: [ 4777.047813]  r4:80109b98 r3:96b3c785
kernel: [ 4777.047830] [<80109b64>] (arch_cpu_idle) from [<808eea44>] (default_idle_call+0x34/0x48)
kernel: [ 4777.047843] [<808eea10>] (default_idle_call) from [<80153bc0>] (do_idle+0xec/0x170)
kernel: [ 4777.047854] [<80153ad4>] (do_idle) from [<80153f20>] (cpu_startup_entry+0x28/0x2c)
kernel: [ 4777.047865]  r8:00000000 r7:80c69a40 r6:80dac5c0 r5:80dac618 r4:000000ce r3:80d01f50
kernel: [ 4777.047877] [<80153ef8>] (cpu_startup_entry) from [<808e7e68>] (rest_init+0xbc/0xc0)
kernel: [ 4777.047890] [<808e7dac>] (rest_init) from [<80c00bd8>] (arch_call_rest_init+0x18/0x1c)
kernel: [ 4777.047897]  r4:80d04f48 r3:80c99260
kernel: [ 4777.047907] [<80c00bc0>] (arch_call_rest_init) from [<80c010a8>] (start_kernel+0x4cc/0x500)
kernel: [ 4777.047917] [<80c00bdc>] (start_kernel) from [<00000000>] (0x0)
kernel: [ 4777.047953] ---[ end trace 9a5a0325809920e6 ]---

@syphr42
Copy link

syphr42 commented Aug 29, 2020

As an additional data point, I have an older 10/100 USB ethernet adapter with a Realtek chip and the same crash occurs.

Bus 001 Device 004: ID 0bda:8150 Realtek Semiconductor Corp. RTL8150 Fast Ethernet Adapter
kernel: [  336.076919] ------------[ cut here ]------------
kernel: [  336.076977] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x310/0x314
kernel: [  336.077002] NETDEV WATCHDOG: lan0 (lan78xx): transmit queue 0 timed out
kernel: [  336.077015] Modules linked in: cmac bnep hci_uart btbcm bluetooth ecdh_generic ecc nf_log_ipv6 ip6_tables nft_chain_nat xt_nat xt_MASQUERADE nf_nat xt_HL nf_log_ipv4 nf_log_common nft_limit nft_counter xt_LOG xt_limit xt_mac xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv4 nft_compat nf_tables nfnetlink evdev rtl8150 brcmfmac brcmutil sha256_generic libsha256 cfg80211 raspberrypi_hwmon rfkill bcm2835_codec(C) bcm2835_v4l2(C) bcm2835_isp(C) v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig snd_bcm2835(C) videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_pcm videobuf2_common snd_timer videodev snd mc vc_sm_cma(C) 8021q garp stp llc fixed uio_pdrv_genirq uio ip_tables x_tables ipv6 nf_defrag_ipv6
kernel: [  336.077310] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.4.51-v7+ #1333
kernel: [  336.077315] Hardware name: BCM2835
kernel: [  336.077319] Backtrace: 
kernel: [  336.077341] [<8010d480>] (dump_backtrace) from [<8010d778>] (show_stack+0x20/0x24)
kernel: [  336.077352]  r6:80d00000 r5:00000000 r4:80d95cb8 r3:96b3c785
kernel: [  336.077367] [<8010d758>] (show_stack) from [<808cdbe4>] (dump_stack+0xe0/0x124)
kernel: [  336.077381] [<808cdb04>] (dump_stack) from [<8012002c>] (__warn+0xec/0x104)
kernel: [  336.077393]  r8:000001c0 r7:00000009 r6:80b0f6a4 r5:00000000 r4:80d01cec r3:96b3c785
kernel: [  336.077404] [<8011ff40>] (__warn) from [<801200c8>] (warn_slowpath_fmt+0x84/0xc0)
kernel: [  336.077415]  r9:80b0f6a4 r8:000001c0 r7:807db124 r6:00000009 r5:80b0f6bc r4:80d04f48
kernel: [  336.077432] [<80120048>] (warn_slowpath_fmt) from [<807db124>] (dev_watchdog+0x310/0x314)
kernel: [  336.077452]  r9:00000e18 r8:b5ea2000 r7:00000000 r6:80d03d00 r5:b5ea22a8 r4:00000000
kernel: [  336.077471] [<807dae14>] (dev_watchdog) from [<8019be6c>] (call_timer_fn+0x40/0x180)
kernel: [  336.077486]  r8:807dae14 r7:00000100 r6:80d00000 r5:b5ea22a8 r4:b6b25440
kernel: [  336.077504] [<8019be2c>] (call_timer_fn) from [<8019cfbc>] (run_timer_softirq+0x288/0x654)
kernel: [  336.077518]  r9:00000000 r8:b5ea22a8 r7:80d00000 r6:00000e18 r5:80d01dc4 r4:b6b25440
kernel: [  336.077535] [<8019cd34>] (run_timer_softirq) from [<8010249c>] (__do_softirq+0x1a4/0x418)
kernel: [  336.077551]  r10:00000004 r9:00000082 r8:b645e000 r7:00000100 r6:80d00000 r5:00000001
kernel: [  336.077562]  r4:80d03084
kernel: [  336.077580] [<801022f8>] (__do_softirq) from [<80126130>] (irq_exit+0x100/0x110)
kernel: [  336.077596]  r10:b73ff900 r9:80d00000 r8:b645e000 r7:00000001 r6:00000000 r5:00000000
kernel: [  336.077605]  r4:80ca42e4
kernel: [  336.077624] [<80126030>] (irq_exit) from [<80181200>] (__handle_domain_irq+0x70/0xc4)
kernel: [  336.077638] [<80181190>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
kernel: [  336.077652]  r8:80d9b18d r7:80d01f0c r6:ffffffff r5:60000013 r4:00000000 r3:80d01ed8
kernel: [  336.077664] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
kernel: [  336.077670] Exception stack(0x80d01ed8 to 0x80d01f20)
kernel: [  336.077679] 1ec0:                                                       80109b94 00000000
kernel: [  336.077690] 1ee0: 60000093 8011a520 80d04f68 80d00000 80d04fb0 00000001 80d9b18d 80c69a40
kernel: [  336.077701] 1f00: b73ff900 80d01f34 80d0535c 80d01f28 00000000 80109b98 60000013 ffffffff
kernel: [  336.077708]  r4:80109b98 r3:96b3c785
kernel: [  336.077723] [<80109b64>] (arch_cpu_idle) from [<808eea44>] (default_idle_call+0x34/0x48)
kernel: [  336.077737] [<808eea10>] (default_idle_call) from [<80153bc0>] (do_idle+0xec/0x170)
kernel: [  336.077747] [<80153ad4>] (do_idle) from [<80153f20>] (cpu_startup_entry+0x28/0x2c)
kernel: [  336.077759]  r8:00000000 r7:80c69a40 r6:80dac5c0 r5:80dac618 r4:000000ce r3:80d01f50
kernel: [  336.077771] [<80153ef8>] (cpu_startup_entry) from [<808e7e68>] (rest_init+0xbc/0xc0)
kernel: [  336.077786] [<808e7dac>] (rest_init) from [<80c00bd8>] (arch_call_rest_init+0x18/0x1c)
kernel: [  336.077792]  r4:80d04f48 r3:80c99260
kernel: [  336.077803] [<80c00bc0>] (arch_call_rest_init) from [<80c010a8>] (start_kernel+0x4cc/0x500)
kernel: [  336.077815] [<80c00bdc>] (start_kernel) from [<00000000>] (0x0)
kernel: [  336.077827] ---[ end trace 24a6d02eeb0175e0 ]---

dwrobel added a commit to dwrobel/kernel that referenced this issue Sep 14, 2020
Fixes the kernel panic[1] which occurs very frequently when
one or two AX88179 based Ethernet cards are attached to Raspberry.

Original Vendor driver seems to fix the issue.

[1] raspberrypi/linux#3401

Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
@dwrobel
Copy link

dwrobel commented Sep 14, 2020

For the record on the latest 5.4.64 on 64-bit version this was also much frequently reproducible in comparison to 4.19.y 32-bit version (with the same headless configuration as I mentioned here):

kernel: ------------[ cut here ]------------
kernel: NETDEV WATCHDOG: enp1s0u1u1 (ax88179_178a): transmit queue 0 timed out
kernel: WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x394/0x3a0
kernel: Modules linked in: tun nft_nat nft_masq nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 n
ft_reject nft_ct nf_tables_set nft_chain_nat ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables nfnetlink 
ip6table_filter ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod brcmfmac brcmutil cfg80211 bcm2835_codec(C) bcm2835_isp(C) vc4 v4l2_mem2mem bcm2835_v4l2(C) videobuf2_dma_contig cec bcm2835_mmal_vchiq(C) ax88179_178a rfkil
l drm_kms_helper videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common snd_soc_core videodev snd_compress snd_pcm_dmaengine snd_bcm2835(C) v3d mc dwc2 vc_sm_cma(C) snd_pcm snd_timer gpu_sched raspberrypi_hwmon snd syscopyarea
 rpivid_mem sysfillrect sysimgblt fb_sys_fops uio_pdrv_genirq uio gpio_fan
kernel:  sch_fq_codel zram zsmalloc drm fuse drm_panel_orientation_quirks
kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.4.64-1.rpi4.fc32.aarch64 #1
kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT)
kernel: pstate: 00000005 (nzcv daif -PAN -UAO)
kernel: pc : dev_watchdog+0x394/0x3a0
kernel: lr : dev_watchdog+0x394/0x3a0
kernel: sp : ffffffc010003d00
kernel: x29: ffffffc010003d00 x28: ffffff80f2631a80 
kernel: x27: 0000000000000004 x26: ffffff80ee1df480 
kernel: x25: 0000000000000140 x24: 00000000ffffffff 
kernel: x23: ffffff80ee1df45c x22: 0000000000000000 
kernel: x21: ffffffc010f56000 x20: ffffff80ee1df000 
kernel: x19: 0000000000000000 x18: 0000000000000030 
kernel: x17: 0000000000000000 x16: 0000000000000000 
kernel: x15: ffffffc010f60478 x14: ffffffffffffffff 
kernel: x13: ffffffc090003a07 x12: ffffffc010003a0f 
kernel: x11: 7274203a29613837 x10: 315f393731383878 
kernel: x9 : 0000000000000000 x8 : 3020657565757120 
kernel: x7 : ffffffc0106566a0 x6 : ffffffc010f59000 
kernel: x5 : 0000000000000000 x4 : ffffffc010ae44d0 
kernel: x3 : fffffffffffffffe x2 : 0000000000000004 
kernel: x1 : a6a675dc55bf7e00 x0 : 0000000000000000 
kernel: Call trace:
kernel:  dev_watchdog+0x394/0x3a0
kernel:  call_timer_fn+0x40/0x1e4
kernel:  expire_timers+0xc8/0x18c
kernel:  run_timer_softirq+0xdc/0x210
kernel:  __do_softirq+0x1a0/0x488
kernel:  irq_exit+0x118/0x120
kernel:  __handle_domain_irq+0xb4/0x130
kernel:  gic_handle_irq+0x5c/0xb0
kernel:  el1_irq+0xbc/0x180
kernel:  arch_cpu_idle+0x38/0x220
kernel:  do_idle+0x234/0x2b0
kernel:  cpu_startup_entry+0x2c/0x90
kernel:  rest_init+0xdc/0xe8
kernel:  arch_call_rest_init+0x18/0x20
kernel:  start_kernel+0x490/0x4c4
kernel: ---[ end trace 52602d8ce620bd83 ]---

So I switched to use vendor driver v1.20.0 with dtoverlay=dwc2 and the following cmdline.txt:

coherent_pool=16M dwc_otg.lpm_enable=0 earlyprintk console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 ro rootfstype=ext4 fsck.repair=yes rootwait libahci.ignore_sss=1 raid=noautodetect nortc selinux=0

It's too early to say whether it will be fully stable but at least I wasn't able to reproduce the issue so far.

@tomasmark79
Copy link

tomasmark79 commented May 10, 2021

Syslog from RPI4B

May 10 10:01:53 raspiwall kernel: [43686.815046] ------------[ cut here ]------------
May 10 10:01:53 raspiwall kernel: [43686.815100] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:443 dev_watchdog+0x308/0x30c
May 10 10:01:53 raspiwall kernel: [43686.815122] NETDEV WATCHDOG: wan0 (ax88179_178a): transmit queue 0 timed out
May 10 10:01:53 raspiwall kernel: [43686.815140] Modules linked in: cmac bnep hci_uart btbcm bluetooth ecdh_generic ecc nft_chain_nat xt_MASQUERADE xt_nat nf_nat nf_log_ipv4 nf_log_common nft_limit 8021q garp stp llc nft_counter xt_mult$
May 10 10:01:53 raspiwall kernel: [43686.818691] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G C 5.10.17-v7l+ #1414
May 10 10:01:53 raspiwall kernel: [43686.819010] Hardware name: BCM2711
May 10 10:01:53 raspiwall kernel: [43686.819019] Backtrace:
May 10 10:01:53 raspiwall kernel: [43686.819054] [] (dump_backtrace) from [] (show_stack+0x20/0x24)
May 10 10:01:53 raspiwall kernel: [43686.819394] r7:ffffffff r6:00000000 r5:60000113 r4:c12e69fc
May 10 10:01:53 raspiwall kernel: [43686.819410] [] (show_stack) from [] (dump_stack+0xcc/0xf8)
May 10 10:01:53 raspiwall kernel: [43686.819426] [] (dump_stack) from [] (__warn+0xfc/0x114)
May 10 10:01:53 raspiwall kernel: [43686.819439] r10:c133c8b8 r9:00000009 r8:c0a46634 r7:000001bb r6:00000009 r5:c0a46634
May 10 10:01:53 raspiwall kernel: [43686.819448] r4:c0ea7698 r3:c1205094
May 10 10:01:53 raspiwall kernel: [43686.819464] [] (__warn) from [] (warn_slowpath_fmt+0xa4/0xd8)
May 10 10:01:53 raspiwall kernel: [43686.819474] r7:000001bb r6:c0ea7698 r5:c1205048 r4:c0ea765c
May 10 10:01:53 raspiwall kernel: [43686.819490] [] (warn_slowpath_fmt) from [] (dev_watchdog+0x308/0x30c)
May 10 10:01:53 raspiwall kernel: [43686.819502] r9:eff0b540 r8:c3264000 r7:c1203d00 r6:c32a1d00 r5:c32642a8 r4:00000000
May 10 10:01:53 raspiwall kernel: [43686.819517] [] (dev_watchdog) from [] (call_timer_fn+0x40/0x1bc)
May 10 10:01:53 raspiwall kernel: [43686.819529] r8:c1201d9c r7:004233b0 r6:c0a4632c r5:00000100 r4:c32642a8
May 10 10:01:53 raspiwall kernel: [43686.819543] [] (call_timer_fn) from [] (run_timer_softirq+0x5b0/0x698)
May 10 10:01:53 raspiwall kernel: [43686.819555] r8:c1201d9c r7:00000000 r6:c32642a8 r5:004233b0 r4:00000000
May 10 10:01:53 raspiwall kernel: [43686.819571] [] (run_timer_softirq) from [] (__do_softirq+0x198/0x49c)
May 10 10:01:53 raspiwall kernel: [43686.819583] r10:00000082 r9:ffffe000 r8:c1810800 r7:00000100 r6:00000001 r5:00000002
May 10 10:01:53 raspiwall kernel: [43686.819591] r4:c1203084
May 10 10:01:53 raspiwall kernel: [43686.819606] [] (__do_softirq) from [] (irq_exit+0xd0/0xf8)
May 10 10:01:53 raspiwall kernel: [43686.819618] r10:c0e1edfc r9:c1200000 r8:c1810800 r7:00000001 r6:00000000 r5:00000000
May 10 10:01:53 raspiwall kernel: [43686.819626] r4:ffffe000
May 10 10:01:53 raspiwall kernel: [43686.819643] [] (irq_exit) from [] (__handle_domain_irq+0x70/0xc4)
May 10 10:01:53 raspiwall kernel: [43686.819653] r5:00000000 r4:c1094d10
May 10 10:01:53 raspiwall kernel: [43686.819667] [] (__handle_domain_irq) from [] (gic_handle_irq+0x90/0xa4)
May 10 10:01:53 raspiwall kernel: [43686.819679] r9:c1200000 r8:c1094d1c r7:c1201ec8 r6:f081400c r5:f0814000 r4:c1205b3c
May 10 10:01:53 raspiwall kernel: [43686.819692] [] (gic_handle_irq) from [] (__irq_svc+0x5c/0x7c)
May 10 10:01:53 raspiwall kernel: [43686.819701] Exception stack(0xc1201ec8 to 0xc1201f10)
May 10 10:01:53 raspiwall kernel: [43686.819714] 1ec0: 00000000 0400fe78 eff132c4 c021ac00 ffffe000 c120509c
May 10 10:01:53 raspiwall kernel: [43686.820050] 1ee0: c12050e4 00000001 00000001 c133ba3f c0e1edfc c1201f24 c1201f28 c1201f18
May 10 10:01:53 raspiwall kernel: [43686.820060] 1f00: c02088c0 c02088c4 60000013 ffffffff
May 10 10:01:53 raspiwall kernel: [43686.820072] r9:c1200000 r8:00000001 r7:c1201efc r6:ffffffff r5:60000013 r4:c02088c4
May 10 10:01:53 raspiwall kernel: [43686.820408] [] (arch_cpu_idle) from [] (default_idle_call+0x4c/0x118)
May 10 10:01:53 raspiwall kernel: [43686.820423] [] (default_idle_call) from [] (do_idle+0x118/0x168)
May 10 10:01:53 raspiwall kernel: [43686.820437] [] (do_idle) from [] (cpu_startup_entry+0x28/0x30)
May 10 10:01:53 raspiwall kernel: [43686.820449] r10:00000197 r9:c1053a60 r8:ffffffff r7:c1053a60 r6:c1205040 r5:c1205048
May 10 10:01:53 raspiwall kernel: [43686.820459] r4:000000d8 r3:c108a294
May 10 10:01:53 raspiwall kernel: [43686.820474] [] (cpu_startup_entry) from [] (rest_init+0xbc/0xc4)
May 10 10:01:53 raspiwall kernel: [43686.820489] [] (rest_init) from [] (arch_call_rest_init+0x18/0x1c)
May 10 10:01:53 raspiwall kernel: [43686.820499] r5:c1205048 r4:c1354068
May 10 10:01:53 raspiwall kernel: [43686.820511] [] (arch_call_rest_init) from [] (start_kernel+0x568/0x59c)
May 10 10:01:53 raspiwall kernel: [43686.820524] [] (start_kernel) from [<00000000>] (0x0)

Update:

I Updated the driver to version 1.20 and at until now no issue. We will see if this helped, or not.

@ProactiveServices
Copy link

Seeing the same behaviour on a Pi 4b 1.2, Raspbian OS 10.10, kernel 5.10.17. Using the onboard wired Ethernet and official PSU. Headless device running from a small SSD in a USB3 enclosure plugged into a USB3 port. It's currently pinging about a dozen hosts every 15s and logging (via telegraf to influxdb) and doing nothing else. The hangs are happening multiple times a day, usually only for a few minutes but have seen one occasion of about half an hour.
stacktrace.txt

@sulewicz
Copy link

Same thing on Raspberry Pi 4, Kernel 5.10.103-v7l+.

Oct 17 20:04:27 customecho kernel: [25878.571042] rcu: INFO: rcu_sched self-detected stall on CPU
Oct 17 20:04:27 customecho kernel: [25878.571053] rcu: 	0-....: (8403 ticks this GP) idle=da6/1/0x40000002 softirq=340855/340855 fqs=2873
Oct 17 20:04:27 customecho kernel: [25878.571058] 	(t=8404 jiffies g=658509 q=5360)
Oct 17 20:04:27 customecho kernel: [25878.571063] NMI backtrace for cpu 0
Oct 17 20:04:27 customecho kernel: [25878.571070] CPU: 0 PID: 919 Comm: CompositorTileW Tainted: G         C        5.10.103-v7l+ #1529
Oct 17 20:04:27 customecho kernel: [25878.571074] Hardware name: BCM2711
Oct 17 20:04:27 customecho kernel: [25878.571078] Backtrace:
Oct 17 20:04:27 customecho kernel: [25878.571097] [<c0b875f0>] (dump_backtrace) from [<c0b87984>] (show_stack+0x20/0x24)
Oct 17 20:04:27 customecho kernel: [25878.571103]  r7:ffffffff r6:00000000 r5:68000193 r4:c12e6fd8
Oct 17 20:04:27 customecho kernel: [25878.571110] [<c0b87964>] (show_stack) from [<c0b8bd74>] (dump_stack+0xcc/0xf8)
Oct 17 20:04:27 customecho kernel: [25878.571119] [<c0b8bca8>] (dump_stack) from [<c07782cc>] (nmi_cpu_backtrace+0xc8/0xf4)
Oct 17 20:04:27 customecho kernel: [25878.571124]  r10:88000193 r9:c0e284a4 r8:c0c02060 r7:c0c02064 r6:00000000 r5:00000000
Oct 17 20:04:27 customecho kernel: [25878.571129]  r4:00000000 r3:6e4df213
Oct 17 20:04:27 customecho kernel: [25878.571136] [<c0778204>] (nmi_cpu_backtrace) from [<c0778420>] (nmi_trigger_cpumask_backtrace+0x128/0x140)
Oct 17 20:04:27 customecho kernel: [25878.571140]  r5:c1205b24 r4:c020ed30
Oct 17 20:04:27 customecho kernel: [25878.571148] [<c07782f8>] (nmi_trigger_cpumask_backtrace) from [<c020fc5c>] (arch_trigger_cpumask_backtrace+0x20/0x24)
Oct 17 20:04:27 customecho kernel: [25878.571153]  r7:c1276640 r6:c12050a4 r5:c1205148 r4:00000000
Oct 17 20:04:27 customecho kernel: [25878.571160] [<c020fc3c>] (arch_trigger_cpumask_backtrace) from [<c0b896bc>] (rcu_dump_cpu_stacks+0x10c/0x144)
Oct 17 20:04:27 customecho kernel: [25878.571168] [<c0b895b0>] (rcu_dump_cpu_stacks) from [<c029fd6c>] (rcu_sched_clock_irq+0x7c0/0xa38)
Oct 17 20:04:27 customecho kernel: [25878.571174]  r10:c1203d00 r9:2ee5a000 r8:00000000 r7:c10982c0 r6:c10982c0 r5:efef22c0
Oct 17 20:04:27 customecho kernel: [25878.571178]  r4:c1276640
Oct 17 20:04:27 customecho kernel: [25878.571186] [<c029f5ac>] (rcu_sched_clock_irq) from [<c02addd4>] (update_process_times+0x70/0x9c)
Oct 17 20:04:27 customecho kernel: [25878.571192]  r10:c02c1cb8 r9:efeeb640 r8:ffffe000 r7:00001789 r6:f14d7b9d r5:00000000
Oct 17 20:04:27 customecho kernel: [25878.571195]  r4:ffffe000
Oct 17 20:04:27 customecho kernel: [25878.571203] [<c02add64>] (update_process_times) from [<c02c15a4>] (tick_sched_handle+0x64/0x70)
Oct 17 20:04:27 customecho kernel: [25878.571208]  r7:00001789 r6:42486d27 r5:c59c7db0 r4:efeeb988
Oct 17 20:04:27 customecho kernel: [25878.571214] [<c02c1540>] (tick_sched_handle) from [<c02c1d14>] (tick_sched_timer+0x5c/0xb8)
Oct 17 20:04:27 customecho kernel: [25878.571221] [<c02c1cb8>] (tick_sched_timer) from [<c02ae92c>] (__hrtimer_run_queues+0xe8/0x384)
Oct 17 20:04:27 customecho kernel: [25878.571225]  r7:efeeb680 r6:c120509c r5:efeeb640 r4:efeeb988
Oct 17 20:04:27 customecho kernel: [25878.571233] [<c02ae844>] (__hrtimer_run_queues) from [<c02af868>] (hrtimer_interrupt+0x130/0x2b8)
Oct 17 20:04:27 customecho kernel: [25878.571239]  r10:efeeb790 r9:efeeb7c0 r8:ffffffff r7:7fffffff r6:00000003 r5:28000193
Oct 17 20:04:27 customecho kernel: [25878.571243]  r4:efeeb640
Oct 17 20:04:27 customecho kernel: [25878.571251] [<c02af738>] (hrtimer_interrupt) from [<c09c0430>] (arch_timer_handler_phys+0x38/0x40)
Oct 17 20:04:27 customecho kernel: [25878.571257]  r10:c1205048 r9:c59c6000 r8:c1818800 r7:0000001b r6:c18e8480 r5:c1205b74
Oct 17 20:04:27 customecho kernel: [25878.571261]  r4:c18e6700
Oct 17 20:04:27 customecho kernel: [25878.571268] [<c09c03f8>] (arch_timer_handler_phys) from [<c028e120>] (handle_percpu_devid_irq+0x8c/0x21c)
Oct 17 20:04:27 customecho kernel: [25878.571275] [<c028e094>] (handle_percpu_devid_irq) from [<c028734c>] (generic_handle_irq+0x44/0x54)
Oct 17 20:04:27 customecho kernel: [25878.571280]  r7:00000001 r6:00000000 r5:00000000 r4:c1096d90
Oct 17 20:04:27 customecho kernel: [25878.571287] [<c0287308>] (generic_handle_irq) from [<c0287aac>] (__handle_domain_irq+0x6c/0xc4)
Oct 17 20:04:27 customecho kernel: [25878.571295] [<c0287a40>] (__handle_domain_irq) from [<c020135c>] (gic_handle_irq+0x90/0xa4)
Oct 17 20:04:27 customecho kernel: [25878.571300]  r9:c59c6000 r8:c1096d9c r7:c59c7db0 r6:f081400c r5:f0814000 r4:c1205b74
Oct 17 20:04:27 customecho kernel: [25878.571306] [<c02012cc>] (gic_handle_irq) from [<c0200abc>] (__irq_svc+0x5c/0x7c)
Oct 17 20:04:27 customecho kernel: [25878.571311] Exception stack(0xc59c7db0 to 0xc59c7df8)
Oct 17 20:04:27 customecho kernel: [25878.571316] 7da0:                                     d8800014 00000000 00000000 0000000a
Oct 17 20:04:27 customecho kernel: [25878.571322] 7dc0: c59c7e90 c59afd68 da70ac58 00000001 ffefefe0 d8800014 c1205048 c59c7e0c
Oct 17 20:04:27 customecho kernel: [25878.571327] 7de0: c59c7e10 c59c7e00 c03f2944 c0b97bec 28000113 ffffffff
Oct 17 20:04:27 customecho kernel: [25878.571332]  r9:c59c6000 r8:ffefefe0 r7:c59c7de4 r6:ffffffff r5:28000113 r4:c0b97bec
Oct 17 20:04:27 customecho kernel: [25878.571341] [<c0b97bac>] (_raw_spin_lock) from [<c03f2944>] (alloc_set_pte+0x2a0/0x32c)
Oct 17 20:04:27 customecho kernel: [25878.571347] [<c03f26a4>] (alloc_set_pte) from [<c03f2a24>] (finish_fault+0x54/0xcc)
Oct 17 20:04:27 customecho kernel: [25878.571353]  r10:c1205048 r9:00000040 r8:00000255 r7:00000200 r6:c59c7e90 r5:00000001
Oct 17 20:04:27 customecho kernel: [25878.571357]  r4:c59c7e90
Oct 17 20:04:27 customecho kernel: [25878.571362] [<c03f29d0>] (finish_fault) from [<c03f37f0>] (handle_mm_fault+0xc00/0x1124)
Oct 17 20:04:27 customecho kernel: [25878.571366]  r5:00000001 r4:ac3fc000
Oct 17 20:04:27 customecho kernel: [25878.571373] [<c03f2bf0>] (handle_mm_fault) from [<c0b97fdc>] (do_page_fault+0x144/0x314)
Oct 17 20:04:27 customecho kernel: [25878.571379]  r10:00000255 r9:00000800 r8:c2302244 r7:c2302200 r6:00000a06 r5:ac3fc000
Oct 17 20:04:27 customecho kernel: [25878.571383]  r4:c59c7fb0
Oct 17 20:04:27 customecho kernel: [25878.571390] [<c0b97e98>] (do_page_fault) from [<c0b9828c>] (do_translation_fault+0xe0/0x10c)
Oct 17 20:04:27 customecho kernel: [25878.571396]  r10:b6fc0968 r9:00000200 r8:00000a06 r7:ac3fc000 r6:c0b981ac r5:00000a06
Oct 17 20:04:27 customecho kernel: [25878.571400]  r4:c120af78
Oct 17 20:04:27 customecho kernel: [25878.571407] [<c0b981ac>] (do_translation_fault) from [<c0214de4>] (do_DataAbort+0x40/0xc0)
Oct 17 20:04:27 customecho kernel: [25878.571412]  r9:00000200 r8:c59c7fb0 r7:ac3fc000 r6:c0b981ac r5:00000a06 r4:c120af78
Oct 17 20:04:27 customecho kernel: [25878.571418] [<c0214da4>] (do_DataAbort) from [<c0200da4>] (__dabt_usr+0x44/0x60)
Oct 17 20:04:27 customecho kernel: [25878.571422] Exception stack(0xc59c7fb0 to 0xc59c7ff8)
Oct 17 20:04:27 customecho kernel: [25878.571427] 7fa0:                                     ac3fc000 00000000 00000200 00000800
Oct 17 20:04:27 customecho kernel: [25878.571433] 7fc0: 00000200 000001fc 00000000 afd51718 afd51718 00000200 b6fc0968 afd515f0
Oct 17 20:04:27 customecho kernel: [25878.571439] 7fe0: 00000100 afd515e8 ac3fc000 0391a620 28000010 ffffffff
Oct 17 20:04:27 customecho kernel: [25878.571444]  r8:30c5387d r7:30c5383d r6:ffffffff r5:28000010 r4:0391a620

@teewhey
Copy link

teewhey commented Nov 28, 2022

Same thing for me on my Pi4. It happens as soon as I plugged in the LAN cable and all networks just unresponsive till I unplug it and use wlan instead.

Linux pipipi 5.15.76-v7l+ #1597 SMP Fri Nov 4 12:14:58 GMT 2022 armv7l GNU/Linux

[Mon Nov 28 17:19:34 2022] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:478 dev_watchdog+0x304/0x308
[Mon Nov 28 17:19:34 2022] NETDEV WATCHDOG: eth0 (bcmgenet): transmit queue 2 timed out
[Mon Nov 28 17:19:34 2022] Modules linked in: xt_nat xt_tcpudp veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_counter xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge rfcomm cmac algif_hash aes_arm_bs crypto_simd cryptd algif_skcipher af_alg bnep hci_uart btbcm bluetooth ecdh_generic ecc cp210x usbserial overlay 8021q garp stp llc vc4 sg snd_soc_hdmi_codec cec brcmfmac drm_kms_helper bcm2835_v4l2(C) bcm2835_codec(C) brcmutil bcm2835_isp(C) snd_soc_core cfg80211 bcm2835_mmal_vchiq(C) rpivid_hevc(C) v4l2_mem2mem videobuf2_dma_contig videobuf2_vmalloc v3d videobuf2_memops snd_compress gpu_sched snd_pcm_dmaengine videobuf2_v4l2 videobuf2_common raspberrypi_hwmon rfkill snd_bcm2835(C) snd_pcm videodev snd_timer i2c_brcmstb vc_sm_cma(C) mc snd syscopyarea sysfillrect sysimgblt uio_pdrv_genirq fb_sys_fops nvmem_rmem uio drm fuse i2c_dev drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[Mon Nov 28 17:19:34 2022] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G         C        5.15.76-v7l+ #1597
[Mon Nov 28 17:19:34 2022] Hardware name: BCM2711
[Mon Nov 28 17:19:34 2022] Backtrace:
[Mon Nov 28 17:19:34 2022] [<c0bd7354>] (dump_backtrace) from [<c0bd75a0>] (show_stack+0x20/0x24)
[Mon Nov 28 17:19:34 2022]  r7:000001de r6:c0e3f708 r5:00000000 r4:60000113
[Mon Nov 28 17:19:34 2022] [<c0bd7580>] (show_stack) from [<c0bdbcb0>] (dump_stack_lvl+0x70/0x94)
[Mon Nov 28 17:19:34 2022] [<c0bdbc40>] (dump_stack_lvl) from [<c0bdbcec>] (dump_stack+0x18/0x1c)
[Mon Nov 28 17:19:34 2022]  r7:000001de r6:00000009 r5:c0aac28c r4:c0ec6d04
[Mon Nov 28 17:19:34 2022] [<c0bdbcd4>] (dump_stack) from [<c02226c0>] (__warn+0xfc/0x114)
[Mon Nov 28 17:19:34 2022] [<c02225c4>] (__warn) from [<c0bd7c94>] (warn_slowpath_fmt+0xa4/0xd8)
[Mon Nov 28 17:19:34 2022]  r7:000001de r6:c0ec6d04 r5:c1205048 r4:c0ec6cc8
[Mon Nov 28 17:19:34 2022] [<c0bd7bf4>] (warn_slowpath_fmt) from [<c0aac28c>] (dev_watchdog+0x304/0x308)
[Mon Nov 28 17:19:34 2022]  r9:00000005 r8:c1203d00 r7:c2390000 r6:c2378a00 r5:c23902e8 r4:00000002
[Mon Nov 28 17:19:34 2022] [<c0aabf88>] (dev_watchdog) from [<c02b05c0>] (call_timer_fn+0x40/0x198)
[Mon Nov 28 17:19:34 2022]  r9:eff12540 r8:fffffaf0 r7:c0aabf88 r6:00000100 r5:c23902e8 r4:fffffaf0
[Mon Nov 28 17:19:34 2022] [<c02b0580>] (call_timer_fn) from [<c02b1c10>] (run_timer_softirq+0x5e8/0x658)
[Mon Nov 28 17:19:34 2022]  r8:c155de0c r7:00000000 r6:c23902e8 r5:00000000 r4:fffffaf0
[Mon Nov 28 17:19:34 2022] [<c02b1628>] (run_timer_softirq) from [<c0201488>] (__do_softirq+0x198/0x474)
[Mon Nov 28 17:19:34 2022]  r10:ffffe000 r9:c155c000 r8:00000082 r7:00000100 r6:00000001 r5:00000002
[Mon Nov 28 17:19:34 2022]  r4:c1203084
[Mon Nov 28 17:19:34 2022] [<c02012f0>] (__do_softirq) from [<c02293d0>] (irq_exit+0xd8/0x120)
[Mon Nov 28 17:19:34 2022]  r10:c0e35ae0 r9:c155c000 r8:c1097158 r7:0000001e r6:00000000 r5:00000000
[Mon Nov 28 17:19:34 2022]  r4:ffffe000
[Mon Nov 28 17:19:34 2022] [<c02292f8>] (irq_exit) from [<c028b6c8>] (handle_domain_irq+0x70/0x88)
[Mon Nov 28 17:19:34 2022]  r5:00000000 r4:c109714c
[Mon Nov 28 17:19:34 2022] [<c028b658>] (handle_domain_irq) from [<c02012dc>] (gic_handle_irq+0x8c/0xa0)
[Mon Nov 28 17:19:34 2022]  r7:c155df30 r6:f081400c r5:f0814000 r4:c1205b6c
[Mon Nov 28 17:19:34 2022] [<c0201250>] (gic_handle_irq) from [<c0200aec>] (__irq_svc+0x4c/0x64)
[Mon Nov 28 17:19:34 2022] Exception stack(0xc155df30 to 0xc155df78)
[Mon Nov 28 17:19:34 2022] df20:                                     00042744 00000000 00000001 c021af00
[Mon Nov 28 17:19:34 2022] df40: ffffe000 00000004 c120509c c12050ec 00000001 c134209f c0e35ae0 c155df8c
[Mon Nov 28 17:19:34 2022] df60: c155df90 c155df80 c0208860 c0208864 60000013 ffffffff
[Mon Nov 28 17:19:34 2022]  r9:c155c000 r8:00000001 r7:c155df64 r6:ffffffff r5:60000013 r4:c0208864
[Mon Nov 28 17:19:34 2022] [<c020881c>] (arch_cpu_idle) from [<c0be90a0>] (default_idle_call+0x4c/0xd8)
[Mon Nov 28 17:19:34 2022] [<c0be9054>] (default_idle_call) from [<c025dc70>] (do_idle+0x114/0x164)
[Mon Nov 28 17:19:34 2022] [<c025db5c>] (do_idle) from [<c025dffc>] (cpu_startup_entry+0x28/0x2c)
[Mon Nov 28 17:19:34 2022]  r10:00000000 r9:410fd083 r8:00007000 r7:c155dff8 r6:30c0387d r5:00000002
[Mon Nov 28 17:19:34 2022]  r4:00000097 r3:60000093
[Mon Nov 28 17:19:34 2022] [<c025dfd4>] (cpu_startup_entry) from [<c020faa4>] (secondary_start_kernel+0x170/0x17c)
[Mon Nov 28 17:19:34 2022] [<c020f934>] (secondary_start_kernel) from [<00201810>] (0x201810)
[Mon Nov 28 17:19:34 2022]  r5:00000000 r4:01506040
[Mon Nov 28 17:19:34 2022] ---[ end trace 38ff62284e3b6609 ]---
[Mon Nov 28 17:19:42 2022] bcmgenet fd580000.ethernet eth0: Link is Down



@kindacute
Copy link

Same thing here:
Feb 16 22:17:29 kernel: ------------[ cut here ]------------
Feb 16 22:17:29 kernel: WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:478 dev_watchdog+0x304/0x308
Feb 16 22:17:29 kernel: NETDEV WATCHDOG: eth_lpd (cdc_ether): transmit queue 0 timed out
Feb 16 22:17:29 kernel: Modules linked in: xt_MASQUERADE nft_counter nft_chain_nat xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink cmac algif_hash aes_arm_bs crypto_simd cryptd algif_skcipher af_alg bnep hci_uart btbcm bluetooth ecdh_generic ecc brcmfmac vc4 brcmutil 8021q garp stp llc snd_soc_hdmi_codec cfg80211 cec drm_kms_helper cdc_ether v3d gpu_sched cdc_acm rfkill raspberrypi_hwmon snd_soc_core i2c_brcmstb bcm2835_v4l2(C) bcm2835_codec(C) rpivid_hevc(C) bcm2835_isp(C) bcm2835_mmal_vchiq(C) v4l2_mem2mem videobuf2_vmalloc videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common snd_compress snd_pcm_dmaengine snd_bcm2835(C) snd_pcm videodev snd_timer vc_sm_cma(C) snd mc syscopyarea sysfillrect sysimgblt fb_sys_fops nvmem_rmem uio_pdrv_genirq uio drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
Feb 16 22:17:29 kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: G C 5.15.72-v7l+ #1591
Feb 16 22:17:29 kernel: Hardware name: BCM2711
Feb 16 22:17:29 kernel: Backtrace:
Feb 16 22:17:29 kernel: [] (dump_backtrace) from [] (show_stack+0x20/0x24)
Feb 16 22:17:29 kernel: r7:000001de r6:c0e3f4b8 r5:00000000 r4:60000113
Feb 16 22:17:29 kernel: [] (show_stack) from [] (dump_stack_lvl+0x70/0x94)
Feb 16 22:17:29 kernel: [] (dump_stack_lvl) from [] (dump_stack+0x18/0x1c)
Feb 16 22:17:29 kernel: r7:000001de r6:00000009 r5:c0aab250 r4:c0ec69f0
Feb 16 22:17:29 kernel: [] (dump_stack) from [] (__warn+0xfc/0x114)
Feb 16 22:17:29 kernel: [] (__warn) from [] (warn_slowpath_fmt+0xa4/0xd8)
Feb 16 22:17:29 kernel: r7:000001de r6:c0ec69f0 r5:c1205048 r4:c0ec69b4
Feb 16 22:17:29 kernel: [] (warn_slowpath_fmt) from [] (dev_watchdog+0x304/0x308)
Feb 16 22:17:29 kernel: r9:00000001 r8:c1203d00 r7:c1792800 r6:c1722a00 r5:c1792ae8 r4:00000000
Feb 16 22:17:29 kernel: [] (dev_watchdog) from [] (call_timer_fn+0x40/0x198)
Feb 16 22:17:29 kernel: r9:efef2540 r8:1cf4fca0 r7:c0aaaf4c r6:00000100 r5:c1792ae8 r4:1cf4fca0
Feb 16 22:17:29 kernel: [] (call_timer_fn) from [] (run_timer_softirq+0x5e8/0x658)
Feb 16 22:17:29 kernel: r8:c1201d94 r7:00000000 r6:c1792ae8 r5:00000000 r4:1cf4fca0
Feb 16 22:17:29 kernel: [] (run_timer_softirq) from [] (__do_softirq+0x198/0x474)
Feb 16 22:17:29 kernel: r10:ffffe000 r9:c1200000 r8:00000082 r7:00000100 r6:00000001 r5:00000002
Feb 16 22:17:29 kernel: r4:c1203084
Feb 16 22:17:29 kernel: [] (__do_softirq) from [] (irq_exit+0xd8/0x120)
Feb 16 22:17:29 kernel: r10:c0e358b8 r9:c1200000 r8:c1096158 r7:0000001e r6:00000000 r5:00000000
Feb 16 22:17:29 kernel: r4:ffffe000
Feb 16 22:17:29 kernel: [] (irq_exit) from [] (handle_domain_irq+0x70/0x88)
Feb 16 22:17:29 kernel: r5:00000000 r4:c109614c
Feb 16 22:17:29 kernel: [] (handle_domain_irq) from [] (gic_handle_irq+0x8c/0xa0)
Feb 16 22:17:29 kernel: r7:c1201eb8 r6:f081400c r5:f0814000 r4:c1205b6c
Feb 16 22:17:29 kernel: [] (gic_handle_irq) from [] (__irq_svc+0x4c/0x64)
Feb 16 22:17:29 kernel: Exception stack(0xc1201eb8 to 0xc1201f00)
Feb 16 22:17:29 kernel: 1ea0: 8f8c5e64 00000000
Feb 16 22:17:29 kernel: 1ec0: 00000001 c021af00 ffffe000 00000001 c120509c c12050ec 00000001 c134201f
Feb 16 22:17:29 kernel: 1ee0: c0e358b8 c1201f14 c1201f18 c1201f08 c0208860 c0208864 60000013 ffffffff
Feb 16 22:17:29 kernel: r9:c1200000 r8:00000001 r7:c1201eec r6:ffffffff r5:60000013 r4:c0208864
Feb 16 22:17:29 kernel: [] (arch_cpu_idle) from [] (default_idle_call+0x4c/0xd8)
Feb 16 22:17:29 kernel: [] (default_idle_call) from [] (do_idle+0x114/0x164)
Feb 16 22:17:29 kernel: [] (do_idle) from [] (cpu_startup_entry+0x28/0x2c)
Feb 16 22:17:29 kernel: r10:effff900 r9:00000000 r8:00000000 r7:00000000 r6:ffffffff r5:c135b000
Feb 16 22:17:29 kernel: r4:000000e6 r3:c108f294
Feb 16 22:17:29 kernel: [] (cpu_startup_entry) from [] (rest_init+0xc8/0xd0)
Feb 16 22:17:29 kernel: [] (rest_init) from [] (arch_call_rest_init+0x18/0x1c)
Feb 16 22:17:29 kernel: r5:c135b000 r4:c135b068
Feb 16 22:17:29 kernel: [] (arch_call_rest_init) from [] (start_kernel+0x708/0x74c)
Feb 16 22:17:29 kernel: [] (start_kernel) from [<00000000>] (0x0)
Feb 16 22:17:29 kernel: ---[ end trace ce6c43b87da596b2 ]---

@pelwell
Copy link
Contributor

pelwell commented Feb 20, 2023

@kindacute That's not the same - you seem to be using your Pi 4 as an Ethernet gadget (notice the cdc_ether). It's not even clear if you are using the onboard Ethernet at all.

To everyone else:

This issue clearly persistently affects a small number of users, while most have no problems.

Please report what the other end of the Ethernet cable is plugged into (dedicated Ethernet switch, Wireless AP etc., specifying the model). As long as the kernel message is NETDEV WATCHDOG: eth0 (bcmgenet): transmit queue 2 timed out there is no need to post the stack traces as above.

@teewhey
Copy link

teewhey commented Feb 21, 2023

@kindacute That's not the same - you seem to be using your Pi 4 as an Ethernet gadget (notice the cdc_ether). It's not even clear if you are using the onboard Ethernet at all.

To everyone else:

This issue clearly persistently affects a small number of users, while most have no problems.

Please report what the other end of the Ethernet cable is plugged into (dedicated Ethernet switch, Wireless AP etc., specifying the model). As long as the kernel message is NETDEV WATCHDOG: eth0 (bcmgenet): transmit queue 2 timed out there is no need to post the stack traces as above.

For my case, the other end is my main wireless router (Asus RT-AX55)

@pelwell
Copy link
Contributor

pelwell commented Feb 21, 2023

There have been numerous reports of the power-saving Energy Efficient Ethernet setting causing problems when using BCM2711-based device with certain switches and routers. Provided you are running at least the most recent stable kernel (5.15.84) or later, try adding dtparam=eee=off to config.txt to see if that helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests