PI 3B+ wifi crash, firmware halt and hangs in dongle

[mrnatrium]

Describe the bug
Wifi firmware halts after a while of operation

To reproduce
no specific steps, leave PI running. can hang after a day, can take several days

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

• Which model of Raspberry Pi? Pi3B+
• Which OS and version (cat /etc/rpi-issue)?
  Raspberry Pi reference 2018-03-13
  Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 00013d7972122d1304 aacda8fff5098f073ceb43, stage5
• Which firmware version (vcgencmd version)?
  Jul 9 2019 14:40:53
  Copyright (c) 2012 Broadcom
  version 6c3fe3f096a93de3b34252ad98cdccadeb534be2 (clean) (release) (start)
• Which kernel version (uname -a)?
  Linux hawkeye 4.19.58-v7+ Master #1245 SMP Fri Jul 12 17:25:51 BST 2019 armv7l GNU/Linux

Logs
Syslog:
Aug 15 00:34:47 hawkeye kernel: [567883.509936] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted
Aug 15 00:34:50 hawkeye kernel: [567885.950023] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
Aug 15 00:34:50 hawkeye kernel: [567885.950367] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
Aug 15 00:34:50 hawkeye kernel: [567885.950384] brcmfmac: brcmf_run_escan: error (-110)
Aug 15 00:34:50 hawkeye kernel: [567885.950394] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
Aug 15 00:35:50 hawkeye kernel: [567945.950176] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
Aug 15 00:35:50 hawkeye kernel: [567945.950499] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
Aug 15 00:35:50 hawkeye kernel: [567945.950523] brcmfmac: brcmf_run_escan: error (-110)
Aug 15 00:35:50 hawkeye kernel: [567945.950533] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
Aug 15 00:36:50 hawkeye kernel: [568005.950344] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
Aug 15 00:36:50 hawkeye kernel: [568005.950697] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
Aug 15 00:36:50 hawkeye kernel: [568005.950713] brcmfmac: brcmf_run_escan: error (-110)
Aug 15 00:36:50 hawkeye kernel: [568005.950723] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
** continues every minute untill a NTP request is made, then the resume and trap messages disappear **
Aug 15 00:52:04 hawkeye systemd-timesyncd[285]: Timed out waiting for reply from 213.109.127.82:123 (2.debian.pool.ntp.org).
Aug 15 00:52:15 hawkeye systemd-timesyncd[285]: Timed out waiting for reply from 185.51.192.34:123 (2.debian.pool.ntp.org).
Aug 15 00:52:25 hawkeye systemd-timesyncd[285]: Timed out waiting for reply from 83.98.155.30:123 (2.debian.pool.ntp.org).
Aug 15 00:52:35 hawkeye systemd-timesyncd[285]: Timed out waiting for reply from 5.200.6.34:123 (2.debian.pool.ntp.org).
Aug 15 00:52:50 hawkeye kernel: [568965.952703] brcmfmac: brcmf_run_escan: error (-110)
Aug 15 00:52:50 hawkeye kernel: [568965.952718] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
Aug 15 00:53:50 hawkeye kernel: [569025.952866] brcmfmac: brcmf_run_escan: error (-110)
Aug 15 00:53:50 hawkeye kernel: [569025.952883] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
Aug 15 00:54:50 hawkeye kernel: [569085.952998] brcmfmac: brcmf_run_escan: error (-110)
Aug 15 00:54:50 hawkeye kernel: [569085.953015] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
Aug 15 00:55:50 hawkeye kernel: [569145.953156] brcmfmac: brcmf_run_escan: error (-110)
Aug 15 00:55:50 hawkeye kernel: [569145.953171] brcmfmac: brcmf_cfg80211_scan: scan error (-110)

then continues indefinitely every minute untill Pi reboots

Additional context
Not sure if it is related to issue #2453

[lategoodbye]

Please provide more information about your environment:
Does the Rpi 3B+ operate as station or access point?
Can you please provide your Wifi configuration (without credentials)?
Please describe the other connected devices in your Wifi network?
What is the Wifi firmware version (should be available after boot via dmesg)?

[mrnatrium]

The Pi operates as a client, other connecte devices to the network are WIFI printer, a Pi3B, Samsung phone and tablet, laptop and a smart thermostate.

I think you are looking for these dmesg lines:
[ 6.878693] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 6.892203] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04

Wifi config:
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=NL

network={
ssid="##"
psk="##"
key_mgmt=WPA-PSK
}

[kwasmich]

I am struggling with the same issue for about two months now.
I tried the Raspberry Pi forum with no success.
https://lb.raspberrypi.org/forums/viewtopic.php?f=28&t=249750&p=1544636&hilit=brcmfmac%3A+brcmf_sdio_hostmail

For me it must have started when I upgraded from Stretch to Buster and I did a firmware upgrade along that. Just to have everything latest. Before that I had my RPi 3 B+ running for months with no issue over 5GHz WiFi.

I tried reverting to Stretch realizing that all releases of Raspbian (since the release of RPi 3 B+) expose the same issue.
That is why I think it must be the firmware. I tried digging around but have come to no solution that fixes the issue so far.

[pjuzeliunas]

I have the same issue: WiFi freezes after some time on Raspberry Pi 3 B+ and I have to reboot the system to make it work again. Rasperry Pi uses built-in WiFi antena. Sometimes it takes few hours for issue to occur, and sometimes everything works fine for days. When it happens, I can see in /var/logs/syslog occurences of the following lines:

Jan 19 12:19:12 raspberrypi kernel: [155662.953243] brcmfmac: brcmf_run_escan: error (-110)
Jan 19 12:19:12 raspberrypi kernel: [155662.953261] brcmfmac: brcmf_cfg80211_scan: scan error (-110)

The same issue was discovered on Ubuntu (see report). One suggestion was to replace firmware that is shipped with the kernel by downloading and manually replacing brcmfmac43455-sdio.bin and brcmfmac43430-sdio.bin files. On Raspbian, firmware files are located in /lib/firmware/brcm/. However, original files from Raspian (kernel 4.19.66-v7+) and the suggested ones are identical.

[j64bit]

I have the same issue on Raspberry 4.

syslog

Jan 30 00:46:26 masterpi kernel: [1320334.491483] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted
Jan 30 00:46:28 masterpi kernel: [1320336.963492] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
Jan 30 00:46:28 masterpi kernel: [1320336.963889] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
Jan 30 00:46:28 masterpi kernel: [1320336.963906] brcmfmac: brcmf_run_escan: error (-110)
Jan 30 00:46:28 masterpi kernel: [1320336.963920] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
Jan 30 00:47:28 masterpi kernel: [1320396.964572] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
Jan 30 00:47:28 masterpi kernel: [1320396.964915] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
Jan 30 00:47:28 masterpi kernel: [1320396.964932] brcmfmac: brcmf_run_escan: error (-110)
Jan 30 00:47:28 masterpi kernel: [1320396.964945] brcmfmac: brcmf_cfg80211_scan: scan error (-110)

Last two lines then repeat forever until reboot, which I need to do by switch-off/on as i use the Raspi headless.

System Information

Raspberry Pi 4 Model B Rev 1.1^@
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"

Raspberry Pi reference 2019-09-26
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc3fc13cf3a2f8b464e129be, stage5

Linux masterpi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux
Revision : c03111
Serial : 100000003d877d18
Model : Raspberry Pi 4 Model B Rev 1.1
Throttled flag : throttled=0x0
Camera : supported=0 detected=0

[magnum61]

I am having the same problem on my Raspberry PI 4B

Syslog sys regarding brcmfmac:

Jan 31 17:14:13 Raspi4 kernel: [    4.088153] brcmfmac: F1 signature read @0x18000000=0x15264345
Jan 31 17:14:13 Raspi4 kernel: [    4.099201] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
Jan 31 17:14:13 Raspi4 kernel: [    4.099531] usbcore: registered new interface driver brcmfmac
Jan 31 17:14:13 Raspi4 kernel: [    4.345955] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
Jan 31 17:14:13 Raspi4 kernel: [    4.363165] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
Jan 31 17:14:13 Raspi4 kernel: [    6.267433] brcmfmac: power management disabled
Jan 31 17:14:14 Raspi4 kernel: [    6.571187] brcmfmac: power management disabled
Jan 31 17:16:36 Raspi4 kernel: [  149.078394] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted

System:
Linux Raspi4 4.19.93-v7l+ #1290 SMP Fri Jan 10 16:45:11 GMT 2020 armv7l GNU/Linux

The issue comes up every now and then. Sometimes twice a day, sometimes it runs for days.
Device is running headless as an ADS-B receiver without any additional tasks.

Would an external WiFi adapter bring more stability?

The device reboots and get connected again.

[criceto2]

Same logs and issue reported by other guys... !!

Some info from my logs:

Linux raspberrypi 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux

Raspberry Pi reference 2020-02-13
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 5f884374b6ac6e155330c58caa1fb7249b8badf1, stage4

Feb 12 2020 12:38:08
Copyright (c) 2012 Broadcom
version 53a54c770c493957d99bf49762dfabc4eee00e45 (clean) (release) (start)

[ 5.947279] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 5.963830] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 5.964543] usbcore: registered new interface driver brcmfmac
[ 6.337056] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 6.353510] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar 2 2020 23:30:41 version 7.45.202 (r724630 CY) FWID 01-72f6ece2
[ 9.061100] brcmfmac: power management disabled

Other:

May 9 00:01:06 raspberrypi kernel: [54979.845596] brcmfmac: brcmf_run_escan: error (-110)
May 9 00:01:06 raspberrypi kernel: [54979.845612] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
May 9 00:02:06 raspberrypi kernel: [55039.846099] brcmfmac: brcmf_run_escan: error (-110)
May 9 00:02:06 raspberrypi kernel: [55039.846115] brcmfmac: brcmf_cfg80211_scan: scan error (-110)

Any help or solution?

[aicastell]

Hello all. Same issue here with a Raspberry pi 4 model B Rev 1.1 using the on-board Wifi. I am using this Raspberry as a WiFi access point. This issue happens one or two times after a day of normal use. Usually when here is a lot of traffic in the network (copying files, or reproducing video with the Chromecast). After this happens, all wifi-clients are disconnected and you have to power-off and power-on the Raspberry to get it working again.

You can find all the relevant information attached in this tarball:

wifi-halted-report.tar.gz

Inside the tarball you can find: daemon.log, dpkg.log, hostapd.conf, ifconfig.log, kernel.log, lsb_release.log, lspci.log and syslog.log

Most relevant logs are located into kernel.log file, lines between 3538 and 3547. There is a time issue into this log file 20:56:37 --> 22:52:47 --> ... --> 22:17:05, not sure if it's related with the issue or not, but is strange that just in that moment of time happens the issue.

Here you can see the error:

3538 Jun 16 20:56:37 raspberrypi kernel: [13183.398916] usb 1-1.1.5.5: USB disconnect, device number 8
3539 Jun 16 22:52:47 raspberrypi kernel: [20153.194747] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted
3540 Jun 16 22:53:30 raspberrypi kernel: [20196.728384] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
3541 Jun 16 22:53:30 raspberrypi kernel: [20196.728829] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
3542 Jun 16 22:53:33 raspberrypi kernel: [20199.288384] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
3543 Jun 16 22:53:33 raspberrypi kernel: [20199.288757] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
3544 Jun 16 22:53:33 raspberrypi kernel: [20199.288775] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110
3545 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@     ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@     ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@     ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@     ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@     ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Jun 16 22:17:05 raspberrypi kernel: [    0.000000] Booting Linux on physical CPU 0x0
3546 Jun 16 22:17:05 raspberrypi kernel: [    0.000000] Linux version 4.19.118-v7l+ (dom@buildbot) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1311 SMP Mon Apr 27 14:26:42 BST 2020
3547 Jun 16 22:17:05 raspberrypi kernel: [    0.000000] CPU: ARMv7 Processor [410fd083] revision 3 (ARMv7), cr=30c5383d
3548 Jun 16 22:17:05 raspberrypi kernel: [    0.000000] CPU: div instructions available: patching division code
3549 Jun 16 22:17:05 raspberrypi kernel: [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
3550 Jun 16 22:17:05 raspberrypi kernel: [    0.000000] OF: fdt: Machine model: Raspberry Pi 4 Model B Rev 1.1
3551 Jun 16 22:17:05 raspberrypi kernel: [    0.000000] Memory policy: Data cache writealloc
3552 Jun 16 22:17:05 raspberrypi kernel: [    0.000000] cma: Reserved 256 MiB at 0x000000001ec00000

Basically I have a bridge br0 between wlan0 and eth0. DHCP server is running on my home router connected to eth0. I am using hostapd to convert wlan0 into an Access Point. If you need more detailed information, I can provide it.

Hope this helps!