Wifi with 802.1x stopped working on rpi-6.6.y branch

[iucoen]

Describe the bug

• After updating to 8c590ad, wifi stops working. With a kernel warning message in brcmf_cfg80211_set_pmk
• Update: the problematic commit seems to be: 15f000a. After reverting EAP-TLS works again.

Steps to reproduce the behaviour

Boot up, error happens as soon as wpa_supplicant starts.

Device (s)

Raspberry Pi 5

System

I use gentoo linux. Kernel version 6.6.16-v8-16k+. For WIFI I'm using wpa_supplicant in EAP-TLS mode.

Logs

dmesg

[   44.160554] ------------[ cut here ]------------
[   44.165757] WARNING: CPU: 0 PID: 310 at drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c:5987 brcmf_cfg80211_set_pmk+0x12c/0x150 [brcmfmac]
[   44.179912] Modules linked in: joydev vc4 snd_soc_hdmi_codec cec brcmfmac_wcc drm_display_helper hci_uart drm_dma_helper pisp_be drm_kms_helper rpivid_hevc(C) btbcm v4l2_mem2mem videobuf2_dma_contig spidev videobuf2_memops aes_ce_blk snd_soc_core videobuf2_v4l2 snd_compress bluetooth aes_ce_cipher snd_pcm_dmaengine brcmfmac snd_pcm videodev ghash_ce ecdh_generic ecc videobuf2_common brcmutil snd_timer gf128mul libaes snd mc spi_bcm2835 raspberrypi_hwmon sha2_ce cfg80211 raspberrypi_gpiomem rp1_adc sha256_arm64 rfkill sha1_ce nvmem_rmem uio_pdrv_genirq gpio_keys uio pwm_fan sch_fq_codel drm fuse drm_panel_orientation_quirks backlight dm_mod
[   44.239470] CPU: 0 PID: 310 Comm: wpa_supplicant Tainted: G        WC         6.6.16-v8-16k+ #48
[   44.249020] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[   44.255617] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   44.263354] pc : brcmf_cfg80211_set_pmk+0x12c/0x150 [brcmfmac]
[   44.269998] lr : brcmf_cfg80211_set_pmk+0x50/0x150 [brcmfmac]
[   44.276554] sp : ffffc00080d7b7c0
[   44.280652] x29: ffffc00080d7b810 x28: ffff8001012fae80 x27: 0000000000000002
[   44.288590] x26: 0000000000000000 x25: ffff80010674b710 x24: ffffd0008587cb40
[   44.296530] x23: ffff800106608008 x22: ffffc00080d7b940 x21: ffff800101581000
[   44.304472] x20: ffffc00080d7b888 x19: ffff800101f683c0 x18: 0000000000000000
[   44.312422] x17: 0000000000000000 x16: ffffd00084e648a0 x15: 0000000000000000
[   44.320372] x14: 000000000000001f x13: 0000000000000147 x12: 00000000000085d5
[   44.328322] x11: 00000000be41a770 x10: ffffd00027116cc8 x9 : 29350a6fcda93600
[   44.336279] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 000000000000003f
[   44.344238] x5 : 0000000000000040 x4 : 0000000000000000 x3 : 0000000000000030
[   44.352184] x2 : ffffd000271613e7 x1 : ffffd00027163f69 x0 : 0000000000000002
[   44.360137] Call trace:
[   44.363403]  brcmf_cfg80211_set_pmk+0x12c/0x150 [brcmfmac]
[   44.369751]  rdev_set_pmk+0x44/0x1b0 [cfg80211]
[   44.375309]  nl80211_set_pmk+0x118/0x170 [cfg80211]
[   44.381189]  genl_rcv_msg+0x2e8/0x320
[   44.385717]  netlink_rcv_skb+0x128/0x150
[   44.390505]  genl_rcv+0x40/0x60
[   44.394516]  netlink_unicast+0x300/0x500
[   44.399314]  netlink_sendmsg+0x2d8/0x3e0
[   44.404114]  ____sys_sendmsg+0x1a8/0x2d0
[   44.408920]  __sys_sendmsg+0x128/0x1a0
[   44.413554]  __arm64_sys_sendmsg+0x30/0x50
[   44.418541]  invoke_syscall+0x4c/0x120
[   44.423172]  el0_svc_common+0xb8/0x100
[   44.427791]  do_el0_svc+0x28/0x40
[   44.431967]  el0_svc+0x38/0x90
[   44.435876]  el0t_64_sync_handler+0x84/0x100
[   44.441003]  el0t_64_sync+0x190/0x198
[   44.445493] ---[ end trace 0000000000000000 ]---

Additional context

No response

[pelwell]

It's working for me, but I've not tried EAP-TLS. Which firmware is this with?

$ strings /lib/firmware/brcm/brcmfmac43455-sdio.bin | grep Ver
$ dmesg | grep Firmware:

[iucoen]

It's working for me, but I've not tried EAP-TLS. Which firmware is this with?

$ strings /lib/firmware/brcm/brcmfmac43455-sdio.bin | grep Ver
$ dmesg | grep Firmware:

$ strings /lib/firmware/brcm/brcmfmac43455-sdio.bin | grep Ver
43455c0-roml/43455_sdio-pno-aoe-pktfilter-pktctx-lpc-pwropt-43455_ftrs-wfds-mfp-dfsradar-wowlpf-idsup-idauth-noclminc-clm_min-obss-obssdump-swdiv Version: 7.45.241 (1a2f2fa CY) CRC: 959ad1c7 Date: Mon 2021-11-01 00:40:29 PDT Ucode Ver: 1043.2164 FWID 01-703fd60
$ grep Firmware: dmesg.txt 
[    4.954859] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Nov  1 2021 00:37:25 version 7.45.241 (1a2f2fa CY) FWID 01-703fd60

[pelwell]

That's not our current firmware - we moved to a generic firmware from April 2021 (https://github.com/RPi-Distro/firmware-nonfree/blob/bookworm/debian/config/brcm80211/cypress/cyfmac43455-sdio-standard.bin) because it supports SAE (WPA3) in the firmware, although sadly only with iwd, not wpa_supplicant. There's also a newer trial version, which you can download from here (https://drive.google.com/file/d/13u_Ipf6yUATl38HyVNiHG3H2BVxCzfkB/view?usp=drive_link), that lets wpa_supplicant do the SAE handshake. I'm curious as to whether either of those, particularly the latter, works in your environment.

Note that those chanspec errors are non-fatal - it's complaining about channels that the firmware doesn't support, and at least in my environment it is right not to support them - but annoying. They can be disabled when we're convinced that they don't indicate some real problem.

[iucoen]

Is iwd supported yet? Last time I tried the defconfig files did not have the requisite kernel CONFIG_CRYPTO* options enabled.

[pelwell]

Configuring Network Manager to use iwd allowed a Pi 4 running that firmware to connect to a WPA3 network. Beyond that, I don't know.

[iucoen]

Configuring Network Manager to use iwd allowed a Pi 4 running that firmware to connect to a WPA3 network. Beyond that, I don't know.
I had to add these for EAP-TLS to work:

diff --git a/arch/arm64/configs/bcm2712_defconfig b/arch/arm64/configs/bcm2712_defconfig
index 5e7a777e9b59..6cb3c1e3a333 100644
--- a/arch/arm64/configs/bcm2712_defconfig
+++ b/arch/arm64/configs/bcm2712_defconfig
@@ -1625,20 +1625,29 @@ CONFIG_NLS_ISO8859_15=m
 CONFIG_NLS_KOI8_R=m
 CONFIG_NLS_KOI8_U=m
 CONFIG_DLM=m
+CONFIG_KEYS=y
+CONFIG_KEY_DH_OPERATIONS=y
 CONFIG_SECURITY=y
 CONFIG_SECURITY_APPARMOR=y
 CONFIG_LSM=""
 CONFIG_CRYPTO_USER=m
 CONFIG_CRYPTO_CRYPTD=m
+CONFIG_CRYPTO_RSA=m
+CONFIG_CRYPTO_DH=m
 CONFIG_CRYPTO_AES=m
 CONFIG_CRYPTO_CAST5=m
 CONFIG_CRYPTO_DES=y
 CONFIG_CRYPTO_TWOFISH=m
 CONFIG_CRYPTO_ADIANTUM=m
 CONFIG_CRYPTO_CBC=m
+CONFIG_CRYPTO_ECB=m
 CONFIG_CRYPTO_CHACHA20POLY1305=m
 CONFIG_CRYPTO_HMAC=m
 CONFIG_CRYPTO_MD4=m
+CONFIG_CRYPTO_MD5=m
+CONFIG_CRYPTO_SHA1=m
+CONFIG_CRYPTO_SHA256=m
+CONFIG_CRYPTO_SHA512=m
 CONFIG_CRYPTO_WP512=m
 CONFIG_CRYPTO_XCBC=m
 CONFIG_CRYPTO_LZ4=m
@@ -1658,6 +1667,11 @@ CONFIG_CRYPTO_AES_ARM64_BS=m
 CONFIG_CRYPTO_SM4_ARM64_CE=m
 CONFIG_CRYPTO_AES_ARM64_CE_CCM=m
 # CONFIG_CRYPTO_HW is not set
+CONFIG_ASYMMETRIC_KEY_TYPE=y
+CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=y
+CONFIG_X509_CERTIFICATE_PARSER=m
+CONFIG_PKCS8_PRIVATE_KEY_PARSER=m
+CONFIG_PKCS7_MESSAGE_PARSER=m
 CONFIG_CRC_ITU_T=y
 CONFIG_LIBCRC32C=y
 CONFIG_DMA_CMA=y

[pelwell]

Of those, only CONFIG_KEY_DH_OPERATIONS=y and CONFIG_CRYPTO_DH=m are non-standard (except for some 'm'/'y' differences).

[pelwell]

Are you going to be able to try the other firmwares? Unfortunately my test kit doesn't support EAP-TLS.

[iucoen]

Are you going to be able to try the other firmwares? Unfortunately my test kit doesn't support EAP-TLS.

Hi I tried the April firmware. It didn’t solve my problem. I think the problematic commit is 15f000a. After reverting that change I can verify that the April firmware does work with WPA3 Enterprise (in EAP-TLS mode) with iwd.

[iucoen]

Of those, only CONFIG_KEY_DH_OPERATIONS=y and CONFIG_CRYPTO_DH=m are non-standard (except for some 'm'/'y' differences).

I only pasted half of the patch earlier (fixed). Here's the second half:

@@ -1658,6 +1667,11 @@ CONFIG_CRYPTO_AES_ARM64_BS=m
 CONFIG_CRYPTO_SM4_ARM64_CE=m
 CONFIG_CRYPTO_AES_ARM64_CE_CCM=m
 # CONFIG_CRYPTO_HW is not set
+CONFIG_ASYMMETRIC_KEY_TYPE=y
+CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=y
+CONFIG_X509_CERTIFICATE_PARSER=m
+CONFIG_PKCS8_PRIVATE_KEY_PARSER=m
+CONFIG_PKCS7_MESSAGE_PARSER=m
 CONFIG_CRC_ITU_T=y
 CONFIG_LIBCRC32C=y

[lategoodbye]

Note that those chanspec errors are non-fatal - it's complaining about channels that the firmware doesn't support, and at least in my environment it is right not to support them - but annoying. They can be disabled when we're convinced that they don't indicate some real problem.

FWIW i reported this issue to linux-wireless:
https://marc.info/?l=linux-wireless&m=169972111422935&w=2

[iucoen]

Note that those chanspec errors are non-fatal - it's complaining about channels that the firmware doesn't support, and at least in my environment it is right not to support them - but annoying. They can be disabled when we're convinced that they don't indicate some real problem.

FWIW i reported this issue to linux-wireless: https://marc.info/?l=linux-wireless&m=169972111422935&w=2

ACK on that "chanspec errors are non-fatal". I believe that the error in brcmf_cfg80211_set_pmk() is legit and that's what's causing my wifi to not work.

[iucoen]

I think the problematic line of code is here: 15f000a#diff-3d76df104aa1b96e86b7acdaa0609be702f7973c7ac60b9eaa32f99d18f924a2R2490

Basically, if BRCMF_FEAT_FWSUP is enabled, and if the current connection is not using Pre-shared key (!sme->crypto.psk), then it sets profile->use_fwsup to BRCMF_PROFILE_FWSUP_NONE. Later brcmf_cfg80211_set_pmk() fails because it expects profile->use_fwsup to be BRCMF_PROFILE_FWSUP_1X.

I think patch 15f000a basically assumes that the connection is either PSK, or NONE, and didn't handle the third possibility that it could be 802.1X.

[iucoen]

Configuring Network Manager to use iwd allowed a Pi 4 running that firmware to connect to a WPA3 network. Beyond that, I don't know.
I had to add these for EAP-TLS to work:

I created PR #5974 for this.

[pelwell]

Thank you for all your work on this problem.