JH7110_visionfive2_upstream: store (or AMO) access fault

[SeTSeR]

On NixOS with latest upstream kernel without additional patches every Nix command, such as nix build nixpkgs#verilog, leads to kernel panic with the following trace:

[  286.987668] Oops - store (or AMO) access fault [#1]
[  286.987687] Modules linked in: xt_MASQUERADE xt_mark nft_chain_nat nf_nat ctr aes_generi
c libaes ccm af_packet rtl8192cu rtl_usb rtl8192c_common rtlwifi 8021q mac80211 cfg80211 rf
kill libarc4 sd_mod uas usb_storage scsi_mod btrfs scsi_common onboard_usb_hub blake2b_gene
ric xor raid6_pq xhci_pci xhci_pci_renesas xhci_hcd nvme nvme_core t10_pi crc64_rocksoft_ge
neric crc64_rocksoft crc_t10dif usbcore motorcomm dwmac_starfive stmmac_platform axp20x_reg
ulator crct10dif_generic crc64 crct10dif_common imx219 cdns_csi2rx ofpart cmdlinepart stmma
c v4l2_fwnode spi_nor cdns3 snd_soc_spdif_tx cdns_usb_common v4l2_async axp20x_i2c axp20x r
oles mfd_core snd_soc_simple_card videodev snd_soc_simple_card_utils mtd spidev regmap_i2c 
jh7110_pwmdac designware_i2s jh7110_tdm snd_soc_core mc vs_drm drm_dma_helper snd_compress 
drm_kms_helper ac97_bus snd_pcm_dmaengine starfive_wdt snd_pcm cdns3_starfive usb_common cl
k_starfive_jh7110_vout pcs_xpcs phy_jh7110_dphy_rx phylink watchdog of_mdio fixed_phy fwnod
e_mdio libphy snd_timer
[  286.988118]  pcie_starfive ptp snd dw_axi_dmac_platform pps_core led_class pinctrl_starf
ive_jh7110_aon virt_dma clk_starfive_jh7110_isp clk_starfive_jh7110_aon spi_cadence_quadspi
 jh7110_trng clk_starfive_jh7110_stg rng_core sfctemp spi_pl022 soundcore phy_jh7110_usb i2
c_designware_platform i2c_designware_core phy_jh7110_pcie xt_conntrack nf_conntrack nf_defr
ag_ipv6 nf_defrag_ipv4 uio_pdrv_genirq uio ip6t_rpfilter ipt_rpfilter xt_pkttype xt_LOG nf_
log_syslog xt_tcpudp nft_compat nf_tables libcrc32c nfnetlink sch_fq_codel atkbd libps2 ser
io vivaldi_fmap loop tun tap macvlan drm bridge fuse stp llc backlight i2c_core efi_pstore 
pstore firmware_class zlib_deflate configfs ip_tables x_tables autofs4 ext4 crc32c_generic 
crc16 mbcache jbd2 mmc_block dw_mmc_starfive dw_mmc_pltfm dw_mmc mmc_core dm_mod dax
[  286.988455] CPU: 0 PID: 2088 Comm: nix-daemon Not tainted 6.6.0 #1-NixOS
[  286.988469] Hardware name: StarFive VisionFive 2 v1.3B (DT)
[  286.988475] epc : __memcpy+0x60/0xf8
[  286.988497]  ra : copy_user_highpage.constprop.0+0x42/0x64
[  286.988513] epc : ffffffff80898b28 ra : ffffffff8020ff02 sp : ffffffc800b8bc70
[  286.988521]  gp : ffffffff818f7b58 tp : ffffffd8d8058000 t0 : 0000000000000000
[  286.988529]  t1 : 0000000000000000 t2 : 0000003f14011c80 s0 : ffffffc800b8bc80
[  286.988536]  s1 : ffffffc800b8bd48 a0 : ffffffd800000000 a1 : ffffffd8f3bef000
[  286.988544]  a2 : 0000000000001000 a3 : ffffffd8f3bf0000 a4 : 0000000000000120
[  286.988551]  a5 : 00000000000000c4 a6 : 0000000003f1400e a7 : 83ce8fce711f6a2d
[  286.988559]  s2 : ffffffc704cefbc0 s3 : ffffffc701000000 s4 : ffffffd8dd5c3580
[  286.988566]  s5 : ffffffc800b8bca0 s6 : ffffffd8c257b600 s7 : 0000000000001a55
[  286.988573]  s8 : ffffffff81b7f2b8 s9 : 0000003f1400e000 s10: 0000000000000000
[  286.988581]  s11: 0000002ae19b42b0 t3 : 0000000000000031 t4 : 0000000000000031

[  286.988588]  t5 : 00007367616c4665 t6 : ffffffd800000000
[  286.988595] status: 0000000200000120 badaddr: ffffffd800000000 cause: 0000000000000007
[  286.988603] [<ffffffff80898b28>] __memcpy+0x60/0xf8
[  286.988618] [<ffffffff80212f74>] do_wp_page+0x242/0xd14
[  286.988630] [<ffffffff80215cba>] __handle_mm_fault+0x64c/0xda0
[  286.988644] [<ffffffff8021658e>] handle_mm_fault+0x180/0x27c
[  286.988656] [<ffffffff8000d692>] handle_page_fault+0xfe/0x45c
[  286.988671] [<ffffffff808a5f20>] do_page_fault+0x22/0x3e
[  286.988687] [<ffffffff80003e04>] ret_from_exception+0x0/0x64
[  286.988715] Code: b303 0285 b383 0305 be03 0385 be83 0405 bf03 0485 (b023) 00ef
[  286.988728] ---[ end trace 0000000000000000 ]---
[  286.988730] Oops - store (or AMO) access fault [#2]
[  286.988735] Kernel panic - not syncing: Fatal exception in interrupt
[  286.988741] Modules linked in:
[  286.988741] SMP: stopping secondary CPUs
[  286.988746]  xt_MASQUERADE xt_mark nft_chain_nat nf_nat ctr aes_generic libaes ccm af_packet rtl8192cu rtl_usb rtl8192c_common rtlwifi 8021q mac80211 cfg80211 rfkill libarc4 sd_mod uas usb_storage scsi_mod btrfs scsi_common onboard_usb_hub blake2b_generic xor raid6_pq xhci_pci xhci_pci_renesas xhci_hcd nvme nvme_core t10_pi crc64_rocksoft_generic crc64_rocksoft crc_t10dif usbcore motorcomm dwmac_starfive stmmac_platform axp20x_regulator crct10dif_generic crc64 crct10dif_common imx219 cdns_csi2rx ofpart cmdlinepart stmmac v4l2_fwnode spi_nor cdns3 snd_soc_spdif_tx cdns_usb_common v4l2_async axp20x_i2c axp20x roles mfd_core snd_soc_simple_card videodev snd_soc_simple_card_utils mtd spidev regmap_i2c jh7110_pwmdac designware_i2s jh7110_tdm snd_soc_core mc vs_drm drm_dma_helper snd_compress drm_kms_helper ac97_bus snd_pcm_dmaengine starfive_wdt snd_pcm cdns3_starfive usb_common clk_starfive_jh7110_vout pcs_xpcs phy_jh7110_dphy_rx phylink watchdog of_mdio fixed_phy fwnode_mdio libphy snd_timer pcie_starfive ptp snd
[  286.989117]  dw_axi_dmac_platform pps_core led_class pinctrl_starfive_jh7110_aon virt_dma clk_starfive_jh7110_isp clk_starfive_jh7110_aon spi_cadence_quadspi jh7110_trng clk_starfive_jh7110_stg rng_core sfctemp spi_pl022 soundcore phy_jh7110_usb i2c_designware_platform i2c_designware_core phy_jh7110_pcie xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 uio_pdrv_genirq uio ip6t_rpfilter ipt_rpfilter xt_pkttype xt_LOG nf_log_syslog xt_tcpudp nft_compat nf_tables libcrc32c nfnetlink sch_fq_codel atkbd libps2 serio vivaldi_fmap loop tun tap macvlan drm bridge fuse stp llc backlight i2c_core efi_pstore pstore firmware_class zlib_deflate configfs ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 mmc_block dw_mmc_starfive dw_mmc_pltfm dw_mmc mmc_core dm_mod dax

[  286.989391] CPU: 1 PID: 3050 Comm: nix-daemon Tainted: G      D            6.6.0 #1-NixOS
[  286.989403] Hardware name: StarFive VisionFive 2 v1.3B (DT)
[  286.989408] epc : __memcpy+0x60/0xf8
[  286.989424]  ra : copy_user_highpage.constprop.0+0x42/0x64
[  286.989435] epc : ffffffff80898b28 ra : ffffffff8020ff02 sp : ffffffc8082cbc70
[  286.989443]  gp : ffffffff818f7b58 tp : ffffffd8d8048d00 t0 : 0000000000000000
[  286.989451]  t1 : 0000000000000000 t2 : 0000000000000060 s0 : ffffffc8082cbc80
[  286.989458]  s1 : ffffffc8082cbd48 a0 : ffffffd800020000 a1 : ffffffd8f389c000
[  286.989465]  a2 : 0000000000001000 a3 : ffffffd8f389d000 a4 : 0000003f700d8015
[  286.989472]  a5 : 7265730100000000 a6 : 7261562000726576 a7 : 000000006873696e
[  286.989480]  s2 : ffffffc704ce2700 s3 : ffffffc701000800 s4 : ffffffd8e4823c60
[  286.989487]  s5 : ffffffc8082cbca0 s6 : ffffffd8c257b600 s7 : 0000000000001a55
[  286.989495]  s8 : ffffffff81b7f2b8 s9 : 0000003f700d8000 s10: 0000000000000001
[  286.989502]  s11: 0000000000800000 t3 : 0000000000000024 t4 : 0000003f8f2bcd58
[  286.989509]  t5 : 0000003f700d9620 t6 : ffffffd800020000
[  286.989515] status: 0000000200000120 badaddr: ffffffd800020000 cause: 0000000000000007
[  286.989523] [<ffffffff80898b28>] __memcpy+0x60/0xf8
[  286.989536] [<ffffffff80212f74>] do_wp_page+0x242/0xd14
[  286.989548] [<ffffffff80215cba>] __handle_mm_fault+0x64c/0xda0
[  286.989561] [<ffffffff8021658e>] handle_mm_fault+0x180/0x27c
[  286.989572] [<ffffffff8000d692>] handle_page_fault+0xfe/0x45c
[  286.989585] [<ffffffff808a5f20>] do_page_fault+0x22/0x3e
[  286.989601] [<ffffffff80003e04>] ret_from_exception+0x0/0x64
[  286.989624] Code: b303 0285 b383 0305 be03 0385 be83 0405 bf03 0485 (b023) 00ef
[  286.989632] ---[ end trace 0000000000000000 ]---
[  288.623424] SMP: failed to stop secondary CPUs 0-1
[  288.623436] Kernel Offset: 0x0 from 0xffffffff80000000
[  289.290764] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

I believe the problem is pretty old, I encountered it in upstream kernels several times last year. Kernel is built with GCC 12.3.

[malteneuss]

@SeTSeR Does it happen with the latest 6.9 relese in nixpkgs-unstable? Would you know how to fix NixOS/nixos-hardware#958?

[onny]

How did you run mainline kernel with NixOS?

[SeTSeR]

How did you run mainline kernel with NixOS?

I didn't run mainline kernel, just the kernel from branch with preparations for upstream. I run it using boot.kernelPackages option. As for latest kernel for nixos-unstable, I did not try it, but it appears to be unbootable for now.

[SeTSeR]

It seems on current upstream kernel (6.11) the problem is not present anymore.