Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pi Compute Module 5, kernel panic. #1394

Open
gizahNL opened this issue Dec 12, 2024 · 6 comments · May be fixed by #1397
Open

Pi Compute Module 5, kernel panic. #1394

gizahNL opened this issue Dec 12, 2024 · 6 comments · May be fixed by #1397

Comments

@gizahNL
Copy link

gizahNL commented Dec 12, 2024

Description
The kernel panics on a CM5

[    0.886917] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
[    0.893526] SError Interrupt on CPU3, code 0x00000000be000011 -- SError
[    0.893529] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 6.6.36-v8-16k #1
[    0.893531] Hardware name: Raspberry Pi Compute Module 5 Rev 1.0 (DT)
[    0.893532] pstate: 804000c9 (Nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    0.893534] pc : bcm2712_pull_config_set+0x68/0xe8
[    0.893540] lr : bcm2712_pull_config_set+0x4c/0xe8
[    0.893541] sp : ffffc00080043ab0
[    0.893542] x29: ffffc00080043ab0 x28: ffffd000853cd068 x27: ffffd00085454008
[    0.893545] x26: ffffd00085300118 x25: ffff800016d58140 x24: 0000000000000000
[    0.893547] x23: 0000000000000018 x22: ffff800016aa7740 x21: 0000000000000122
[    0.893549] x20: ffff800016aa7680 x19: 0000000000000000 x18: ffffffffffffffff
[    0.893551] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800016d58148
[    0.893553] x14: ffffc000800437f0 x13: ffff800016d5814a x12: 3030306330356437
[    0.893555] x11: 0000000000000040 x10: ffff8000165db6e0 x9 : ffffd000847900d4
[    0.893557] x8 : ffff80001803a938 x7 : 0000000000000000 x6 : 0000000000000000
[    0.893559] x5 : ffff80001803a910 x4 : 0000000000000024 x3 : 0000000000000018
[    0.893561] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
[    0.893563] Kernel panic - not syncing: Asynchronous SError Interrupt
[    0.893565] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 6.6.36-v8-16k #1
[    0.893566] Hardware name: Raspberry Pi Compute Module 5 Rev 1.0 (DT)
[    0.893567] Call trace:
[    0.893568]  dump_backtrace+0x9c/0x100
[    0.893572]  show_stack+0x20/0x38
[    0.893573]  dump_stack_lvl+0x48/0x60
[    0.893577]  dump_stack+0x18/0x28
[    0.893579]  panic+0x318/0x378
[    0.893583]  nmi_panic+0x94/0xa0
[    0.893585]  arm64_serror_panic+0x78/0x98
[    0.893588]  arm64_is_fatal_ras_serror+0x44/0xc0
[    0.893589]  do_serror+0x64/0x80
[    0.893591]  el1h_64_error_handler+0x30/0x48
[    0.893594]  el1h_64_error+0x64/0x68
[    0.893596]  bcm2712_pull_config_set+0x68/0xe8
[    0.893597]  bcm2712_pinconf_set+0x6c/0xe0
[    0.893599]  pinconf_apply_setting+0xbc/0x138
[    0.893603]  pinctrl_commit_state+0xe8/0x220
[    0.893605]  pinctrl_select_state+0x24/0x40
[    0.893607]  pinctrl_bind_pins+0x144/0x160
[    0.893611]  really_probe+0x5c/0x2c0
[    0.893613]  __driver_probe_device+0x80/0x140
[    0.893615]  driver_probe_device+0x44/0x168
[    0.893617]  __driver_attach+0x98/0x1b0
[    0.893618]  bus_for_each_dev+0x84/0xf0
[    0.893620]  driver_attach+0x2c/0x40
[    0.893621]  bus_add_driver+0xec/0x218
[    0.893623]  driver_register+0x64/0x138
[    0.893625]  __platform_driver_register+0x30/0x48
[    0.893627]  brcmuart_init+0x44/0x88
[    0.893630]  do_one_initcall+0x60/0x2c8
[    0.893632]  kernel_init_freeable+0x1ec/0x3e8
[    0.893636]  kernel_init+0x2c/0x1f8
[    0.893639]  ret_from_fork+0x10/0x20
[    0.893642] SMP: stopping secondary CPUs
[    0.893645] Kernel Offset: 0x100004000000 from 0xffffc00080000000
[    0.893646] PHYS_OFFSET: 0x0
[    0.893646] CPU features: 0x0,00000001,70028143,0000720b
[    0.893648] Memory Limit: none
[    1.174851] ---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---

Steps to reproduce the issue:

  1. Build RPi5 machine config
  2. Enable uart console (panics regardless)
    config.txt:
dtparam=uart0=on
enable_uart=1
enable_rp1_uart=1
pciex4_reset=0
dtoverlay=disable-bt

cmdline.txt:
dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait net.ifnames=0 console=ttyAMA0,115200 earlycon=pl011,0x1c00030000,115200n8 (not sure if ttyAMA0 is correct, it panics regardless of that being set)
3. Boot and observe the kernel panic.

Additional details (revisions used, host distro, etc.):
I've tried adding the CM5 overlays to the raspberry5.conf & include, however this doesn't fix the panic:

meta-raspberrypi (master)$ git diff
diff --git a/conf/machine/include/rpi-base.inc b/conf/machine/include/rpi-base.inc
index 0b5909b..f1fb593 100644
--- a/conf/machine/include/rpi-base.inc
+++ b/conf/machine/include/rpi-base.inc
@@ -100,6 +100,8 @@ RPI_KERNEL_DEVICETREE ?= " \
     broadcom/bcm2711-rpi-cm4.dtb \
     broadcom/bcm2711-rpi-cm4s.dtb \
     broadcom/bcm2712-rpi-5-b.dtb \
+    broadcom/bcm2712-rpi-cm5-cm5io.dtb \
+    broadcom/bcm2712-rpi-cm5-cm4io.dtb \
     "
 
 KERNEL_DEVICETREE ??= " \
diff --git a/conf/machine/raspberrypi5.conf b/conf/machine/raspberrypi5.conf
index 03f15c3..bdd3bf3 100644
--- a/conf/machine/raspberrypi5.conf
+++ b/conf/machine/raspberrypi5.conf
@@ -15,6 +15,8 @@ MACHINE_EXTRA_RRECOMMENDS += "\
 
 RPI_KERNEL_DEVICETREE = " \
     broadcom/bcm2712-rpi-5-b.dtb \
+    broadcom/bcm2712-rpi-cm5-cm5io.dtb \
+    broadcom/bcm2712-rpi-cm5-cm4io.dtb \
 "
 
 SDIMG_KERNELIMAGE ?= "kernel_2712.img"

Full boot log:

P1_UART 0000001c00030000
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x414fd0b1]
[    0.000000] Linux version 6.6.36-v8-16k (oe-user@oe-host) (aarch64-poky-linux-gcc (GCC) 14.2.0, GNU ld (GNU Binutils) 2.43.1) #1 SMP PREEMPT Mon Jul  1 19:39:21 UTC 2024
[    0.000000] KASLR enabled
[    0.000000] random: crng init done
[    0.000000] Machine model: Raspberry Pi Compute Module 5 Rev 1.0
[    0.000000] earlycon: pl11 at MMIO 0x0000001c00030000 (options '115200n8')
[    0.000000] printk: bootconsole [pl11] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created CMA memory pool at 0x0000000002000000, size 320 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000000002000000..0x0000000015ffffff (327680 KiB) map reusable linux,cma
[    0.000000] OF: reserved mem: 0x0000000000000000..0x000000000007ffff (512 KiB) nomap non-reusable atf@0
[    0.000000] OF: reserved mem: 0x000000003fd16b00..0x000000003fd16be6 (0 KiB) nomap non-reusable nvram@0
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000000007ffff]
[    0.000000]   node   0: [mem 0x0000000000080000-0x000000003fafffff]
[    0.000000]   node   0: [mem 0x0000000040000000-0x00000000ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000] On node 0, zone DMA: 320 pages in unavailable ranges
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 14 pages/cpu s184936 r8192 d36248 u229376
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Virtualization Host Extensions
[    0.000000] CPU features: detected: Hardware dirty bit management
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe  smsc95xx.macaddr=2C:CF:67:B9:B2:AC vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  dw8[    0.000000] Dentry cache hash table entries: 524288 (order: 8, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 7, 2097152 bytes, linear)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 260800
[    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[    0.000000] Memory: 3809760K/4189184K available (14528K kernel code, 2274K rwdata, 4720K rodata, 2176K init, 1225K bss, 51744K reserved, 327680K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 46673 entries in 46 pages
[    0.000000] ftrace: allocated 46 pages with 4 groups
[    0.000000] trace event string verifier disabled
[    0.000000] rcu: Preemptible hierarchical RCU implementation.                                                                                                                              [    0.000000] rcu: 1152RCU event tracing is enabled.VT102 | Offline | ttyUSB0                                                                                                                [    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000]  Trampoline variant of Tasks RCU enabled.
[    0.000000]  Rude variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 54.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns
[    0.000000] sched_clock: 56 bits at 54MHz, resolution 18ns, wraps every 4398046511102ns
[    0.008363] Console: colour dummy device 80x25
[    0.012940] Calibrating delay loop (skipped), value calculated using timer frequency.. 108.00 BogoMIPS (lpj=216000)
[    0.023677] pid_max: default: 32768 minimum: 301
[    0.028435] LSM: initializing lsm=capability,integrity
[    0.033761] Mount-cache hash table entries: 8192 (order: 2, 65536 bytes, linear)
[    0.041375] Mountpoint-cache hash table entries: 8192 (order: 2, 65536 bytes, linear)
[    0.049717] cgroup: Disabling memory control group subsystem
[    0.055856] RCU Tasks: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1.
[    0.063122] RCU Tasks Rude: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1.
[    0.070831] RCU Tasks Trace: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1.
[    0.078654] rcu: Hierarchical SRCU implementation.
[    0.083575] rcu:     Max phase no-delay instances is 1000.
[    0.089338] EFI services will not be available.
[    0.094036] smp: Bringing up secondary CPUs ...
[    0.098807] Detected PIPT I-cache on CPU1
[    0.098834] CPU1: Booted secondary processor 0x0000000100 [0x414fd0b1]
[    0.098958] Detected PIPT I-cache on CPU2
[    0.098981] CPU2: Booted secondary processor 0x0000000200 [0x414fd0b1]
[    0.099098] Detected PIPT I-cache on CPU3
[    0.099117] CPU3: Booted secondary processor 0x0000000300 [0x414fd0b1]
[    0.099136] smp: Brought up 1 node, 4 CPUs
[    0.135819] SMP: Total of 4 processors activated.
[    0.140652] CPU features: detected: 32-bit EL0 Support
[    0.145935] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    0.154794] CPU features: detected: Common not Private translations
[    0.161237] CPU features: detected: CRC32 instructions
[    0.166517] CPU features: detected: RCpc load-acquire (LDAPR)
[    0.172424] CPU features: detected: LSE atomic instructions
[    0.178151] CPU features: detected: Privileged Access Never
[    0.183876] CPU features: detected: RAS Extension Support
[    0.189423] CPU features: detected: Speculative Store Bypassing Safe (SSBS)
[    0.196606] CPU: All CPU(s) started at EL2
[    0.200812] alternatives: applying system-wide alternatives
[    0.208112] devtmpfs: initialized
[    0.213468] Enabled cp15_barrier support
[    0.217501] Enabled setend support
[    0.221067] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.231098] futex hash table entries: 1024 (order: 2, 65536 bytes, linear)
[    0.238809] pinctrl core: initialized pinctrl subsystem
[    0.244282] DMI not present or invalid.
[    0.248367] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.254760] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[    0.262128] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.270208] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.278450] audit: initializing netlink subsys (disabled)
[    0.284030] audit: type=2000 audit(0.180:1): state=initialized audit_enabled=0 res=1
[    0.284116] thermal_sys: Registered thermal governor 'step_wise'
[    0.292002] cpuidle: using governor menu
[    0.302238] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.309234] ASID allocator initialised with 32768 entries
[    0.314966] Serial: AMBA PL011 UART driver
[    0.319799] bcm2835-mbox 107c013880.mailbox: mailbox enabled
[    0.325852] 107d001000.serial: ttyAMA10 at MMIO 0x107d001000 (irq = 15, base_baud = 0) is a PL011 rev2
[    0.343432] raspberrypi-firmware soc:firmware: Attached to firmware from 2024-11-27T14:28:19, variant start_cd
[    0.357727] raspberrypi-firmware soc:firmware: Firmware hash is 0b72471b00000000000000000000000000000000
[    0.373013] Modules: 2G module region forced by RANDOMIZE_MODULE_REGION_FULL
[    0.380266] Modules: 0 pages in range for non-PLT usage
[    0.380268] Modules: 129492 pages in range for PLT usage
[    0.392302] bcm2835-dma 1000010000.dma: DMA legacy API manager, dmachans=0x1
[    0.405247] iommu: Default domain type: Translated
[    0.410177] iommu: DMA domain TLB invalidation policy: strict mode
[    0.416768] SCSI subsystem initialized
[    0.420646] usbcore: registered new interface driver usbfs
[    0.426290] usbcore: registered new interface driver hub
[    0.431752] usbcore: registered new device driver usb
[    0.437002] pps_core: LinuxPPS API ver. 1 registered
[    0.442104] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.451501] PTP clock support registered
[    0.455557] Advanced Linux Sound Architecture Driver Initialized.
[    0.461975] vgaarb: loaded
[    0.464824] clocksource: Switched to clocksource arch_sys_counter
[    0.541666] VFS: Disk quotas dquot_6.6.0
[    0.545705] VFS: Dquot-cache hash table entries: 2048 (order 0, 16384 bytes)
[    0.552970] FS-Cache: Loaded
[    0.555950] CacheFiles: Loaded
[    0.560080] NET: Registered PF_INET protocol family
[    0.565159] IP idents hash table entries: 65536 (order: 5, 524288 bytes, linear)
[    0.573620] tcp_listen_portaddr_hash hash table entries: 2048 (order: 1, 32768 bytes, linear)
[    0.582417] Table-perturb hash table entries: 65536 (order: 4, 262144 bytes, linear)
[    0.590382] TCP established hash table entries: 32768 (order: 4, 262144 bytes, linear)
[    0.598606] TCP bind hash table entries: 32768 (order: 6, 1048576 bytes, linear)
[    0.606877] TCP: Hash tables configured (established 32768 bind 32768)
[    0.613665] MPTCP token hash table entries: 4096 (order: 2, 98304 bytes, linear)
[    0.621328] UDP hash table entries: 2048 (order: 2, 65536 bytes, linear)
[    0.628248] UDP-Lite hash table entries: 2048 (order: 2, 65536 bytes, linear)
[    0.635646] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.641554] RPC: Registered named UNIX socket transport module.
[    0.647640] RPC: Registered udp transport module.
[    0.652472] RPC: Registered tcp transport module.
[    0.657303] RPC: Registered tcp-with-tls transport module.
[    0.662940] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.669566] PCI: CLS 0 bytes, default 64
[    0.673734] kvm [1]: IPA Size Limit: 40 bits
[    0.678145] kvm [1]: GICV region size/alignment is unsafe, using trapping (reduced performance)
[    0.687112] kvm [1]: vgic interrupt IRQ9
[    0.691148] kvm [1]: VHE mode initialized successfully
[    0.696718] Initialise system trusted keyrings
[    0.701329] workingset: timestamp_bits=46 max_order=18 bucket_order=0
[    0.707959] zbud: loaded
[    0.710666] NFS: Registering the id_resolver key type
[    0.715859] Key type id_resolver registered
[    0.720155] Key type id_legacy registered
[    0.724274] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.731164] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.738791] F2FS not supported on PAGE_SIZE(16384) != 4096
[    0.744467] Key type asymmetric registered
[    0.748675] Asymmetric key parser 'x509' registered
[    0.753694] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[    0.761324] io scheduler mq-deadline registered
[    0.765978] io scheduler kyber registered
[    0.770100] io scheduler bfq registered
[    0.774201] irq_brcmstb_l2: registered L2 intc (/soc/interrupt-controller@7c502000, parent irq: 26)
[    0.783552] irq_brcmstb_l2: registered L2 intc (/soc/intc@7d503000, parent irq: 27)
[    0.791452] irq_brcmstb_l2: registered L2 intc (/soc/intc@7d508380, parent irq: 28)
[    0.799351] irq_brcmstb_l2: registered L2 intc (/soc/intc@7d508400, parent irq: 29)
[    0.807254] irq_brcmstb_l2: registered L2 intc (/soc/interrupt-controller@7d510600, parent irq: 30)
[    0.816585] irq_brcmstb_l2: registered L2 intc (/soc/intc@7d517b00, parent irq: 31)
[    0.825304] simple-framebuffer 3fb00000.framebuffer: framebuffer at 0x3fb00000, 0x96000 bytes
[    0.834076] simple-framebuffer 3fb00000.framebuffer: format=r5g6b5, mode=640x480x16, linelength=1280
[    0.843692] Console: switching to colour frame buffer device 80x30
[    0.850260] simple-framebuffer 3fb00000.framebuffer: fb0: simplefb registered!
[    0.886917] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
[    0.893526] SError Interrupt on CPU3, code 0x00000000be000011 -- SError
[    0.893529] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 6.6.36-v8-16k #1
[    0.893531] Hardware name: Raspberry Pi Compute Module 5 Rev 1.0 (DT)
[    0.893532] pstate: 804000c9 (Nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    0.893534] pc : bcm2712_pull_config_set+0x68/0xe8
[    0.893540] lr : bcm2712_pull_config_set+0x4c/0xe8
[    0.893541] sp : ffffc00080043ab0
[    0.893542] x29: ffffc00080043ab0 x28: ffffd000853cd068 x27: ffffd00085454008
[    0.893545] x26: ffffd00085300118 x25: ffff800016d58140 x24: 0000000000000000
[    0.893547] x23: 0000000000000018 x22: ffff800016aa7740 x21: 0000000000000122
[    0.893549] x20: ffff800016aa7680 x19: 0000000000000000 x18: ffffffffffffffff
[    0.893551] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800016d58148
[    0.893553] x14: ffffc000800437f0 x13: ffff800016d5814a x12: 3030306330356437
[    0.893555] x11: 0000000000000040 x10: ffff8000165db6e0 x9 : ffffd000847900d4
[    0.893557] x8 : ffff80001803a938 x7 : 0000000000000000 x6 : 0000000000000000
[    0.893559] x5 : ffff80001803a910 x4 : 0000000000000024 x3 : 0000000000000018
[    0.893561] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
[    0.893563] Kernel panic - not syncing: Asynchronous SError Interrupt
[    0.893565] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 6.6.36-v8-16k #1
[    0.893566] Hardware name: Raspberry Pi Compute Module 5 Rev 1.0 (DT)
[    0.893567] Call trace:
[    0.893568]  dump_backtrace+0x9c/0x100
[    0.893572]  show_stack+0x20/0x38
[    0.893573]  dump_stack_lvl+0x48/0x60
[    0.893577]  dump_stack+0x18/0x28
[    0.893579]  panic+0x318/0x378
[    0.893583]  nmi_panic+0x94/0xa0
[    0.893585]  arm64_serror_panic+0x78/0x98
[    0.893588]  arm64_is_fatal_ras_serror+0x44/0xc0
[    0.893589]  do_serror+0x64/0x80
[    0.893591]  el1h_64_error_handler+0x30/0x48
[    0.893594]  el1h_64_error+0x64/0x68
[    0.893596]  bcm2712_pull_config_set+0x68/0xe8
[    0.893597]  bcm2712_pinconf_set+0x6c/0xe0
[    0.893599]  pinconf_apply_setting+0xbc/0x138
[    0.893603]  pinctrl_commit_state+0xe8/0x220
[    0.893605]  pinctrl_select_state+0x24/0x40
[    0.893607]  pinctrl_bind_pins+0x144/0x160
[    0.893611]  really_probe+0x5c/0x2c0
[    0.893613]  __driver_probe_device+0x80/0x140
[    0.893615]  driver_probe_device+0x44/0x168
[    0.893617]  __driver_attach+0x98/0x1b0
[    0.893618]  bus_for_each_dev+0x84/0xf0
[    0.893620]  driver_attach+0x2c/0x40
[    0.893621]  bus_add_driver+0xec/0x218
[    0.893623]  driver_register+0x64/0x138
[    0.893625]  __platform_driver_register+0x30/0x48
[    0.893627]  brcmuart_init+0x44/0x88
[    0.893630]  do_one_initcall+0x60/0x2c8
[    0.893632]  kernel_init_freeable+0x1ec/0x3e8
[    0.893636]  kernel_init+0x2c/0x1f8
[    0.893639]  ret_from_fork+0x10/0x20
[    0.893642] SMP: stopping secondary CPUs
[    0.893645] Kernel Offset: 0x100004000000 from 0xffffc00080000000
[    0.893646] PHYS_OFFSET: 0x0
[    0.893646] CPU features: 0x0,00000001,70028143,0000720b
[    0.893648] Memory Limit: none
[    1.174851] ---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---
@stu-spp
Copy link

stu-spp commented Dec 14, 2024

Are you testing with the cm5 plugged into the IO Board?

It looks like the official Raspberry Pi OS images are shipping with:

x@xps:~/Downloads$ tar -tf bcm2712-official.tar.gz
home/x/Downloads/cm5-official-boot/bcm2712d0-rpi-5-b.dtb
home/x/Downloads/cm5-official-boot/bcm2712-rpi-5-b.dtb
home/x/Downloads/cm5-official-boot/bcm2712-rpi-cm5-cm4io.dtb
home/x/Downloads/cm5-official-boot/bcm2712-rpi-cm5-cm5io.dtb
home/x/Downloads/cm5-official-boot/bcm2712-rpi-cm5l-cm4io.dtb
home/x/Downloads/cm5-official-boot/bcm2712-rpi-cm5l-cm5io.dtb
home/x/Downloads/cm5-official-boot/initramfs_2712
home/x/Downloads/cm5-official-boot/kernel_2712.img
x@xps:~/Downloads$

bcm2712-official.tar.gz

I'll try to reproduce the panic you show, but while also using these extra sets of .dtb files. I guess the cm5l is for the cm5 lite variant? No eMMC? Which one are you using?

@stu-spp
Copy link

stu-spp commented Dec 15, 2024

I'm not sure what's wrong. I can't even get console uart out on my builds. I tried to go with a newer kernel than your 6.6.36 (because I wanted to match the recent rpi firmware). Here are my patches:

diff --git a/conf/machine/include/rpi-base.inc b/conf/machine/include/rpi-base.inc
index 0b5909b..0aeb5d0 100644
--- a/conf/machine/include/rpi-base.inc
+++ b/conf/machine/include/rpi-base.inc
@@ -100,6 +100,7 @@ RPI_KERNEL_DEVICETREE ?= " \
     broadcom/bcm2711-rpi-cm4.dtb \
     broadcom/bcm2711-rpi-cm4s.dtb \
     broadcom/bcm2712-rpi-5-b.dtb \
+    broadcom/bcm2712-rpi-cm5-cm5io.dtb \
     "

 KERNEL_DEVICETREE ??= " \
diff --git a/conf/machine/raspberrypi5.conf b/conf/machine/raspberrypi5.conf
index 03f15c3..c649ebf 100644
--- a/conf/machine/raspberrypi5.conf
+++ b/conf/machine/raspberrypi5.conf
@@ -15,6 +15,7 @@ MACHINE_EXTRA_RRECOMMENDS += "\

 RPI_KERNEL_DEVICETREE = " \
     broadcom/bcm2712-rpi-5-b.dtb \
+    broadcom/bcm2712-rpi-cm5-cm5io.dtb \
 "

 SDIMG_KERNELIMAGE ?= "kernel_2712.img"
diff --git a/recipes-bsp/bootfiles/rpi-bootfiles.bb b/recipes-bsp/bootfiles/rpi-bootfiles.bb
index b04f24b..ffb9388 100644
--- a/recipes-bsp/bootfiles/rpi-bootfiles.bb
+++ b/recipes-bsp/bootfiles/rpi-bootfiles.bb
@@ -5,14 +5,14 @@ LIC_FILES_CHKSUM = "file://LICENCE.broadcom;md5=c403841ff2837657b2ed8e5bb474ac8d

 inherit deploy nopackages

-RPIFW_DATE ?= "20240319"
-SRCREV = "9f24f4bc2bdd07ffd158cfbb4bce88a2efc4c1f5"
+RPIFW_DATE ?= "20241210"
+SRCREV = "24247756bb3e962f0429b048cec5644e4a4253ab"
 SHORTREV = "${@d.getVar("SRCREV", False).__str__()[:7]}"
-RPIFW_SRC_URI ?= "https://api.github.com/repos/raspberrypi/firmware/tarball/9f24f4bc2bdd07ffd158cfbb4bce88a2efc4c1f5;downloadfilename=raspberrypi-firmware-${SHORTREV}.tar.gz"
+RPIFW_SRC_URI ?= "https://api.github.com/repos/raspberrypi/firmware/tarball/${SRCREV};downloadfilename=raspberrypi-firmware-${SHORTREV}.tar.gz"
 RPIFW_S ?= "${WORKDIR}/raspberrypi-firmware-${SHORTREV}"

 SRC_URI = "${RPIFW_SRC_URI}"
-SRC_URI[sha256sum] = "4b436f8946b139c6a1202375ef55d4848e3bcd8c1a9cb47000e06d7ecec828f7"
+SRC_URI[sha256sum] = "50eceb608a3b3fa6b1f7cea7cd9618a7b910c5c3f541fb9e7c23eadcbdaee706"

 PV = "${RPIFW_DATE}"

diff --git a/recipes-kernel/linux/linux-raspberrypi_6.6.bb b/recipes-kernel/linux/linux-raspberrypi_6.6.bb
index d12a78b..703c5ec 100644
--- a/recipes-kernel/linux/linux-raspberrypi_6.6.bb
+++ b/recipes-kernel/linux/linux-raspberrypi_6.6.bb
@@ -1,8 +1,8 @@
-LINUX_VERSION ?= "6.6.36"
+LINUX_VERSION ?= "6.6.64"
 LINUX_RPI_BRANCH ?= "rpi-6.6.y"
 LINUX_RPI_KMETA_BRANCH ?= "yocto-6.6"

-SRCREV_machine = "769634f344626ed73bcda14c91b567067974d7b2"
+SRCREV_machine = "80533a952218696c0ef1b346bab50dc401e6b74c"
 SRCREV_meta = "733366844f5e114221372929392bf237fc8d823c"

 KMETA = "kernel-meta"

config.txt:

x@xps:~/workspace/code/usbboot$ cat /mntc/config.txt  | grep uart
#init_uart_baud=115200
#init_uart_clock=3000000
#dtparam=uart0=on
enable_uart=1
 dtparam=uart0=on
dtoverlay=uart0
dtparam=uart0_console
x@xps:~/workspace/code/usbboot$

cmdline.txt:

x@xps:~/workspace/code/usbboot$ cat /mntc/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait  net.ifnames=0 earlycon=pl011,0x1c00030000,115200n8
x@xps:~/workspace/code/usbboot$

Binwalk of the kernel shows 6.6.64 as built out above by my patches (which matches my rpi-bootfiles firmware):

x@xps:~/workspace/code/usbboot$ binwalk /mntc/kernel_2712.img

                                                     /mntc/kernel_2712.img
-------------------------------------------------------------------------------------------------------------------------------
DECIMAL                            HEXADECIMAL                        DESCRIPTION
-------------------------------------------------------------------------------------------------------------------------------
0                                  0x0                                Linux kernel ARM64 boot image, little endian, effective
                                                                      image size: 26607616 bytes
15466624                           0xEC0080                           Linux version 6.6.64-v8-16k (oe-user@oe-host)
                                                                      (aarch64-poky-linux-gcc (GCC) 14.2.0, GNU ld (GNU
                                                                      Binutils) 2.43.1) #1 SMP PREEMPT Thu Dec 12 19:45:21 UTC
                                                                      2024, has symbol table: false
15515648                           0xECC000                           ELF binary, 64-bit shared object, ARM 64-bit for
                                                                      System-V (Unix), little endian
16007208                           0xF44028                           SHA256 hash constants, little endian
16025280                           0xF486C0                           CRC32 polynomial table, little endian
16696536                           0xFEC4D8                           Linux version 6.6.64-v8-16k (oe-user@oe-host)
                                                                      (aarch64-poky-linux-gcc (GCC) 14.2.0, GNU ld (GNU
                                                                      Binutils) 2.43.1) #1 SMP PREEMPT Thu Dec 12 19:45:21 UTC
                                                                      2024, has symbol table: false
19597943                           0x12B0A77                          Copyright text: "Copyright 2005-2007 Rodolfo Giometti
                                                                      <giometti@linux.it> "
19628777                           0x12B82E9                          Copyright text: "Copyright(c) Pierre Ossman "
21139994                           0x142921A                          JBOOT STAG header, system upgrade image, header size: 16
                                                                      bytes, kernel data size: 3145480 bytes
21973764                           0x14F4B04                          CPIO ASCII archive, file count: 3
-------------------------------------------------------------------------------------------------------------------------------

Analyzed 1 file for 85 file signatures (187 magic patterns) in 864.0 milliseconds
x@xps:~/workspace/code/usbboot$

Bootfiles on eMMC:

x@xps:~/workspace/code/usbboot$ sudo ls -laht /mntc/
total 46M
-rwxr-xr-x  1 root root  79K Dec 14 23:08 bcm2712-rpi-cm5-cm4io.dtb
-rwxr-xr-x  1 root root  79K Dec 14 23:08 bcm2712-rpi-cm5-cm5io.dtb
-rwxr-xr-x  1 root root  79K Dec 14 23:08 bcm2712-rpi-cm5l-cm4io.dtb
-rwxr-xr-x  1 root root  79K Dec 14 23:08 bcm2712-rpi-cm5l-cm5io.dtb
-rwxr-xr-x  1 root root  141 Dec 14 23:02 cmdline.txt
-rwxr-xr-x  1 root root 2.6K Dec 14 23:00 config.txt
drwxr-xr-x 22 root root 4.0K Dec 14 19:57 ..
-rwxr-xr-x  1 root root  52K Apr  5  2011 bootcode.bin
-rwxr-xr-x  1 root root 3.2K Apr  5  2011 fixup4cd.dat
-rwxr-xr-x  1 root root 5.4K Apr  5  2011 fixup4.dat
-rwxr-xr-x  1 root root 8.3K Apr  5  2011 fixup4db.dat
-rwxr-xr-x  1 root root 8.3K Apr  5  2011 fixup4x.dat
-rwxr-xr-x  1 root root 3.2K Apr  5  2011 fixup_cd.dat
-rwxr-xr-x  1 root root 7.2K Apr  5  2011 fixup.dat
-rwxr-xr-x  1 root root  11K Apr  5  2011 fixup_db.dat
-rwxr-xr-x  1 root root  11K Apr  5  2011 fixup_x.dat
-rwxr-xr-x  1 root root  25M Apr  5  2011 kernel_2712.img
drwxr-xr-x  2 root root 8.0K Apr  5  2011 overlays
-rwxr-xr-x  1 root root    0 Apr  5  2011 rpi-bootfiles-20241210.stamp
-rwxr-xr-x  1 root root 794K Apr  5  2011 start4cd.elf
-rwxr-xr-x  1 root root 3.6M Apr  5  2011 start4db.elf
-rwxr-xr-x  1 root root 2.2M Apr  5  2011 start4.elf
-rwxr-xr-x  1 root root 2.9M Apr  5  2011 start4x.elf
-rwxr-xr-x  1 root root 794K Apr  5  2011 start_cd.elf
-rwxr-xr-x  1 root root 4.7M Apr  5  2011 start_db.elf
-rwxr-xr-x  1 root root 2.9M Apr  5  2011 start.elf
-rwxr-xr-x  1 root root 3.6M Apr  5  2011 start_x.elf
drwxr-xr-x  3 root root  16K Dec 31  1969 .
x@xps:~/workspace/code/usbboot$

@gizahNL
Copy link
Author

gizahNL commented Dec 16, 2024

Are you testing with the cm5 plugged into the IO Board?

Yes

It looks like the official Raspberry Pi OS images are shipping with:

x@xps:~/Downloads$ tar -tf bcm2712-official.tar.gz
home/x/Downloads/cm5-official-boot/bcm2712d0-rpi-5-b.dtb
home/x/Downloads/cm5-official-boot/bcm2712-rpi-5-b.dtb
home/x/Downloads/cm5-official-boot/bcm2712-rpi-cm5-cm4io.dtb
home/x/Downloads/cm5-official-boot/bcm2712-rpi-cm5-cm5io.dtb
home/x/Downloads/cm5-official-boot/bcm2712-rpi-cm5l-cm4io.dtb
home/x/Downloads/cm5-official-boot/bcm2712-rpi-cm5l-cm5io.dtb
home/x/Downloads/cm5-official-boot/initramfs_2712
home/x/Downloads/cm5-official-boot/kernel_2712.img
x@xps:~/Downloads$

bcm2712-official.tar.gz

I'll try to reproduce the panic you show, but while also using these extra sets of .dtb files. I guess the cm5l is for the cm5 lite variant? No eMMC? Which one are you using?

I'm working with one of the "fat ones", tbh not sure on the capacity :/

I'm not sure what's wrong. I can't even get console uart out on my builds. I tried to go with a newer kernel than your 6.6.36 (because I wanted to match the recent rpi firmware). Here are my patches:

config.txt:

x@xps:~/workspace/code/usbboot$ cat /mntc/config.txt  | grep uart
#init_uart_baud=115200
#init_uart_clock=3000000
#dtparam=uart0=on
enable_uart=1
 dtparam=uart0=on
dtoverlay=uart0
dtparam=uart0_console
x@xps:~/workspace/code/usbboot$

To get earlycon uart out on the GPIO you need to add pciex4_reset=0 to your config.txt, the GPIO uart on it is controlled by the RP1 chip, which is connected via pciex.
Not sure if this works with u-boot (I disabled u-boot to exclude it while trying to rootcause).

After creating this issue I've also tried with an image created with buildroot, yielding the same results.

@gokamura
Copy link

I saw a similar kernel panic like SError Interrupt on CPUx, code 0x00000000be000011 -- SError here. Googling the code 0x00000000be000011 led me to this discussion:
https://hub.mender.io/t/asynchronous-serror-interrupt-rpi5-boot-issue/7275/12

After adding
RPI_KERNEL_DEVICETREE_OVERLAYS:append = " overlays/bcm2712d0.dtbo" to my local.conf, the kernel panic has been fixed. Tested on the scarthgap branch.

@gizahNL
Copy link
Author

gizahNL commented Dec 20, 2024

@gokamura that indeed did fix it for me! It's booting now.

@stu-spp
Copy link

stu-spp commented Dec 20, 2024

Yes, I think that was my problem as well (adding bcm2712d0.dtb0 to the overlays). Not sure why I couldn't get UART console on boot earlier. My config.txt is pretty messy/nonsensical.

For posterity:

root@raspberrypi5:~# dmesg | head
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x414fd0b1]
[    0.000000] Linux version 6.6.64-v8-16k (oe-user@oe-host) (aarch64-poky-linux-gcc (GCC) 14.2.0, GNU ld (GNU Binutils) 2.43.1) #1 SMP PREEMPT Thu Dec 12 19:45:21 UTC 2024
[    0.000000] KASLR enabled
[    0.000000] random: crng init done
[    0.000000] Machine model: Raspberry Pi Compute Module 5 Rev 1.0
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created CMA memory pool at 0x0000000002000000, size 64 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000000002000000..0x0000000005ffffff (65536 KiB) map reusable linux,cma
[    0.000000] OF: reserved mem: 0x0000000000000000..0x000000000007ffff (512 KiB) nomap non-reusable atf@0
root@raspberrypi5:~#
root@raspberrypi5:~# ls -laht /boot/overlays/
total 24K
drwxr-xr-x    2 root     root        4.0K Apr  5  2011 .
-rwxr-xr-x    1 root     root        1.5K Apr  5  2011 bcm2712d0.dtbo
drwxr-xr-x    3 root     root       16.0K Jan  1  1970 ..
root@raspberrypi5:~# uname -a
Linux raspberrypi5 6.6.64-v8-16k #1 SMP PREEMPT Thu Dec 12 19:45:21 UTC 2024 aarch64 GNU/Linux
root@raspberrypi5:~# ls -laht /boot/
total 46M
drwxr-xr-x   18 root     root        4.0K Dec 20 19:06 ..
-rwxr-xr-x    1 root     root         104 Dec 20 19:05 cmdline.txt
-rwxr-xr-x    1 root     root       79.3K Apr  5  2011 bcm2712-rpi-5-b.dtb
-rwxr-xr-x    1 root     root       79.6K Apr  5  2011 bcm2712-rpi-cm5-cm4io.dtb
-rwxr-xr-x    1 root     root       79.7K Apr  5  2011 bcm2712-rpi-cm5-cm5io.dtb
-rwxr-xr-x    1 root     root       51.2K Apr  5  2011 bootcode.bin
-rwxr-xr-x    1 root     root        2.7K Apr  5  2011 config.txt
-rwxr-xr-x    1 root     root        7.2K Apr  5  2011 fixup.dat
-rwxr-xr-x    1 root     root        5.3K Apr  5  2011 fixup4.dat
-rwxr-xr-x    1 root     root        3.2K Apr  5  2011 fixup4cd.dat
-rwxr-xr-x    1 root     root        8.3K Apr  5  2011 fixup4db.dat
-rwxr-xr-x    1 root     root        8.3K Apr  5  2011 fixup4x.dat
-rwxr-xr-x    1 root     root        3.2K Apr  5  2011 fixup_cd.dat
-rwxr-xr-x    1 root     root       10.1K Apr  5  2011 fixup_db.dat
-rwxr-xr-x    1 root     root       10.1K Apr  5  2011 fixup_x.dat
-rwxr-xr-x    1 root     root       24.1M Apr  5  2011 kernel_2712.img
drwxr-xr-x    2 root     root        4.0K Apr  5  2011 overlays
-rwxr-xr-x    1 root     root           0 Apr  5  2011 rpi-bootfiles-20241210.stamp
-rwxr-xr-x    1 root     root        2.8M Apr  5  2011 start.elf
-rwxr-xr-x    1 root     root        2.2M Apr  5  2011 start4.elf
-rwxr-xr-x    1 root     root      793.7K Apr  5  2011 start4cd.elf
-rwxr-xr-x    1 root     root        3.6M Apr  5  2011 start4db.elf
-rwxr-xr-x    1 root     root        2.9M Apr  5  2011 start4x.elf
-rwxr-xr-x    1 root     root      793.7K Apr  5  2011 start_cd.elf
-rwxr-xr-x    1 root     root        4.6M Apr  5  2011 start_db.elf
-rwxr-xr-x    1 root     root        3.6M Apr  5  2011 start_x.elf
drwxr-xr-x    3 root     root       16.0K Jan  1  1970 .
root@raspberrypi5:~# cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait  net.ifnames=0
root@raspberrypi5:~# cat /boot/config.txt | tail
 dtparam=i2c_arm=on
 dtparam=i2c_vc=on
 dtoverlay=uart0
 dtoverlay=uart0_console
 dtparam=uart0=on
 enable_rp1_uart=1
 pciex4_reset=0
 dtoverlay=disable-bt
 dtoverlay=dwc2,dr_mode=host

root@raspberrypi5:~# cat /etc/os-release
ID=poky
NAME="Poky (Yocto Project Reference Distro)"
VERSION="5.1.1 (styhead)"
VERSION_ID=5.1.1
VERSION_CODENAME="styhead"
PRETTY_NAME="Poky (Yocto Project Reference Distro) 5.1.1 (styhead)"
CPE_NAME="cpe:/o:openembedded:poky:5.1.1"
root@raspberrypi5:~#

@gizahNL gizahNL linked a pull request Dec 23, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants