Debugging "Exception occurred 'StoreProhibited' " Where do I start?

[yalcinozhabes]

Hi,

I am trying to read GPS data from i2c and share it on BLE. These are lengthy reads on i2c (order of 1kbyte per sec). When I enable BLE (nothing fancy just copy-paste of the example code) I start getting these errors.

Exception occurred 'StoreProhibited'
Context
PC=0x4000c440       PS=0x00060015
A0=0x80088e76       A1=0x3ffdb530       A2=0x0000019a       A3=0x00000000       A4=0x00000018
A5=0x0000019a       A6=0x00000040       A7=0x00000000       A8=0x00000000       A9=0x00000000
A10=0x00060001      A11=0x00000000      A12=0x400fc030      A13=0x00000000      A14=0x400f9984
0x400fc030 - <esp_hal::sync::CriticalSection as critical_section::Impl>::acquire
    at .cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-hal-0.23.1/src/sync.rs:447
0x400f9984 - esp_hal::time::now
    at ??:??
A15=0x0011311f
SAR=0000000f
EXCCAUSE=0x0000001d EXCVADDR=0x0000019a
LBEG=0x4000c349     LEND=0x4000c36b     LCOUNT=0xffffffff
THREADPTR=0x00000000
SCOMPARE1=0x00000100
BR=0x00000000
ACCLO=0x00000000    ACCHI=0x00000000
M0=0x00000000       M1=0x00000000       M2=0x00000000       M3=0x00000000
F64R_LO=0x00000000  F64R_HI=0x00000000  F64S=0x00000000
FCR=0x00000000      FSR=0x00000080
F0=0x433b8000       F1=0x426c0000       F2=0x43800000       F3=0x00000000       F4=0x00000000
F5=0x00000000       F6=0x00000000       F7=0x00000000       F8=0x00000000       F9=0x42c7fae1
F10=0x42c7fae1      F11=0x00000000      F12=0x00000000      F13=0x00000000      F14=0x00000000
F15=0x00000000
0x40088777
coex_core_request
    at ??:??
0x40088475
coex_bt_request
    at ??:??
0x400f703c
esp_wifi::ble::btdm::ble_os_adapter_chip_specific::coex_bt_request
    at .cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-wifi-0.12.0/src/ble/os_adapter_esp32.rs:403
0x40083221
coex_sw_event_from_isr
    at ??:??
0x400854e2
r_lld_evt_schedule
    at ??:??
0x4008378d
r_ea_finetimer_isr
    at ??:??
0x400552d0
0x400881de
r_rwbtdm_isr_wrapper
    at ??:??
0x400f5686
RWBT
    at .cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-wifi-0.12.0/src/timer/timer_esp32.rs:74
0x40080f48
esp_hal::interrupt::xtensa::vectored::handle_interrupt
    at .cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-hal-0.23.1/src/interrupt/xtensa.rs:658

How do I start debugging this?

[bugadani]

First, please update to esp-hal 1.0.0-beta.0, esp-wifi 0.13 and see if this is still an issue.

[yalcinozhabes]

I'm working on the migration to 1.0.0-beta.0 and esp-wifi 0.13 but having linker issues:

.rustup/toolchains/esp/xtensa-esp-elf/esp-14.2.0_20240906/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/14.2.0/../../../../xtensa-esp-elf/bin/ld: ../target/xtensa-esp32-none-elf/debug/deps/libesp_wifi-36e2b5d81eb75eec.rlib(esp_wifi-36e2b5d81eb75eec.esp_wifi.bd7a3a12df30e77a-cgu.3.rcgu.o):(.literal._ZN8esp_wifi7preempt6enable17h74225fbb4c004a94E+0x0): undefined reference to `esp_wifi_preempt_enable'
          collect2: error: ld returned 1 exit status
          
  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#rustc-link-lib)

Is there also a beta channel for the toolchain that I need to switch to?

[bugadani]

Make sure you don't have default-features = false set for esp-wifi, or that you enable the builtin-scheduler feature for it.

[yalcinozhabes]

Ok thanks. It compiled and hit a similar error.

I will update the post with the new error after I verify the behavior is the same.

[yalcinozhabes]

It is still happening with 1.0.0-beta.0 and esp-wifi 0.13:

Exception occurred 'InstrProhibited'
Context
PC=0x00000000       PS=0x00060011
A0=0x800f8366       A1=0x3ffc8eb0       A2=0x3ffd8318       A3=0x400f9fb4       A4=0x4000c2c8
0x3ffc8eb0 - sh_udp::____embassy_main_task::{{closure}}::HEAP
    at ??:??
0x3ffd8318 - r_ip_funcs_p
    at ??:??
0x400f9fb4 - <esp_hal::sync::LockGuard<L> as core::ops::drop::Drop>::drop
    at .cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-hal-1.0.0-beta.0/src/sync.rs:482
A5=0x400f4fc8       A6=0x3ffd7fec       A7=0x3ffc80a0       A8=0x800881be       A9=0x3ffd8150
0x400f4fc8 - esp_hal::sync::LockGuard<L>::new
    at .cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-hal-1.0.0-beta.0/src/sync.rs:462
0x3ffd7fec - esp_wifi::common_adapter::chip_specific::PHY_ACCESS_REF
    at ??:??
0x3ffc80a0 - sh_udp::____embassy_main_task::{{closure}}::HEAP
    at ??:??
0x3ffd8150 - esp_alloc::HEAP
    at ??:??
A10=0x3ffd8220      A11=0x3ffc8178      A12=0x000000d8      A13=0x3ffd82f8      A14=0x00000000
0x3ffd8220 - is_lc_lmp_rx_flooding
    at ??:??
0x3ffc8178 - sh_udp::____embassy_main_task::{{closure}}::HEAP
    at ??:??
0x3ffd82f8 - lld_pdu_rx_env
    at ??:??
A15=0x00000000
SAR=0000001a
EXCCAUSE=0x00000014 EXCVADDR=0x00000000
LBEG=0x4000c349     LEND=0x4000c36b     LCOUNT=0xffffffff
THREADPTR=0x00000000
SCOMPARE1=0x00000100
BR=0x00000000
ACCLO=0x00000000    ACCHI=0x00000000
M0=0x00000000       M1=0x00000000       M2=0x00000000       M3=0x00000000
F64R_LO=0x00000000  F64R_HI=0x00000000  F64S=0x00000000
FCR=0x00000000      FSR=0x00000000
F0=0x00000000       F1=0x00000000       F2=0x00000000       F3=0x00000000       F4=0x00000000
F5=0x00000000       F6=0x00000000       F7=0x00000000       F8=0x00000000       F9=0x00000000
F10=0x00000000      F11=0x00000000      F12=0x00000000      F13=0x00000000      F14=0x00000000
F15=0x00000000
0x40080f34
esp_hal::interrupt::xtensa::vectored::handle_interrupt
    at .cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-hal-1.0.0-beta.0/src/interrupt/xtensa.rs:653
0x400810bd
handle_interrupts
    at ??:??
0x40081356
.RestoreContext
    at ??:??
0x40040001
0x400f5ff7
esp_wifi::compat::common::receive_queued
    at .cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-wifi-0.13.0/src/compat/common.rs:381
0x40108666
btdm_controller_task
    at ??:??

Is there anything I can do to further debug this issue?

Let me summarize what I am doing to draw a better picture: I have three embassy tasks: one does a simple BLE, one reads large data from the gps module using i2c, and a third task that drives a small oled screen on the same i2c bus. I eventually want to enable all three but when BLE and gps tasks are enabled together, it crashes. BLE and oled screen works fine. Two i2c tasks work fine.

[iButcat]

I am using the v1.0.0 beta currently and was facing the same issue with similar error output. However following @bugadani comment, I was able to compile. I am using the esp-wifi with version 0.13.0.

[raphaelhetzel]

I'm actually currently debugging a similar issue.
The Exceptions vary depending on the static memory used.
I'm basically always near the point where the compiler won't link as it is out of ram.
esp-rs/esp-wifi-sys#416 suggests that this may be related to the stack.
Is esp-wifi using the stack section or the embassy-net arena?
Is there a way to increase the reseved stack for the main core?

[bjoernQ]

The main stack is what is left free - i.e. RAM - rwtext - data - bss (including what is reserved for the heap)

But you can place the heap into dram2 which is otherwise not useable (see

esp-hal/examples/src/bin/wifi_coex.rs

Lines 62 to 63 in c8822b4

	// COEX needs more RAM - add some more
	esp_alloc::heap_allocator!(#[link_section = ".dram2_uninit"] size: 64 * 1024);

) - then you can reduce or maybe even remove the heap allocated in the other RAM area

Especially on ESP32 using Bluetooth reduces RAM by 64k (we might be able to reduce that in future but currently it's 64k)

[raphaelhetzel]

Is it possible that this is already used on the S3?

    esp_alloc::heap_allocator!(#[link_section = ".dram2_uninit"] 64 * 1024);
    esp_alloc::heap_allocator!(46 * 1024);

EDIT: The attribute does not work in this way.

crashes:

DEBUG - Unhandled event: StaBeaconTimeout
DEBUG - Unhandled event: HomeChannelChange



Exception occurred 'IllegalInstruction'
Context
PC=0x4212261c       PS=0x00060f10
0x4212261c - supplicant_sta_disconn_handler
    at ??:??
A0=0x820ec680       A1=0x3fc9a7f0       A2=0x000000c8       A3=0x00030103       A4=0x00000001
0x3fc9a7f0 - edgeless_embedded_esp32::__xtensa_lx_rt_main::HEAP
    at ??:??
A5=0x00000000       A6=0x00000000       A7=0x3fcd7950       A8=0x82102e9b       A9=0x3fc9a7d0
0x3fcd7950 - g_cnxMgr
    at ??:??
0x3fc9a7d0 - edgeless_embedded_esp32::__xtensa_lx_rt_main::HEAP
    at ??:??
A10=0x000000c8      A11=0x0000000a      A12=0x00000004      A13=0x3c1778e5      A14=0x00000000
A15=0x00000001
SAR=00000017
EXCCAUSE=0x00000000 EXCVADDR=0x00000000
LBEG=0x4037e4b0     LEND=0x4037e4bc     LCOUNT=0x00000000
0x4037e4b0 - ram_wifi_tx_dig_gain
    at ??:??
0x4037e4bc - ram_wifi_tx_dig_gain
    at ??:??
THREADPTR=0x00000000
SCOMPARE1=0x00000100
BR=0x00000000
ACCLO=0x00000000    ACCHI=0x00000000
M0=0x00000000       M1=0x00000000       M2=0x00000000       M3=0x00000000
F64R_LO=0x3fcef91c  F64R_HI=0x00000001  F64S=0x00000004
FCR=0x00000000      FSR=0x00000000
F0=0x00000000       F1=0x00000000       F2=0x00000000       F3=0x00000000       F4=0x00000000
F5=0x00000000       F6=0x00000000       F7=0x00000000       F8=0x00000000       F9=0x00000000
F10=0x00000000      F11=0x00000000      F12=0x00000000      F13=0x00000000      F14=0x00000000
F15=0x00000000
0x420f192f
mgd_probe_send_timeout_process
    at ??:??
0x420ef2d7
0x420ef4fe
ieee80211_timer_do_process
    at ??:??
0x4037a7cc
ppTask
    at ??:??

while reducing the second call to 32 kB works. (110 without an anotation crashes in the same way).
Disabling the second core also enables the app to work.

It also seems like I can't use the additional memory elsewhere (e.g. for the second core stack (edit: using the bytes I did not use due to using dram2 for the heap) ).

[yanshay]

If you are on S3 I recommend you use PSRAM for allocations, otherwise you'll keep hitting the wall with lack of memory. From experience.

[raphaelhetzel]

Thanks for the Advice. I was planning to use that for the more dynamic part of my application. I don't think it is possible to use that for anything critical (stack, wifi, embassy)...

[yanshay]

Stack and WiFi no, but if you use alloc then most of that can go there and it frees up space for the stack.

I didn't reach the point where I needed to yet, but I looked into that and embassy may be able to use the PSRAM, just need to allocate the task space dynamically.

[raphaelhetzel]

Note about the snipped posted above: @bjoernQ's suggestion is not directly applicable to esp_alloc 0.6 (as I tried above), whose macro just takes a number and not the size parameter. The attribute will be silently ignored.