ESP32 JTAG debug is unstable

[bvernoux]

I have tested lot of things to debug ESP32 over JTAG (like debugging ROM code or my own application/sample)

Note: I have checked without commit 3098897 and it is the same ...

First problem is when trying to debug for example a very simple reproducible mode "Download mode" it seems JTAG cannot "take the hand"

Steps by step way to reproduce it:

• I'm using ESP32-DevKitC (from Olimex ESP32-CoreBoard V2) and JTAG Debugger FTDI 232HM DDHSL-0 as see connection here

• Connect ESP32-DevKitC USB to PC with "Boot" button pressed (to enter Download Mode)

• Check with a Terminal@115200 8N1 all is ok

  ets Jun  8 2016 00:22:57
  
  rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2))
  waiting for download
  

• Connect JTAG Debugger USB to PC (in my example FTDI C232HM DDHSL-0)

  • The terminal shall not change anything and it shall be always on waiting for download ...

• Build & configure ESP32 openocd https://github.com/espressif/openocd-esp32 (i'm using latest version from trunk on 30 Oct 2016)

  • I have built & intstalled openocd ESP32 with success (with Xubuntu 14.04 LTS)

• Start openocd

  • I'm using this esp32.cfg script in openocd directory esp32_cfg.zip

cd ~/esp/openocd-esp32
xubuntu@xubuntu-VirtualBox:~/esp/openocd-esp32$ ./src/openocd -s ./tcl -f ./esp32.cfg
Open On-Chip Debugger 0.10.0-dev-g3098897-dirty (2016-10-28-00:34)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
adapter speed: 8000 kHz
force hard breakpoints
Info : clock speed 8000 kHz
Info : JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : esp32.cpu0: Debug controller was reset (pwrstat=0x5F, after clear 0x0F).
Info : esp32.cpu0: Core was reset (pwrstat=0x5F, after clear 0x0F).
Info : esp32.cpu0: Target halted, pc=0x00000000
esp32.cpu0: target state: halted
Info : esp32.cpu0: Core was reset (pwrstat=0x5F, after clear 0x0F).
Error: Exception reading windowbase!

Info : esp32.cpu0: Target halted, pc=0x00000000
esp32.cpu0: target state: halted
Info : esp32.cpu0: Core was reset (pwrstat=0x1F, after clear 0x0F).
Info : esp32.cpu0: Core was reset (pwrstat=0x3F, after clear 0x40).
Info : esp32.cpu0: Core was reset (pwrstat=0x3F, after clear 0x40).
Info : esp32.cpu0: Debug controller was reset (pwrstat=0x40, after clear 0x60).
Info : esp32.cpu0: Core was reset (pwrstat=0x3F, after clear 0x40).
Info : esp32.cpu0: Core was reset (pwrstat=0x5F, after clear 0x0F).
Info : esp32.cpu0: Core was reset (pwrstat=0x3F, after clear 0x40).
Info : esp32.cpu0: Core was reset (pwrstat=0x3F, after clear 0x40).
Info : esp32.cpu0: Core was reset (pwrstat=0x5F, after clear 0x0F).
Error: esp32.cpu0: esp108_fetch_all_regs (line 649): DSR (00730007) indicates target still busy!
Error: esp32.cpu0: esp108_fetch_all_regs (line 649): DSR (00730007) indicates DIR instruction generated an exception!
Error: esp32.cpu0: esp108_fetch_all_regs (line 673): DSR (00730007) indicates target still busy!
Error: esp32.cpu0: esp108_fetch_all_regs (line 673): DSR (00730007) indicates DIR instruction generated an exception!
Error: Exception reading pc!

Info : esp32.cpu0: Target halted, pc=0x00000000
Error: esp32.cpu0: xtensa_write_memory (line 1024): DSR (00010002) indicates DIR instruction generated an exception!
Warn : esp32.cpu0: Failed writing 4 bytes at address 0x3FF5F064
embedded:startup.tcl:21: Error: 
in procedure 'esp_core_halt' 
in procedure 'esp32.cpu0' called at file "/usr/local/share/openocd/scripts/target/esp32.cfg", line 37
in procedure 'ocd_bouncer' 
at file "embedded:startup.tcl", line 21

esp32.cpu0: target state: halted
Error: esp32.cpu0: esp108_fetch_all_regs (line 649): DSR (00730007) indicates target still busy!
Error: esp32.cpu0: esp108_fetch_all_regs (line 649): DSR (00730007) indicates DIR instruction generated an exception!
Error: esp32.cpu0: esp108_fetch_all_regs (line 673): DSR (00730007) indicates target still busy!
Error: esp32.cpu0: esp108_fetch_all_regs (line 673): DSR (00730007) indicates DIR instruction generated an exception!
Error: Exception reading pc!

Info : esp32.cpu0: Target halted, pc=0x00000000
Error: esp32.cpu0: xtensa_write_memory (line 1024): DSR (00010007) indicates target still busy!
Error: esp32.cpu0: xtensa_write_memory (line 1024): DSR (00010007) indicates DIR instruction generated an exception!
Warn : esp32.cpu0: Failed writing 4 bytes at address 0x3FF5F064
embedded:startup.tcl:21: Error: 
in procedure 'esp_core_halt' 
in procedure 'esp32.cpu0' called at file "/usr/local/share/openocd/scripts/target/esp32.cfg", line 37
in procedure 'ocd_bouncer' 
at file "embedded:startup.tcl", line 21

esp32.cpu0: target state: halted
Info : esp32.cpu0: Core was reset (pwrstat=0x3F, after clear 0x40).
Info : esp32.cpu0: Core was reset (pwrstat=0x3F, after clear 0x40).
Info : esp32.cpu0: Core was reset (pwrstat=0x3F, after clear 0x00).
Info : esp32.cpu0: Debug controller was reset (pwrstat=0x40, after clear 0x60).
Info : esp32.cpu0: Core was reset (pwrstat=0x3F, after clear 0x00).
Info : esp32.cpu0: Core was reset (pwrstat=0x3F, after clear 0x00).
Info : esp32.cpu0: Debug controller was reset (pwrstat=0x40, after clear 0x60).
Info : esp32.cpu0: Core was reset (pwrstat=0x3F, after clear 0x00).
Info : esp32.cpu0: Debug controller was reset (pwrstat=0x40, after clear 0x60).

Like you can see the ESP32 crash often and reboot I can see it clearly in the Terminal with output:

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2))
waiting for download
ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2))
waiting for download
ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2))
waiting for download
ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2))
waiting for download
ets Jun  8 2016 00:22:57
...

I think the main issue is probably watchdog on ESP32 which is not refreshed anymore (because JTAG take the hand and do not stop the watchdog too early ...) and so it reset ...

The questions are:

1. How can we take the hand on ESP32 core with JTAG and stop the cores correctly without having tons of reset ... ?
2. How to reset correctly the ESP32 with JTAG, set hw breakpoint (at ROM boot code at ResetHandler or at _start or even rom_main) and do a continue with GDB (then it stop by itself on the breakpoint) ?

Boot of ESP32 (ROM):
_ResetVector Start of execution (0x40000400) => Jump to _ResetHandler 0x40000450
_ResetHandler 0x40000450 => Call _start 0x40000704
_start 0x40000704 => call rom_main 0x400076c4
rom_main 0x400076c4
...

3. Do we have a way to fully reset the ESP32 cores + peripherals (and clean everything like exceptions ...) through JTAG ?

• If it is not supported by actual silicon hardware do you plan to support that in a future silicon revision ?

4. How to debug correctly with JTAG ROM code or User App (with or without elf/symbols) in a reproducible way (as sometimes it works ...) ?

[Spritetm]

Sorry, for now we do not support debugging code before esp-idf kicks in. If you have similar problems debugging problems after something compiled with esp-idf has started and got past the initial breakpoint, feel free to open another issue.