mcrun hangs on esp32-c3 (linux host)

[tve]

Build environment: Linux
Target device: Lolin C3 Pico (very similar to Adafruit qtpyc3, the differences don't matter here)

Description
Tooling / debugging hangs when using a host + mods

Steps to Reproduce

1. Erase flash using esptool -d /dev/ttyACM0 erase_flash
2. Build, load, and launch ReproHost using mcconfig -d -m -p esp32/qtpyc3
3. Observe that xsbug launches and shows

No Wi-Fi SSID


===== ReproHost starting =====
Host ready, no module installed...
Alive...
Alive...

(the Alive... is printed every 10 seconds)
4. Build and attempt to load ReproApp using mcrun -d -m -p esp32/qtpyc3
5. Observe that xsbug uploads the mod and restarts the esp32, but then hangs; it cannot be aborted using ctrl-C
The output from a build with tracing enabled:

### fxRestart
ESP-ROM:esp32c3-api1-20210207
Build:Feb  7 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0xf (SPI_FAST_FLASH_BOOT)
Saved PC:0x420b3a28
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd5810,len:0x48
load:0x403cc710,len:0x668
load:0x403ce710,len:0x2258
entry 0x403cc710
### set time
### install
### fxCommandReceived: remote operation id 0xff0f SUCCESS with resultCode 0
### installSpace: 262144, fragment size 4096
### install fragment @ 0 size 4
### fxCommandReceived: remote operation id 0xe0e0 SUCCESS with resultCode 0
### install fragment @ 4 size 996
### fxCommandReceived: remote operation id 0xe0e0 SUCCESS with resultCode 0
### update install header
### install fragment @ 4 size 4
### fxCommandReceived: remote operation id 0xe8e8 SUCCESS with resultCode 0
### install complete
..complete
### fxRestart
ESP-ROM:esp32c3-api1-20210207
Build:Feb  7 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0xf (SPI_FAST_FLASH_BOOT)
Saved PC:0x403881b0
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd5810,len:0x48
load:0x403cc710,len:0x668
load:0x403ce710,len:0x2258
entry 0x403cc710
### set time

6. Observe that xsbug got woken up but there's no output there.
7. Kill mcrun (ctrl-z, kill -9 %1)
8. Re-load the host (same mcconfig cmdline as above) and observe that it happily connects to xsbug and runs the mod.

Host

manifest.json:

{
  "include": ["$(MODDABLE)/examples/manifest_base.json"],
  "modules": {
    "*": ["./main", "$(MODULES)/base/modules/*"]
  },
  "defines": {
    "XS_MODS": 1
  },
  "preload": ["modules"],
  "strip": [],
  "build": {
    "name": "ReproHost"
  }
}

main.js

import Timer from "timer"

Timer.repeat(() => trace("Alive...\n"), 10000)

import Modules from "modules"

function launch_app() {
  if (Modules.has("check")) {
    let check = Modules.importNow("check")
    check()
    trace("Check returned\n")
    if (!Modules.has("app")) throw new Error("Module 'app' not found")
    let app = Modules.importNow("app")
    trace("===== Launching app =====\n")
    app()
    trace("App is done\n")
  } else {
    trace("Host ready, no module installed...\n")
  }
}

export default function () {
  trace("\n\n===== ReproHost starting =====\n")
  Timer.delay(1000)

  launch_app()
}

Mod

manifest.json

{
  "include": ["$(MODDABLE)/examples/manifest_mod.json"],
  "modules": {
    "*": ["./*.js"]
  }
}

app.js

import Timer from "timer"

trace("Hello app module!\n")

let hello = 1

export default function () {
  trace("Hello app main!\n")
  Timer.repeat(() => trace(`Hello #${hello++}\n`), 1000)
  trace("Done with app main\n")
}

[mkellner]

Hello @tve, I've been able to reproduce this.

As you note in step 8 above, the host accepts the mod and runs fine after you kill mcrun and reattach to xsbug.

In step #8 above, instead of doing a reinstall, you could use the xsbug target:

mcconfig -d -m -p esp32/qtpyc3 -t xsbug

or launch serial2xsbug directly:

serial2xsbug /dev/ttyACM0 115200 8N1

[tve]

Thanks for testing and the workarounds! Unfortunately, running serial2xsbug manually doesn't help for me. Yes, the mod runs but the process hangs the same again and the xsbug connection thus doesn't work. The -t xsbug target behaves the same.

[phoddie]

@tve - does your board have a reset button? If so, I'd be interested to know what happens if you hit that after you into the stalled state with mcrun (step 5 above).

[tve]

Pressing reset has no visible effect. Not in the mcrun terminal nor in xsbug.

[phoddie]

Thanks for trying. We've been looking into this further here. There is at least one strange behavior (bug) in the ESP32-C3 around USB that is complicating this. Hope to have some news in the coming days.

[tve]

Has there been any update on this? What is the "strange behavior (bug)"? I'm kind'a stuck on this at the moment... Thanks!

[phoddie]

@mkellner?

[mkellner]

@tve, I'm sorry I don't have a good answer for you yet. We have been working on it - the issue combines multiple devices (esp32, nrf52, etc.), access methods (serial, usb and uart) and hosts (mac, lin, win).

The issue I'm currently seeing for the esp32/lolin_c3pico (and other usb-cdc-jtag devices) is that the device doesn't appear to read from the usb_serial_jtag driver after the rts/dtr reset.
I hope to have something for you to try soon - maybe end of week(?)

[tve]

Q: can you reproduce the issue reliably? If so, would it help if you describe to me how to repro and what you're observing? I would give it a spin myself...

[mkellner]

I can reproduce the issue reliably.
I am working through my test matrix. I will have something for you to try tomorrow.

[mkellner]

@tve, I'm working through windows right now, but I've got a patch that should work for you on Linux or macOS.

If you're willing to try, checkout the mcrun branch and apply (patch -p1 < 231117.patch) this patch.

231117.patch

[mkellner]

I worked through esp32/lolin_c3mini and esp32/lolin_s2mini (and moddable_two). I'll eventually need to make a pass through the set to ensure they all operate. If you have problems, let me know. There are suggestive prompts in the output on when to press reset, etc. The s2 is a little picky about the reset.

[tve]

I rebased my fork on your branch and applied the patch. Good news is that I managed to load/run a mod a couple of times. Bad news is that the third or fourth time it hung and I couldn't ctrl-C:

# xsl x1c-heater.xsa
ESP-ROM:esp32c3-api1-20210207
Build:Feb  7 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0xd (SPI_FAST_FLASH_BOOT)
Saved PC:0x40386c1e
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd5820,len:0x48
load:0x403cc710,len:0x694
load:0x403ce710,len:0x24c4
entry 0x403cc710
START USB CONSOLE!!!
Installing mod.^C^C^C^C^C



^Zfish: Job 1, 'mcrun -dl -m -p esp32/lolin_c3_…' has stopped
Took 6.6min

Killing the stuck processes and re-running mcrun resulted in the same. Power cycling the board (and killing the processes again) made it work again.

Update: I turned on mxTrace and mxTraceCommands and here it happened again. What you see is the last trace from the previous mcrun and then the new mcrun (it gets restarted by nodemon when I save a source file):

<?xs.3FC9E6CC?>                                                                 
<xsbug><samples>0,0,0,0,1,5,0,0,0,0,0,148516,0,21048,32768,31360,32752,208,6144,
0,50,0,0,0,0</samples></xsbug>                                                  
ESP-ROM:esp32c3-api1-20210207                                                   
Build:Feb  7 2021                                                               
rst:0x15 (USB_UART_CHIP_RESET),boot:0xd (SPI_FAST_FLASH_BOOT)                   
Saved PC:0x40386c1e                                                             
SPIWP:0xee                                                                      
mode:DIO, clock div:1                                                           
### fxRestart                                                                   
                                                                                
<?xs#00000000?>### fxRestart                                                    
                                                                                
<?xs#00000000?>ESP-ROM:esp32c3-api1-20210207                                    
Build:Feb  7 2021                                                               
rst:0x15 (USB_UART_CHIP_RESET),boot:0xd (SPI_FAST_FLASH_BOOT)                   
Saved PC:0x40386c1e                                                             
SPIWP:0xee                                                                      
mode:DIO, clock div:1                                                           
load:0x3fcd5820,len:0x48                                                        
load:0x3fcd5820,len:0x48                                                        
load:0x403cc710,len:0x694                                                       
load:0x403cc710,len:0x694                                                       
load:0x403ce710,len:0x24c4                                                      
load:0x403ce710,len:0x24c4                                                      
entry 0x403cc710                                                                
entry 0x403cc710                                                                
START USB CONSOLE!!!                                                            
START USB CONSOLE!!!                                                            
<?xs-00000000?>                                                                 
                                                                                
<?xs.3FC9E6CC?>                                                                 
<xsbug><login name="main" value="XS 14.2.0 32-bit LE 16-bit ID" flags="E"/></xsb
ug>                                     
### set time                            
                                        
<?xs#3FC9E6CC?>### install
                    
<?xs#3FC9E6CC?>
<?xs-3FC9E6CC?>

If I disconnect the USB cable when it's hung nothing happens. Serial2xsbug should get signals/errors that the serial device is gone and that should make it react somehow, no?

[tve]

One issue I'm also having is that WaitForNewSerialLinux (used by mcconfig) does not work for me. It just sits there printing dots waiting for the serial device to disappear. Not sure what would make it go away...

[mkellner]

waitForNewSerialLinux is a tool to watch for a USB device with VENDOR_ID:PRODUCT_ID attributes.

If the first parameter is set, and the device is already mounted, we will wait for it to disappear and reappear. That is, we are waiting for the reset of the device which is indicated by the /dev/ttyACM0 device (for example) disappearing and reappearing.

I've had good luck with the lolin_c3mini, using the examples from the iot-product-dev-book specifically some http and https mods in ch3-network and ch5-files preferences and flash-readwrite.

I've also found that I only need to press the RST button, and not the "9" button to get the host firmware installed.

[tve]

If the first parameter is set, and the device is already mounted, we will wait for it to disappear and reappear. That is, we are waiting for the reset of the device which is indicated by the /dev/ttyACM0 device (for example) disappearing and reappearing.

This is not working for me. Why do you need to do this? In the end you just call esptool.py and it deals with the flashing, no? Also, I really do not want to have to press a button (it's not prompting for that, so dunno whether I'm expected to).

[phoddie]

In the end you just call esptool.py and it deals with the flashing, no?

For a mod? No. It is installed through the debugger connection by serial2xsbug.

[tve]

In the end you just call esptool.py and it deals with the flashing, no?

For a mod? No. It is installed through the debugger connection by serial2xsbug.

My comment was about mcconfig. It's a bit off-topic, but the issue appeared with this branch&patch (unless it was there before and I had already commented out the whole reset thing, I don't remember).