upload firmware via USB after USB.begin()

[savejeff]

Board

ESP32-S3

Device Description

Here is what I'm trying to do, but am not able to accomplish:
I want to upload a Firmware after calling USB.begin() without a complete power cycle restart

I need this, as I have a battery-powered device where I can't access the battery connection to disconnect and reconnect it (due to waterproofness requirement)

The Problem I'm facing is: that as soon as I call USB.begin() to allow SerialUSB communication there is nothing I can do to bring the Processor into a state that allows uploading firmware files.
What I mean is, that when I plug in the ESB32 on the USB Data Port it opens a COM port where I can upload a sketch. As soon as I call USB.begin this COM port goes away and another one comes up. This second one can't be used to upload sketches anymore.
I tried doing a Software restart with ESP.restart(); but this does not reset the USB. I have to completely unplug the ESP32 removing power and replug it, to get this upload port back. Another way is to press the "Reset" Button on the Board

What can I do to bring the USB Data Serial connection back into the state that lets me upload Sketches?
I'm open to any suggestions. Like making it so that I can upload to the second comport that opened after USB.begin() or a way to completely reset the ESP through a power-cycle.

Hardware Configuration

ESP32-S3 DevKit C1

The easiest setup is to have both USB ports (UART and USB) into a PC. This way you can unplug the USB Data port without a power cycle.
It is accomplished what I want when the ESP is brought back into a state where I can upload a sketch via the USB Data port without unplugging the UART USB Port

Version

latest master (checkout manually)

IDE Name

PlatformIO

Operating System

Windows 11

Flash frequency

80Mhz

PSRAM enabled

no

Upload speed

115200

Sketch

based on the USBSerial.ino example

USB Mode 0 is required
PlatformIO:

board_build.extra_flags = 
	-D ARDUINO_USB_MODE=0

Arduino:
Select USB Mode: Hardware CDC and JTAG

void setup() {
	// -> at this point I can upload a sketch ONLY if I did a complete power cycle. not when I restart via sending '#' over Serial
	
	// wait for serial input before starting
	Serial.begin(115200);
	while(!Serial.available()) {
		Serial.print(".");
		delay(1000);
	}

	Serial.setDebugOutput(true);


	USB.onEvent(usbEventCallback);
	USBSerial.onEvent(usbEventCallback);

	USBSerial.begin();
	USB.begin();

	// -> from this point on nothing can be done to allow sketch uploads again. not even a restart
}


void loop() {
	// print "loop" every second to make it more transparent in what state the sketch is
	static uint32_t last_time_print = millis();
	if(millis() - last_time_print > 1000) {
		last_time_print = millis();
		Serial.println("loop");
	}

	while (Serial.available()) {
		char c = Serial.read();
		if(c == '#') { // send # to trigger a restart
			USBSerial.end();
			ESP.restart();
		}
		USBSerial.write(c); // forward Serial input to USBSerial
	}

}

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

Edits

Edit1:
seems not even a Crash through Exception causes a complete reset. i tried

volatile uint32_t a = 12;
volatile uint32_t b = 0;
Log("%d", a / b);

Edit2:
I tried USB.enableDFU(); in the hope it enables firmware upload logic on the second SerialUSB after USB.begin() but it still doesn't start a firmware.

its either giving me
[E][esp32-hal-tinyusb.c:552] tinyusb_load_enabled_interfaces(): Descriptor Load Failed
when I call it before USB.begin(); or
[E][esp32-hal-tinyusb.c:664] tinyusb_enable_interface(): TinyUSB has already started! Interface DFU not enabled
when I call it after USB.begin()

I can see the upload process trying to bring the processor into upload mode but it not triggering anything:

CDC LINE CODING: bit_rate: 9600, data_bits: 8, stop_bits: 0, parity: 0
CDC CONNECTED
CDC LINE STATE: dtr: 1, rts: 1
CDC LINE CODING: bit_rate: 115200, data_bits: 8, stop_bits: 0, parity: 0
CDC LINE STATE: dtr: 1, rts: 0
CDC DISCONNECTED
CDC LINE STATE: dtr: 0, rts: 0
CDC LINE STATE: dtr: 1, rts: 0
CDC CONNECTED
CDC LINE STATE: dtr: 1, rts: 1
CDC DISCONNECTED
CDC LINE STATE: dtr: 0, rts: 0
CDC RX [46]:\xC0 \x00 \x08 $\x00 \x00 \x00 \x00 \x00 \x07 \x07 \x12  UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU\xC0 

Edit3:

I think I got something.
with usb_persist_restart(RESTART_BOOTLOADER_DFU); you can trigger the restart to boot mode somehow, but as the usbEventCallback is overwritten in the sketch the data is going nowhere or something

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0xc (RTC_SW_CPU_RST),boot:0x23 (DOWNLOAD(USB/UART0))
Saved PC:0x403773c4
waiting for download

and this is what it looks like when its triggered correctly when USB.begin() is not called

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x23 (DOWNLOAD(USB/UART0))
Saved PC:0x4202ae06
waiting for download

with usb_persist_restart(RESTART_BOOTLOADER); I'm getting even closer but then the USB goes away when the upload starts
here the code of usb_persist_restart

void usb_persist_restart(restart_type_t mode)
{
    if (mode < RESTART_TYPE_MAX && esp_register_shutdown_handler(usb_persist_shutdown_handler) == ESP_OK) {
        usb_persist_mode = mode;
#if CONFIG_IDF_TARGET_ESP32S3
        if (mode == RESTART_BOOTLOADER) {
            usb_switch_to_cdc_jtag(); // <- this seems to be a good location to look for solutions
        }
#endif
        esp_restart();
    }
}

okay I Found a first way (but its hacky):

i made static void usb_switch_to_cdc_jtag() not static and exposed in in the header

esp32-hal-tinyusb.h:
void usb_switch_to_cdc_jtag();

now when usb_switch_to_cdc_jtag() is called it closes the COM Port and resets the USB as i need. now i got the Hardware CRC Com Port back i can upload to.

what would be a good way to access this functionality without changing the esp32 firmware as i did

I copied the complete usb_switch_to_cdc_jtag function out to my project into a .c file. now I can call it to reset the USB state.
This kind of is sufficient for what Im looking for.
Can we expose this function in the USB class.
Like USB.reset() to restore the not initialized state as it is before USB.begin() ?