Recovery of soft bricked native USB boards w/o double-tap impossible without port selection

[per1234]

Describe the problem

The primary microcontrollers of many Arduino boards have USB capabilities (e.g., Leonardo, Micro, MKR). These boards often communicate directly between the primary microcontroller and PC rather than having a separate "bridge" interface chip that provides the USB capability needed to communicate with the computer (e.g., Uno, Mega).

Because the USB capabilities are implemented as part of the same program as the user's sketch code on these "native USB" boards, it is possible for the user's code to disable, interfere with, or break the USB communication with the computer. This is a common occurrence that should be expected as part of the experience of developing firmware for this type of board. That may be the expected outcome (e.g., putting the microcontroller to sleep) or an unexpected result of a bug. When this happens, the board no longer produces a USB CDC serial port. This can make it difficult to upload to the board since that port is used in the upload process (e.g., "1200 bps touch").

In order to facilitate the recovery from this "soft bricked" state, some bootloaders have a feature where they run indefinitely on the second of two resets in quick succession (sometimes referred to as "double-tap"). However, not all bootloaders have this capability. On the bootloaders without support for the "double-tap" mode, the bootloader runs only for a short time after a reset before timing out and exiting to the application. In this case it is necessary for the user to time the manual reset so that the upload process occurs while the bootloader is running. For a combined compile and upload operation, this is a bit tricky since the bootloader might time out during the compilation phase if the reset was done before starting the operation.

The standard recovery procedure for these boards, as documented here, is to wait until the tooling indicates the upload phase is starting, then manually reset the board. This worked fine in Arduino IDE 1.x because the "wait for upload port" phase is triggered even when no port is selected. The appearance of the port produced by the bootloader running after the manual reset is detected during that phase and used for the upload. However, that "wait for upload port" phase is skipped entirely if no port is specified in the Arduino CLI upload command:

arduino-cli/commands/upload/upload.go

Lines 381 to 383 in f239754

	if programmer == nil && !burnBootloader && port.Protocol == "serial" {
	// Perform reset via 1200bps touch if requested and wait for upload port also if requested.

This is the case in the upload command produced by Arduino IDE 2.x when no port is selected, as well as in arduino-cli command line commands which do not contain a --port flag.

For users of the command line, the most "correct" approach would be to do a manual reset while arduino-cli board list --watch is running to determine the address of the bootloader port, then specify that port via the --port flag (manually resetting the board during the upload phase as usual to cause the port to be available for the upload).

This is not possible for Arduino IDE 2.x users, leaving them with the workaround of selecting an arbitrary random serial port that happens to be present on their system in order to cause a serial port to be provided to Arduino CLI so that it will run the "wait for upload port" phase during the upload. Some users may not even have an available port to use for the workaround, and even for those who do have a port it is very hacky.

The final recourse would be to do a "Burn Bootloader" operation, not because there is any need to re-flash the bootloader, but because the operation also erases the problematic sketch application from the board. However, that is very complicated for the average user and also requires additional hardware they might not possess.

🐛 It is difficult or impossible for users to recover these boards from the "soft bricked" state.

To reproduce

Equipment

Native USB board that does not have "double-tap" support:

• Leonardo
• Micro
• Yun
• I would guess any of the other boards that use Arduino's "caterina" bootloader, but I haven't checked others.

Steps

1. "Soft brick" your board by uploading this sketch:
   ⚠ This will put the board into a state that is slightly difficult to recover from.

   void setup() {
     noInterrupts();
   }
   void loop() {}

2. Upload any sketch to the board without having a port selected.
3. Press and release the reset button on the board as soon as the upload process starts.

🐛 The upload fails even though the board was in bootloader mode:

Error during Upload: Failed uploading: no upload port provided

Expected behavior

It is possible for users to easily recover their boards from the "soft brick" state by following the standard recovery procedure.

Arduino CLI version

f239754

Operating system

Windows, Ubuntu

Operating system version

Windows 10, Ubuntu 20.04

Additional context

As I mentioned in the introduction, it is already possible for command line users to accomplish a recovery in a fairly straightforward (though lengthy) manner. So this is mostly a problem for Arduino IDE 2.x users (and any other application that works in a similar manner). So it might be that the fix should be made in Arduino IDE 2.x rather than Arduino CLI.

For example, Arduino IDE 2.x could always pass the last selected port to Arduino CLI (it does not do this now), which would allow the standard recovery procedure to be performed with the addition of the following steps at the start:

1. Press the reset button on the board.
2. Quickly select the port from the "Board Selector" or Tools > Port" menu.

I thought it worth first investigating whether there is a way for this to be handled by Arduino CLI without the requirement of a port selection, since that would allow the standard recovery procedure to continue to be used by the users of all modern Arduino development software. I was able to perform the standard recovery procedure after simply removing the && port.Protocol == "serial" from commands/upload/upload.go#L381, but of course that conditional is there because the "1200 bps touch" process is specific to serial ports.

---

This is more relevant now because previously a bug in Arduino IDE 2.x (arduino/arduino-ide#770) caused the Upload operation to not be initiated when no port was selected, so the Upload operation never got to the stage when the "Failed uploading: no upload port provided" error was returned by Arduino CLI during these recovery attempts.

---

Originally reported at https://forum.arduino.cc/t/ide-2-0-wont-program-arduinos-that-need-to-be-put-into-bootloader-mode/1044198

Additional reports

• https://forum.arduino.cc/t/arduino-2-1-0-breaks-stm32-hidbootloader-setup/1123374

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the nightly build
• My report contains all necessary details

[amowry]

I was wondering if this issue will be addressed at some point? In my opinion, the IDE 1.x behavior was much more convenient. It's common to want to program boards without a CDC serial port exposed, for example class-compliant USB input devices that must have the CDC turned off. With IDE 2.0, you have to put the boards into programming mode twice-- once just to select a port, and then again to initiate the upload. With IDE 1.x you could just click upload, and then put the device into programming mode when the reset procedure was triggered.

[brettski74]

This issue also seems to be the problem behind setting up the HIDBootloader on STM32 boards. The initial load of the bootloader onto the board sets the bootloader running and waiting for an upload, but does not create the CDC serial port. This only gets created after the first successful upload, so reloading the bootloader wouldn't fix this problem for STM32 with HIDBootloader. Using some other device to provide a random serial port does work, but not if you don't have some other convenient device to provide the serial port. What is really needed for this first sketch upload after programming the bootloader is a way to not specify a serial port.

[cmaglie]

I've made a patch here: #2173

Here are the builds -> https://github.com/arduino/arduino-cli/actions/runs/4917205342
@per1234 @amowry @brettski74 may you give it a try?

[amowry]

Is it possible to test this with the UI, or do I need to run it from command line? I haven't done that before :).

[cmaglie]

Is it possible to test this with the UI

You may try to replace arduino-cli.exe inside the Arduino IDE subdirectory:

resources/app/node_modules/arduino-ide-extension/build/arduino-cli.exe

with one of the test builds.

[cmaglie]

You may try to replace arduino-cli.exe inside the Arduino IDE subdirectory:

I was wrong, you cannot test it with the IDE because it does not run the upload without a port selected.

After this bug is fixed in the CLI it will require another patch in the IDE to fully support it.