ESP-prog with esp_tool protocol fails at upload or serial

[OptoStocks]

Hi guys,

I use the esp-prog as serial firmware uploading tool, with esp_tool protocol, on custom boards of mine.
Yesterday, everything was working fine about uploading and monitoring through serial. However I had to reboot my Mac today, and I can't upload or connect to serial since the reboot.

I changed cables, usb ports and esp-progs (I have 3), rebooted my Mac 2 times. Nothing helped.
I checked with a standard dev_board connected with the same usb cable, it uploads flawlessly.
Both VScode and platformio are up-to-date.
The esp-prog itself is still detected by platformio, as it tries to reconnect when I unplug it, and self reconnects when I plug it back (nice feature btw).

I believe that one of the very last versions of platformio got effective with the reboot of VScode, and brought some novelties I wouldn't be aware of, or maybe it's just a bug hidden somewhere.

How to solve this, and if needed, how may I help ?
Thanks

[OptoStocks]

Ok, I did some testings and downgraded platformio to v6.0.2, thanks to the command :
~/.platformio/penv/bin/pip install -U "platformio<6.1.0"
=> Problem appears to be from v6.1.0.

Configuration

Operating system:
macOS Monterey v12.4, or so.

PlatformIO Version (platformio --version):
6.1.0 and 6.1.1

Description of problem

After the v6.1.0 update, eep-prog (programmer tool from Espressif) won't upload any firmware or connect to an ESP32 dev module through serial.
Not it would connect to serial either, for monitoring purposes.

Quick workaround for those who can't wait for next version to solve this.
With platformio downgraded to v6.0.2, I can upload again and connect to Serial.
Use : ~/.platformio/penv/bin/pip install -U "platformio<6.1.0"

Steps to Reproduce

1. Install platformio v6.1.0 or v6.1.1
2. Create a espidf blank project (or blink example or whatever)
3. Upload it with esp-prog dev tool, using serial IDC connector, with esp-tool protocol.

Exemple of "platformio.ini" file content
[env:az-delivery-devkit-v4]
platform = espressif32
board = az-delivery-devkit-v4
framework = espidf
monitor_speed = 115200
upload_speed = 115200
upload_protocol = esptool

Expected result

Looking for upload port...
Auto-detected: /dev/cu.usbserial-14501
Uploading .pio/build/az-delivery-devkit-v4/firmware.bin
esptool.py v3.1
Serial port /dev/cu.usbserial-14501
Connecting....
Chip is ESP32-D0WD (revision 1)
[...]
Leaving...
Hard resetting via RTS pin...
========= [SUCCESS] Took 32.21 seconds =========

• Terminal will be reused by tasks, press any key to close it.

Actual Results

Configuring upload protocol...
AVAILABLE: esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = esptool
Looking for upload port...
Auto-detected: /dev/cu.usbserial-14500
Uploading .pio/build/az-delivery-devkit-v4/firmware.bin
esptool.py v3.1
Serial port /dev/cu.usbserial-14500
Connecting......................................_____

A fatal error occurred: Failed to connect to ESP32: Timed out waiting for packet header
*** [upload] Error 2

[ivankravets]

Serial port /dev/cu.usbserial-14501
Auto-detected: /dev/cu.usbserial-14500

2 different ports were used. Do you have a few boards connected to your machine?

[OptoStocks]

Thank you Ivan for transferring into the right place, sorry for that.

Yes, as I said I tried to upload on 5 different boards, with the 4 ports available, with 3 different esp-prog tools and 3 different cables. All of this with 0% as a success rate.
However, when I tried to upload to a dev board (with it's own usb to serial chip on board) it worked from the first time, flawlessly on the 4 USB ports I have.

Of course, I did the tests with only one board connected at a time, and no other serial consuming software (such as cura for e.g.) was open and running.

But once again, I could upload to dev board without changing anything. And when I went back to v6.0.2, everything went back to normal immediately.

=>Sorry for changing the port used, my mistake I didn't pay attention to it, it was part of my process to ensure there was no HW problem involved.

[ivankravets]

A lot of mixed things. Could you repeat again the testing procedure and back with test results?

1. pio upgrade --dev. Try the latest development version. Only one board connected.
2. pip install "platformio<6.1". Try again with the only one board.

Please ensure that your use PlatformIO Core CLI (terminal icon on the bottom status bar in VSCode)

[OptoStocks]

Sorry Ivan, for not being clear in my explanations.

The "mixed things" were part of my debug process, and this was done before writing the first message here.

1. and 2. are exactly what I did before writing my second message.

But whatever, I just did it all over again.

First : upgrade to latest Dev stage

pio upgrade --dev
Please wait while upgrading PlatformIO ...
PlatformIO has been successfully upgraded to 6.1.4a1
Release notes: https://docs.platformio.org/en/latest/history.html

Upload result :

Building .pio/build/esp32dev/firmware.bin
esptool.py v3.1
Merged 1 ELF section
Configuring upload protocol...
AVAILABLE: esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = esptool
Looking for upload port...
Auto-detected: /dev/cu.usbserial-14500
Uploading .pio/build/esp32dev/firmware.bin
esptool.py v3.1
Serial port /dev/cu.usbserial-14500
Connecting......................................_____

A fatal error occurred: Failed to connect to ESP32: Timed out waiting for packet header
*** [upload] Error 2

Get back to platformio-6.0.2

pip install "platformio<6.1"
Collecting platformio<6.1
Using cached platformio-6.0.2-py3-none-any.whl
Requirement already satisfied: tabulate==0.8.* in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (0.8.10)
Requirement already satisfied: semantic-version==2.10.* in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (2.10.0)
Requirement already satisfied: click<9,>=8.0.4 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (8.1.3)
Requirement already satisfied: zeroconf<1 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (0.38.7)
Requirement already satisfied: marshmallow==3.* in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (3.17.0)
Requirement already satisfied: ajsonrpc==1.* in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (1.2.0)
Requirement already satisfied: pyserial==3.* in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (3.5)
Requirement already satisfied: starlette==0.20.* in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (0.20.4)
Requirement already satisfied: bottle==0.12.* in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (0.12.21)
Requirement already satisfied: colorama in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (0.4.5)
Requirement already satisfied: aiofiles==0.8.* in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (0.8.0)
Requirement already satisfied: requests==2.* in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (2.28.1)
Collecting uvicorn==0.17.*
Using cached uvicorn-0.17.6-py3-none-any.whl (53 kB)
Requirement already satisfied: wsproto==1.1.* in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (1.1.0)
Requirement already satisfied: pyelftools<1,>=0.27 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from platformio<6.1) (0.28)
Requirement already satisfied: packaging>=17.0 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from marshmallow==3.->platformio<6.1) (21.3)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from requests==2.->platformio<6.1) (1.26.9)
Requirement already satisfied: idna<4,>=2.5 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from requests==2.->platformio<6.1) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from requests==2.->platformio<6.1) (2022.6.15)
Requirement already satisfied: charset-normalizer<3,>=2 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from requests==2.->platformio<6.1) (2.1.0)
Requirement already satisfied: typing-extensions>=3.10.0 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from starlette==0.20.->platformio<6.1) (4.2.0)
Requirement already satisfied: anyio<5,>=3.4.0 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from starlette==0.20.->platformio<6.1) (3.6.1)
Requirement already satisfied: h11>=0.8 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from uvicorn==0.17.->platformio<6.1) (0.13.0)
Requirement already satisfied: asgiref>=3.4.0 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from uvicorn==0.17.->platformio<6.1) (3.5.2)
Requirement already satisfied: importlib-metadata in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from click<9,>=8.0.4->platformio<6.1) (4.12.0)
Requirement already satisfied: ifaddr>=0.1.7 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from zeroconf<1->platformio<6.1) (0.2.0)
Requirement already satisfied: sniffio>=1.1 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from anyio<5,>=3.4.0->starlette==0.20.->platformio<6.1) (1.2.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from packaging>=17.0->marshmallow==3.*->platformio<6.1) (2.3.1)
Requirement already satisfied: zipp>=0.5 in /Users/my_name/.platformio/penv/lib/python3.7/site-packages (from importlib-metadata->click<9,>=8.0.4->platformio<6.1) (3.8.0)
Installing collected packages: uvicorn, platformio
Attempting uninstall: uvicorn
Found existing installation: uvicorn 0.18.2
Uninstalling uvicorn-0.18.2:
Successfully uninstalled uvicorn-0.18.2
Attempting uninstall: platformio
Found existing installation: platformio 6.1.4a1
Uninstalling platformio-6.1.4a1:
Successfully uninstalled platformio-6.1.4a1
Successfully installed platformio-6.0.2 uvicorn-0.17.6

Again, Upload results :

---

Obsolete PIO Core v6.0.2 is used (previous was 6.1.4a1)
Please remove multiple PIO Cores from a system:
https://docs.platformio.org/en/latest/core/installation/troubleshooting.html

---

Processing esp32dev (platform: espressif32; board: az-delivery-devkit-v4; framework: espidf)
[...]
Building .pio/build/esp32dev/firmware.bin
esptool.py v3.1
Merged 1 ELF section
Configuring upload protocol...
AVAILABLE: esp-prog, espota, esptool, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa
CURRENT: upload_protocol = esptool
Looking for upload port...
Auto-detected: /dev/cu.usbserial-14501
Uploading .pio/build/esp32dev/firmware.bin
esptool.py v3.1
Serial port /dev/cu.usbserial-14501
Connecting......
Chip is ESP32-D0WD (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 0c:dc:7e:df:34:a8
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00001000 to 0x00007fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Flash will be erased from 0x00010000 to 0x0003afff...
Compressed 25296 bytes to 15417...
Writing at 0x00001000... (100 %)
Wrote 25296 bytes (15417 compressed) at 0x00001000 in 1.8 seconds (effective 115.0 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 103...
Writing at 0x00008000... (100 %)
Wrote 3072 bytes (103 compressed) at 0x00008000 in 0.0 seconds (effective 512.0 kbit/s)...
Hash of data verified.
Compressed 174240 bytes to 90678...
Writing at 0x00010000... (16 %)
Writing at 0x0001b6c8... (33 %)
Writing at 0x00020e4f... (50 %)
Writing at 0x0002715a... (66 %)
Writing at 0x000315cc... (83 %)
Writing at 0x000374e9... (100 %)
Wrote 174240 bytes (90678 compressed) at 0x00010000 in 8.4 seconds (effective 165.6 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
======================================================================= [SUCCESS] Took 106.71 seconds =======================================================================

IMPORTANT

All test were made with the same board, same cable, same port and same esp-prog.
As you can see, port is different again, that may be the core problem.

I confirm you, however, that latest versions won't work at all, whereas 6.0.2 works like a breeze.

[OptoStocks]

From this documentation : https://espressif-docs.readthedocs-hosted.com/projects/espressif-esp-iot-solution/en/latest/hw-reference/ESP-Prog_guide.html

"the PC can identify the board’s downloading and JTAG interfaces (functions) by their port numbers."

"The PC is able to identify the ESP-Prog’s two ports by their port numbers. The bigger port number represents the Program interface, while the other one represents the JTAG interface."

I suspect that the port selection of esp-prog is not correctly done with esp-tool protocol in latest versions.
Try port+1 :)

Hope it helps.

[ivankravets]

Could you provide a remote session to debug this issue? It would be great to fix for other people. Please contact to me@ikravets.com for further details.

[OptoStocks]

Mail sent !

[Sinclair732]

Possibly i had the same "connecting..."-problem with the az-delivery-devkit-v4 board a few days ago.

When i tried the use openocd directly to get a connection to the board and got a message like "the esp-wroom-32.cfg is obsolete". Then i modified the .platformio/platforms/espressif32/boards/az-delivery-devkit-v4.json - file and changed the
openocd_board value from esp-wroom-32.cfg to esp32-wrover-kit-3.3v.cfg. After that, the upload and debug is working for me.

If this is the solution, then please update/modify the az-delivery-devkit-v4.json here in github too!

[ivankravets]

Thanks for the report! Please re-test with pio upgrade --dev.

[OptoStocks]

Hi Ivan,

I just made the test !
Well done, it connects and programs now.
Serial connection and monitoring are working too.

Thank you for your work and time, and thank you for offering the opportunity to give a hand in the process.

@Sinclair732, sorry for not answering earlier : you seem to face a different problem, here was a mis-selection of the port of the Program Interface (applies to esp-prog only).

[ivankravets]

Great! Happy coding with PlatformIO! 🚀