Migrate to ESPtool 4.x

[marcelstoer]

ESPtool v3 only receives patches critical bugs and we should move to v4.

(Un)fortunately, the internal structure we had to rely upon until now was completely refactored. ESPtool now offers a public API: https://github.com/espressif/esptool/releases/tag/v4.0. That's most welcome (I had asked for this for years) but it also means a bit of a migration effort I guess.

[DeeEmm]

The code is already working on V4. I suspect that V4 is backwards compatible with V3

[marcelstoer]

Yes, it does still work but using the API is a more safe approach because it promises to be more stable long-term.

As you can see in the output ("Command: esptool.py --chip...") this app currently simulates being a CLI client for esptool.py. Hence, whenever they change a parameter/flag we depend upon, this app breaks. Worse, we only find out once we start the flashing process. If we were to use the API we would find out when we build the app.

[DeeEmm]

API integration looks like a fairly heavy duty rewrite. Also means replicating a lot of integrated checks into the external code. CLI makes much more sense IMHO. Keep it as a wrapper so that all of the heavy lifting remains in esptool.py

Why not just check the version in esptool/esptool /init.py when building. Would achieve the same.

[marcelstoer]

API integration looks like a fairly heavy duty rewrite.

Yep, that's the main reason this issue is still open.

Why not just check the version in esptool/esptool /init.py when building.

Well, it's not that we get an completely random esptool version. It's somewhat controlled through requirements.txt. What we have in the current master isn't ideal, though. The upcoming release will pin a specific version. Still, any version change will require carefully studying the esptool release notes to look for changes with the CLI flags.

[DeeEmm]

I will take a look on the fork I made, but it is only a cut down version of your code so possibly does not take advantage of all features. If I find anything I will post back.

I did up-rev the requirements.txt to >= V4 and it compiles okay and uploads as it should, but as I mentioned. I did cut back on the code a bit to tailor it for my project so it may not use all flags

[radimkarnis]

@marcelstoer

Still, any version change will require carefully studying the esptool release notes to look for changes with the CLI flags.

As long as you choose any of the v4.* version available, you can rest assured the CLI won't change (IIRC, it didn't even change between v3 and v4 - that's probably why it still seems to work with PyFlasher). Esptool follows semantic versioning, meaning we do our best to guarantee compatibility during minor releases. v5 is now in works, which will probably introduce some breaking changes, but we try to keep them minimal.

[DeeEmm]

I managed to get it working on esptool v4.8.1

Command: esptool.py --chip esp32 --port /dev/cu.usbserial-14410 --baud 460800 --after hard_reset write_flash --flash_size detect --flash_mode dio 0x0000 /Users/mick/Documents/ESP32 Projects/DIY-Flow-Bench/ESP32/DIY-Flow-Bench/release/V2.0-RC8_2501280085_install.bin

esptool.py v4.8.1
Serial port /dev/cu.usbserial-14410
Connecting.....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting.....
Detecting chip type... ESP32
Chip is ESP32-D0WD-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 1c:69:20:ce:88:e4
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00000000 to 0x003fffff...
Compressed 4194304 bytes to 809686...
Wrote 4194304 bytes (809686 compressed) at 0x00000000 in 27.7 seconds (effective 1213.2 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

This is my requirements.txt

esptool>=4.0    
pyserial>=3.5
wxPython>=4.2.2
PyInstaller>=6.11.1
httplib2>=0.22.0
pyinstaller-versionfile>=2.0.0

The main changes were to add the stubs folders to the 'datas' section in the specs file...

build-on-mac.spec

# -*- mode: python -*-

block_cipher = None

added_files = [
   ( "/Users/mick/Documents/ESP32 Projects/DIYFB-firmware-flasher/.venv/lib/python3.13/site-packages/esptool/targets/stub_flasher/1", "./esptool/targets/stub_flasher/1"),
   ( "/Users/mick/Documents/ESP32 Projects/DIYFB-firmware-flasher/.venv/lib/python3.13/site-packages/esptool/targets/stub_flasher/2", "./esptool/targets/stub_flasher/2")
   ]

imported_files = [
   ("wxpython.py"),
   ("pyserial")
]


a = Analysis(['DIYFB-Firmware-Flasher.py'],
             binaries=None,
             datas=added_files,
             hiddenimports=imported_files,
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          name='DIYFB-Firmware-Flasher',
          debug=True,
          strip=False,
          upx=True,
          console=False , icon='images/icon-256.icns')
app = BUNDLE(exe,
             name='DIYFB-Firmware-Flasher.app',
             version='1.0.1',
             icon='./images/icon-256.icns',
             bundle_identifier='com.DeeEmm.DIYFB-Firmware-Flasher')

My Repo - https://github.com/DeeEmm/DIYFB-firmware-flasher

I cannot create a PR for these changes as I'm running a cut down version of your code and I very much doubt you would want to pull in the rest of my changes. My file paths are also hard coded & relative to my VENV.

However. You should be able to easily make the same changes here

[marcelstoer]

Thanks a bunch!

My file paths are also hard coded & relative to my VENV.

Yep, I guess you need to put measures in place that remind you to update the path the next time you bump Python in the VENV 😜

As the .spec files may contain (almost) arbitrary Python code AFAIR, my plan is to write some code that introspects the VENV folder to determine the actual version (to then build the path on the fly).

[marcelstoer]

my plan is to write some code that introspects the VENV folder to determine the actual version

FYI I have such common code in 63420db but I had to rework it later. I forgot how different the VENV dir structure in Windows is from macOS. It's .venv/lib/pythonNNN/site-packages vs. .venv/Lib/site-packages.