Deprecation of Python 2. Solution

[Martin-Laclaustra]

Hardware:

Board: ESP32 Dev Module
Core Installation version: 1.0.4
IDE name: Arduino IDE
Flash Frequency: 80Mhz
PSRAM enabled: no
Upload Speed: 460800
Computer OS: Linux Mint 20.1 Ulyssa (base: Ubuntu 20.04 focal)

Description:

Due to deprecation of Python 2 some Linux distributions have decided to remove the generic "python" executable from the path, forcing all code to explicitly declare which version of Python they require (A wise solution in my opinion, as it serves as "canary" for undefined code and prevents obscure errors, resolving the longstanding ambiguity, and still allows the use of older python 2 code). This change has been annoying when one's mind is in other tasks, but one must adapt to new times and deprecation was announced years ago.
Because of this, installing the ESP32 boards in Arduino IDE results in a compilation error and uploading errors. The first error that appears is:

exec: "python": executable file not found in $PATH
Error compiling for board ESP32 Dev Module.

This issue has been raised before, and dismissed in this project:
#3696
#4017

...while it has been addressed early in the independent project "Arduino core for ESP8266":
esp8266/Arduino#6378

Solution:

It seems reasonable to stop forcing users to modify system-wide python setting when the solution seems relatively easy (while I did not weight in possible adaptations needed in other platforms).
I can confirm that the following modification in the "1.0.4" release files fixes the problem.
Files:

esp32/hardware/esp32/1.0.4/tools/espota.py
esp32/hardware/esp32/1.0.4/tools/esptool.py
esp32/hardware/esp32/1.0.4/tools/gen_esp32part.py
esp32/tools/esptool_py/2.6.1/esptool.py

only need to change the first line to: #!/usr/bin/env python3
In file: esp32/hardware/esp32/1.0.4/platform.txt, 4 lines need to be changed, just adding a "3":

-tools.esptool_py.network_cmd=python "{runtime.platform.path}/tools/espota.py"
+tools.esptool_py.network_cmd=python3 "{runtime.platform.path}/tools/espota.py"
-tools.gen_esp32part.cmd=python "{runtime.platform.path}/tools/gen_esp32part.py"
+tools.gen_esp32part.cmd=python3 "{runtime.platform.path}/tools/gen_esp32part.py"
-recipe.objcopy.hex.pattern.linux=python "{tools.esptool_py.path}/{tools.esptool_py.cmd}" --chip esp32 elf2image --flash_mode "{build.flash_mode}" --flash_freq "{build.flash_freq}" --flash_size "{build.flash_size}" -o "{build.path}/{build.project_name}.bin" "{build.path}/{build.project_name}.elf"
+recipe.objcopy.hex.pattern.linux=python3 "{tools.esptool_py.path}/{tools.esptool_py.cmd}" --chip esp32 elf2image --flash_mode "{build.flash_mode}" --flash_freq "{build.flash_freq}" --flash_size "{build.flash_size}" -o "{build.path}/{build.project_name}.bin" "{build.path}/{build.project_name}.elf"
-tools.esptool_py.upload.pattern.linux=python "{path}/{cmd}" --chip esp32 --port "{serial.port}" --baud {upload.speed}  --before default_reset --after hard_reset write_flash -z --flash_mode {build.flash_mode} --flash_freq {build.flash_freq} --flash_size detect 0xe000 "{runtime.platform.path}/tools/partitions/boot_app0.bin" 0x1000 "{runtime.platform.path}/tools/sdk/bin/bootloader_{build.boot}_{build.flash_freq}.bin" 0x10000 "{build.path}/{build.project_name}.bin" 0x8000 "{build.path}/{build.project_name}.partitions.bin"
+tools.esptool_py.upload.pattern.linux=python3 "{path}/{cmd}" --chip esp32 --port "{serial.port}" --baud {upload.speed}  --before default_reset --after hard_reset write_flash -z --flash_mode {build.flash_mode} --flash_freq {build.flash_freq} --flash_size detect 0xe000 "{runtime.platform.path}/tools/partitions/boot_app0.bin" 0x1000 "{runtime.platform.path}/tools/sdk/bin/bootloader_{build.boot}_{build.flash_freq}.bin" 0x10000 "{build.path}/{build.project_name}.bin" 0x8000 "{build.path}/{build.project_name}.partitions.bin"

That was enough. However, I do not know if files "sdkconfig" and "sdkconfig.h" would need a change in the lines referring to the flag CONFIG_PYTHON="python", adding the "3".

There is still one system-wide dependency to meet, requiring sudo apt install python3-serial. I believe that they solved it with a sort of "local copy of the code" in the "Arduino core for ESP8266" project but I do not know the details. May be @earlephilhower could provide some advice on how they did it.

I hope that this report is useful and that you decide to uptake the proposed changes.

[me-no-dev]

esptool.py and gen_esp32part.py are copied from upstream (ESP-IDF and esptool repositories). Could you please raise the issue there?

[me-no-dev]

As for platform.txt, maybe we can add a separate command just for Linux?

[chegewara]

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/linux-setup.html#setting-up-python-3-as-default-for-ubuntu-and-debian

[Martin-Laclaustra]

Earle was kind to share with me his thoughts about this. Most of the following I am quoting him:

"On the 8266 we actually include a python3 portable interpreter for WIN and MACOS and it makes things WAY easier (lets us not need BAT or BASH scripts...all OSes do work in .PYs).

On Windows, it's hard to get users to reliably install anything, and there are many different versions (python from the MS store, from cygwin, from python.org, etc.) On Mac, it's even worse as the OS used to include python2, now does not include anything, and the python.org and homebrew installers end up kind of incompatible and not always in the same path or in PATH.

So in our package.json there is a python3 tool for Linux, Win, and Mac. (Linux is just a shell script that uses #!/usr/bin/env python3).
https://github.com/esp8266/Arduino/blob/master/package/package_esp8266com_index.template.json

Then, we simply clone the pyserial and esptool repos as part of our core (i.e. no compiles, it's an actual git submodule). To ensure the proper pyserial is included, we have an upload.py wrapper which includes the distributed pyserial path as a library dir and then calls the esptool.py object to do the actual upload.

https://github.com/esp8266/Arduino/tree/master/tools
https://github.com/esp8266/Arduino/blob/master/tools/upload.py

Doing it this way ensures that even on Linux the proper pyserial and esptool version will be used."

He also shared with me that you use a Python compiler to build the esptool.exes. So this might be a different approach, but you may come up with a solution inspired from this.
As you can see, it seems that modern MacOS has also ditched "python" executable in favor of the non-ambiguous "python3".

I opened also upstream issues with respect to the relevant python scripts.

Best regards.

[dobairoland]

To the best of my knowledge, the best solution to solve exec: "python": executable file not found in $PATH is the one linked by @chegewara. The operating system lets you setup which version of your "python" should be and offers no default in the beginning.

[dobairoland]

Here are the official notes: https://wiki.ubuntu.com/FocalFossa/ReleaseNotes#Python3_by_default. Please note the compatibility issues. Installing python-is-python2 or python-is-python3 seems to be also a nice solution. I haven't tried it yet.

[Martin-Laclaustra]

I agree that selecting a default python is a solution. However, that seems temporal and partial. Temporal because it "pushes" the problem forward, that of the ambiguity and that of deprecation. I myself consider the "python" affair (systems dependence on a certain version and set of packages, duality of the language...) something due for improvement since long. The only solution is that everyone uptakes the decision to move forward at certain point. Partial because choosing a default version system-wide means that all programs that are going to need compatibility will be of the same version (aka 3) while it is probably that "older" software still needing support are version 2.
All in all, I do no particularly support or criticize the linux distribution decision. I have no word on it. It is what it is, and adaptation and setting the sight in the future is what I feel is needed on my side. That is why I was bringing the question here, to put it on the table and suggesting future planning on it.
Thank you very much for paying me attention and I hope that the final outcome (in the near or far future) means a step forward for all.
(edit) Note: Similar projects like ESP8266-Arduino have opted to set themselves as "invisible" as possible to the user with the adaptations mentioned above.

[dobairoland]

I agree with these @Martin-Laclaustra. I just don't like hardcoding "python3" into all source files. Similarly, if one has a large set of makefiles then won't hardcode "gcc-10.2.0" everywhere but set it globally like CC=gcc-10.2.0 make.

I hope to find a better solution. We already made a breakthrough on the Windows platform where the next installer will unzip an embedded Python which is used only by ESP-IDF and is not registered in Windows. I hope for something similar for Unix platforms in the near future.

[AlfredoCubitos]

@Martin-Laclaustra 's solution doesn't work 100% for me.
I think the real problem is that some python scripts are called by the interpreter e.g. python esptool.py.
This means that changing #!/usr/bin/env python to #!/usr/bin/env python3 doesn't solve the problem in this case.
I think directly calling an executable esptool.py is a better solution.
You can install the esptool.py as a standalone application. Calling the standalone version always works.
This way you could always use a system wide python environment variable for all python scripts if they are executable.

On Linux you can set an alias alias python=python3 This will always call python3
You want to use python 2 call unalias python

Keep in mind the alias has to be set in the environment of the compiler.

[sej7278]

just changing platform.txt seems to work for me on debian, no need to set a default "python" and break your OS, and no need to edit the scripts as the shebang is not used:

tools.esptool_py.network_cmd=python3 "{runtime.platform.path}/tools/espota.py"
tools.gen_esp32part.cmd=python3 "{runtime.platform.path}/tools/gen_esp32part.py"
recipe.objcopy.hex.pattern.linux=python3 "{tools.esptool_py.path}/{tools.esptool_py.cmd}" ....
tools.esptool_py.upload.pattern.linux=python3 "{path}/{cmd}" ....

[FilipDominec]

Recently I fresh installed Arduino IDE and got ESP32 support. It has not been fixed yet in the official package. This is a critical issue on newer Ubuntu versions (e.g. Focal Fossa), since there is no more python2-pyserial package in the repository.

Python2 is just obsoleted. As a quick and dirty solution - if necessary - one can make a guaranteed Python3 launcher like this:

Move esptool.py to real_esptool.py; then save the following code into as esptool.py in the same directory:

import subprocess, sys, os
subprocess.call(
        "python3 " +
        os.path.dirname(os.path.abspath(__file__)) +
        "/real_esptool.py   " + 
        " ".join(sys.argv[1:]), 
        shell=True)

Voilà, upload works now!

[stale]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[sej7278]

Yup, Debian Bullseye and RHEL8 have no python2 by default. Its annoying to have to edit the core every time it gets an update, espressif should just change it to not support python2

[stale]

[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.

[Avamander]

Yeah, python2 support should be dropped and the python file should be executed directly to use the correct interpreter.

[AIWintermuteAI]

just changing platform.txt seems to work for me on debian, no need to set a default "python" and break your OS, and no need to edit the scripts as the shebang is not used:

tools.esptool_py.network_cmd=python3 "{runtime.platform.path}/tools/espota.py"
tools.gen_esp32part.cmd=python3 "{runtime.platform.path}/tools/gen_esp32part.py"
recipe.objcopy.hex.pattern.linux=python3 "{tools.esptool_py.path}/{tools.esptool_py.cmd}" ....
tools.esptool_py.upload.pattern.linux=python3 "{path}/{cmd}" ....

That worked. Only in latest version of ESP32 package this line has changed:

recipe.objcopy.bin.pattern.linux=python3 "{tools.esptool_py.path}/{tools.esptool_py.cmd}" {recipe.objcopy.bin.pattern_args}

It still feels a very simple and urgent problem to rectify. @VojtechBartoska

[Martin-Laclaustra]

just changing platform.txt seems to work for me on debian, no need to set a default "python" and break your OS, and no need to edit the scripts as the shebang is not used

I confirm that also for Linux Mint. This can do the trick for any version.

sed -i -e 's/=python /=python3 /g' ~/.arduino15/packages/esp32/hardware/esp32/*/platform.txt

[JudgeBeeb]

Slight tweak for macOS:

sed -i -e 's/=python /=python3 /g' ~/Library/Arduino15/packages/esp32/hardware/esp32/*/platform.txt

Works for me on macOS 12.3, Arduino IDE 1.8.19, Python 3.10

[VojtechBartoska]

Seems this can be closed, also this PR was merged: #6448

[Martin-Laclaustra]

It looks that the most recent platform.txt changed all the calls.

Should we wait until a release to close the issue?

[VojtechBartoska]

No need, we can close this as it's already in the Master branch. Is now available by manual branch check out.