Python3.11 on Mac compile issues for esp-idf-sys vX.X.X

[NickAger]

I found that I had to "downgrade" my Python version to 3.9 in order to get esp-idf-sys vX.X.X to compile. I had previously installed Python using homebrew so downgraded Python from 3.11 using:

brew unlink python@3.11

Might be worthwhile adding a note about Python 3.11 compatibility issues in: book/src/02_2_software.md

[igrr]

Do you recall, what was the issue you have observed with Python 3.11?

[MabezDev]

I found that I had to "downgrade" my Python version to 3.9

Hmm, that's odd. I'm using v3.10 at the moment with no issues, did you try that or did you just go straight to 3.9? It would be great to see what the 3.11 error was if you can reproduce it.

[SergioGasquez]

I had some issues with Python 11 on Windows back in the day, the error was that there were no python 3.11 wheels for Windows at that time, not sure if they are available for MacOs atm

[NickAger]

@MabezDev I didn't try with Python v3.10

Here's a transcript:

% brew list | grep python
python@3.10
python@3.11
python@3.7
python@3.8
python@3.9

% brew link python@3.11
==> Downloading https://formulae.brew.sh/api/formula.jws.json
######################################################################## 100.0%
Linking /usr/local/Cellar/python@3.11/3.11.2... 25 symlinks created.

% python3 --version
Python 3.11.2

% cargo clean

% cargo run

.
.
.
   Compiling strum v0.23.0
   Compiling num_enum v0.5.11
   Compiling bytemuck v1.13.1
   Compiling rgb v0.8.36
   Compiling futures v0.3.27
   Compiling no-std-net v0.5.0
   Compiling enumset v1.0.12
   Compiling embedded-svc v0.24.0
    Building [======================>  ] 222/234: esp-idf-sys(build) 
error: failed to run custom build command for `esp-idf-sys v0.32.1`

Caused by:
  process didn't exit successfully: `/Users/nickager/programming/espressif-trainings/intro/hardware-check/target/debug/build/esp-idf-sys-c6afcc95ae75201b/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=ESP_IDF_TOOLS_INSTALL_DIR
  cargo:rerun-if-env-changed=ESP_IDF_SDKCONFIG
  cargo:rerun-if-env-changed=ESP_IDF_SDKCONFIG_DEFAULTS
  cargo:rerun-if-env-changed=MCU
  cargo:rerun-if-env-changed=ESP_IDF_SYS_ROOT_CRATE
  cargo:rerun-if-env-changed=ESP_IDF_VERSION
  cargo:rerun-if-env-changed=ESP_IDF_REPOSITORY
  cargo:rerun-if-env-changed=ESP_IDF_CMAKE_GENERATOR
  cargo:rerun-if-env-changed=IDF_PATH
  cargo:rerun-if-env-changed=EXTRA-COMPONENTS
  cargo:rerun-if-env-changed=ESP_IDF_COMPONENTS
  PATH=/Users/nickager/.espressif/tools/riscv32-esp-elf/esp-2021r2-patch3-8.4.0/riscv32-esp-elf/bin:/Users/nickager/.espressif/tools/cmake/3.20.3/CMake.app/Contents/bin:/Users/nickager/.espressif/tools/ninja/1.10.2/:/Users/nickager/.espressif/esp-idf/v4.4.1/tools:$PATH
  pip 22.3.1 from /usr/local/lib/python3.11/site-packages/pip (python 3.11)
  created virtual environment CPython3.11.2.final.0-64 in 7777ms
    creator CPython3Posix(dest=/Users/nickager/.espressif/python_env/idf4.4_py3.11_env, clear=False, no_vcs_ignore=False, global=False)
    seeder PipInvoke(download=False, pip=bundle, setuptools=bundle, wheel=bundle)
      added seed packages: pip==23.0.1, setuptools==67.4.0, wheel==0.38.4
    activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
  Looking in indexes: https://pypi.org/simple, https://dl.espressif.com/pypi
  Ignoring None: markers 'sys_platform == "win32"' don't match your environment
  Requirement already satisfied: setuptools>=21 in /Users/nickager/.espressif/python_env/idf4.4_py3.11_env/lib/python3.11/site-packages (from -r /Users/nickager/.espressif/esp-idf/v4.4.1/requirements.txt (line 4)) (67.4.0)
  Collecting click>=7.0
    Using cached https://dl.espressif.com/pypi/click/click-8.1.3-py3-none-any.whl (96 kB)
  Collecting pyserial>=3.3
    Using cached https://dl.espressif.com/pypi/pyserial/pyserial-3.5-py2.py3-none-any.whl (90 kB)
  Collecting future>=0.15.2
    Using cached https://dl.espressif.com/pypi/future/future-0.18.3-py3-none-any.whl (492 kB)
  Collecting cryptography>=2.1.4
    Using cached https://dl.espressif.com/pypi/cryptography/cryptography-39.0.2-cp36-abi3-macosx_10_12_x86_64.whl (2.9 MB)
  Collecting pyparsing<2.4.0,>=2.0.3
    Using cached https://dl.espressif.com/pypi/pyparsing/pyparsing-2.3.1-py2.py3-none-any.whl (61 kB)
  Collecting pyelftools>=0.22
    Using cached https://dl.espressif.com/pypi/pyelftools/pyelftools-0.29-py2.py3-none-any.whl (174 kB)
  Collecting idf-component-manager~=1.0
    Using cached https://dl.espressif.com/pypi/idf-component-manager/idf_component_manager-1.2.2-py3-none-any.whl (116 kB)
  Collecting gdbgui==0.13.2.0
    Using cached https://dl.espressif.com/pypi/gdbgui/gdbgui-0.13.2.0-py3-none-any.whl (878 kB)
  Collecting pygdbmi<=0.9.0.2
    Using cached https://dl.espressif.com/pypi/pygdbmi/pygdbmi-0.9.0.2-py3-none-any.whl (16 kB)
  Collecting python-socketio<5
    Using cached https://dl.espressif.com/pypi/python-socketio/python_socketio-4.6.1-py2.py3-none-any.whl (51 kB)
  Collecting jinja2<3.1
    Using cached https://dl.espressif.com/pypi/jinja2/Jinja2-3.0.3-py3-none-any.whl (133 kB)
  Collecting itsdangerous<2.1
    Using cached https://dl.espressif.com/pypi/itsdangerous/itsdangerous-2.0.1-py3-none-any.whl (18 kB)
  Collecting kconfiglib==13.7.1
    Using cached https://dl.espressif.com/pypi/kconfiglib/kconfiglib-13.7.1-py2.py3-none-any.whl (145 kB)
  Collecting reedsolo<=1.5.4,>=1.5.3
    Using cached https://dl.espressif.com/pypi/reedsolo/reedsolo-1.5.4-py3-none-any.whl (28 kB)
  Collecting bitstring>=3.1.6
    Using cached https://dl.espressif.com/pypi/bitstring/bitstring-4.0.1-py3-none-any.whl (45 kB)
  Collecting ecdsa>=0.16.0
    Downloading https://dl.espressif.com/pypi/ecdsa/ecdsa-0.18.0-py2.py3-none-any.whl (142 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.9/142.9 kB 210.6 kB/s eta 0:00:00
  Collecting construct==2.10.54
    Using cached https://dl.espressif.com/pypi/construct/construct-2.10.54-py3-none-any.whl (57 kB)
  Collecting Flask<1.0,>=0.12.2
    Using cached https://dl.espressif.com/pypi/flask/Flask-0.12.5-py2.py3-none-any.whl (81 kB)
  Collecting Flask-Compress<2.0,>=1.4.0
    Using cached https://dl.espressif.com/pypi/flask-compress/Flask_Compress-1.13-py3-none-any.whl (7.9 kB)
  Collecting Flask-SocketIO<3.0,>=2.9
    Using cached https://dl.espressif.com/pypi/flask-socketio/Flask_SocketIO-2.9.6-py2.py3-none-any.whl (16 kB)
  Collecting gevent<2.0,>=1.2.2
    Using cached gevent-1.5.0.tar.gz (5.3 MB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Collecting Pygments<3.0,>=2.2.0
    Using cached https://dl.espressif.com/pypi/pygments/Pygments-2.14.0-py3-none-any.whl (1.1 MB)
  Collecting cffi>=1.12
    Using cached cffi-1.15.1-cp311-cp311-macosx_10_9_x86_64.whl (179 kB)
  Collecting packaging
    Using cached https://dl.espressif.com/pypi/packaging/packaging-23.0-py3-none-any.whl (42 kB)
  Collecting requests<3
    Using cached https://dl.espressif.com/pypi/requests/requests-2.28.2-py3-none-any.whl (62 kB)
  Collecting requests-file
    Using cached https://dl.espressif.com/pypi/requests-file/requests_file-1.5.1-py2.py3-none-any.whl (3.7 kB)
  Collecting requests-toolbelt
    Using cached https://dl.espressif.com/pypi/requests-toolbelt/requests_toolbelt-0.10.1-py2.py3-none-any.whl (54 kB)
  Collecting schema
    Using cached https://dl.espressif.com/pypi/schema/schema-0.7.5-py2.py3-none-any.whl (17 kB)
  Collecting six
    Downloading https://dl.espressif.com/pypi/six/six-1.16.0-py2.py3-none-any.whl (11 kB)
  Collecting tqdm<5
    Using cached https://dl.espressif.com/pypi/tqdm/tqdm-4.65.0-py3-none-any.whl (77 kB)
  Collecting colorama
    Using cached https://dl.espressif.com/pypi/colorama/colorama-0.4.6-py2.py3-none-any.whl (25 kB)
  Collecting pyyaml>5.2
    Using cached PyYAML-6.0-cp311-cp311-macosx_10_9_x86_64.whl (188 kB)
  Collecting cachecontrol[filecache]>0.12.6
    Using cached https://dl.espressif.com/pypi/cachecontrol/CacheControl-0.12.11-py2.py3-none-any.whl (21 kB)
  Collecting contextlib2>0.6.0
    Using cached https://dl.espressif.com/pypi/contextlib2/contextlib2-21.6.0-py2.py3-none-any.whl (13 kB)
  Collecting python-engineio<4,>=3.13.0
    Using cached https://dl.espressif.com/pypi/python-engineio/python_engineio-3.14.2-py2.py3-none-any.whl (51 kB)
  Collecting MarkupSafe>=2.0
    Using cached MarkupSafe-2.1.2-cp311-cp311-macosx_10_9_x86_64.whl (13 kB)
  Collecting msgpack>=0.5.2
    Using cached msgpack-1.0.5-cp311-cp311-macosx_10_9_x86_64.whl (73 kB)
  Collecting lockfile>=0.9
    Using cached https://dl.espressif.com/pypi/lockfile/lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
  Collecting pycparser
    Using cached https://dl.espressif.com/pypi/pycparser/pycparser-2.21-py2.py3-none-any.whl (118 kB)
  Collecting Werkzeug<1.0,>=0.7
    Using cached https://dl.espressif.com/pypi/werkzeug/Werkzeug-0.16.1-py2.py3-none-any.whl (327 kB)
  Collecting brotli
    Using cached Brotli-1.0.9-cp311-cp311-macosx_10_9_x86_64.whl (413 kB)
  Collecting greenlet>=0.4.14
    Using cached greenlet-2.0.2-cp311-cp311-macosx_10_9_universal2.whl (243 kB)
  Collecting charset-normalizer<4,>=2
    Using cached charset_normalizer-3.1.0-cp311-cp311-macosx_10_9_x86_64.whl (123 kB)
  Collecting idna<4,>=2.5
    Using cached https://dl.espressif.com/pypi/idna/idna-3.4-py3-none-any.whl (61 kB)
  Collecting urllib3<1.27,>=1.21.1
    Using cached https://dl.espressif.com/pypi/urllib3/urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
  Collecting certifi>=2017.4.17
    Using cached https://dl.espressif.com/pypi/certifi/certifi-2022.12.7-py3-none-any.whl (155 kB)
  Building wheels for collected packages: gevent
    Building wheel for gevent (pyproject.toml): started
    Building wheel for gevent (pyproject.toml): still running...
    Building wheel for gevent (pyproject.toml): finished with status 'error'
  Failed to build gevent
  Creating a new Python environment in /Users/nickager/.espressif/python_env/idf4.4_py3.11_env
  Installing Python packages from /Users/nickager/.espressif/esp-idf/v4.4.1/requirements.txt

  --- stderr
  Build configuration: BuildConfig {
      esp_idf_tools_install_dir: Some(
          Global,
      ),
      esp_idf_sdkconfig: None,
      esp_idf_sdkconfig_defaults: Some(
          [
              "/Users/nickager/programming/espressif-trainings/intro/hardware-check/./sdkconfig.defaults",
          ],
      ),
      mcu: None,
      native: NativeConfig {
          esp_idf_version: Some(
              Tag(
                  "v4.4.1",
              ),
          ),
          esp_idf_repository: None,
          esp_idf_cmake_generator: None,
          idf_path: None,
          extra_components: [],
          esp_idf_components: None,
      },
      esp_idf_sys_root_crate: None,
  }
  Using managed esp-idf repository: EspIdfRemote { repo_url: None, git_ref: Tag("v4.4.1") }
    error: subprocess-exited-with-error
    
    × Building wheel for gevent (pyproject.toml) did not run successfully.
    │ exit code: 1

[MabezDev]

Failed to build gevent

Looks like we might have missing binary wheels for the gevent package with python 3.11. Will try and figure out how to solve this.

[MabezDev]

Seems like the issues for 3.11 python wheels are still present (espressif/esp-idf#10116). I imagine these will be solved over time, but for now, I recommend you stay on python 3.9.