Unable to compile python3 from inside venv -- missing pyconfig.h

[s1113950]

I've installed all required deps through brew but still am missing python headers.

Also tried installing python3.7.1 on the host via brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/1d4f1d5602eac539f4c02b4a82f78b3a3ed5413f/Formula/python.rb as well as brew install python which now installs python3.7.4; both don't work.

Additionally, I tried using both e3fa7a300fc474bb131077026f6ae198796a8dff and master, but to no avail, I've used that hash before to create the app (on a different laptop this time) but it doesn't appear to work on this new Mac.

Here's my traceback:

error: [Errno 2] No such file or directory: '/Users/ryanrobertson/kivy-ios/dist/hostpython3/include/python3.7m/pyconfig.h'
[DEBUG   ] make: *** [sharedmods] Error 1
Traceback (most recent call last):
  File "./toolchain.py", line 1569, in <module>
    ToolchainCL()
  File "./toolchain.py", line 1284, in __init__
    getattr(self, args.command)()
  File "./toolchain.py", line 1325, in build
    build_recipes(args.recipe, ctx)
  File "./toolchain.py", line 1161, in build_recipes
    recipe.execute()
  File "/Users/ryanrobertson/kivy-ios/toolchain.py", line 728, in execute
    self.build_all()
  File "/Users/ryanrobertson/kivy-ios/toolchain.py", line 84, in _cache_execution
    f(self, *args, **kwargs)
  File "/Users/ryanrobertson/kivy-ios/toolchain.py", line 836, in build_all
    self.build(arch)
  File "/Users/ryanrobertson/kivy-ios/toolchain.py", line 84, in _cache_execution
    f(self, *args, **kwargs)
  File "/Users/ryanrobertson/kivy-ios/toolchain.py", line 810, in build
    self.build_arch(arch)
  File "/Users/ryanrobertson/kivy-ios/toolchain.py", line 873, in build_arch
    getattr(self, build)()
  File "/Users/ryanrobertson/kivy-ios/recipes/hostpython3/__init__.py", line 91, in build_x86_64
    _env=build_env)
  File "/Users/ryanrobertson/kivy-ios/toolchain.py", line 67, in shprint
    for line in cmd:
  File "./tools/external/sh.py", line 565, in next
    self.wait()
  File "./tools/external/sh.py", line 500, in wait
    self.handle_command_exit_code(exit_code)
  File "./tools/external/sh.py", line 516, in handle_command_exit_code
    raise exc(self.ran, self.process.stdout, self.process.stderr)
sh.ErrorReturnCode_2: 

  RAN: '/usr/bin/make -C /Users/ryanrobertson/kivy-ios/build/hostpython3/x86_64/Python-3.7.1 -j4'

  STDOUT:
clang -Qunused-arguments -fcolor-diagnostics -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall --sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -arch x86_64 -mmacosx-version-min=10.12 -I/Users/ryanrobertson/kivy-ios/dist/hostlibffi/usr/local/include -I/Users/ryanrobertson/kivy-ios/dist/include/x86_64/openssl --sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -arch x86_64 -mmacosx-version-min=10.12 -I/Users/ryanrobertson/kivy-ios/dist/hostlibffi/usr/local/include -I/Users/ryanrobertson/kivy-ios/dist/include/x86_64/openssl  -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-mis... (141704 more, please see e.stdout)

  STDERR:

My install process:

python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
./toolchain.py build python3 kivy

[misl6]

I suspect it's a "config-phase" issue, can you please post full logs?

[s1113950]

sorry for the delay, here is the full log: https://gist.github.com/s1113950/ff4523b39fbdfbfeff8d74679851b56f (too long to be a comment)

[s1113950]

Just tried new hash 921f6b209dba37ada26336fcca06ddcdbe1e69b4 and also had the same error :(

[s1113950]

@misl6 I can confirm that pyconfig.h exists here:
build/hostpython3/x86_64/Python-3.7.1/
Maybe I'm doing something wrong so it's not being copied into the dist folder? After the failed compilation though hostpython3 doesn't exist in the dist folder so I can't edit it and recompile.

[s1113950]

Was able to get python to build by running this command from this folder manually:
~/kivy-ios/build/hostpython3/x86_64/Python-3.7.1

CC='clang -Qunused-arguments -fcolor-diagnostics' LDSHARED='clang -Qunused-arguments -fcolor-diagnostics -bundle -undefined dynamic_lookup -lsqlite3 -lffi -L/Users/ryanrobertson/kivy-ios/dist/hostlibffi/usr/local/lib -L/Users/ryanrobertson/kivy-ios/dist/lib -lsqlite3 -lffi -L/Users/ryanrobertson/kivy-ios/dist/hostlibffi/usr/local/lib -L/Users/ryanrobertson/kivy-ios/dist/lib' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python.exe -E ./setup.py  build

which was the command dumped by (from what I can tell):

echo "$(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' \
		_TCLTK_INCLUDES='$(TCLTK_INCLUDES)' _TCLTK_LIBS='$(TCLTK_LIBS)' \
		$(PYTHON_FOR_BUILD) $(srcdir)/setup.py $$quiet build"; \
	$(RUNSHARED) CC='$(CC)' LDSHARED='$(BLDSHARED)' OPT='$(OPT)' \
		_TCLTK_INCLUDES='$(TCLTK_INCLUDES)' _TCLTK_LIBS='$(TCLTK_LIBS)' \
		$(PYTHON_FOR_BUILD) $(srcdir)/setup.py $$quiet build

so it could be a path issue possibly?

[s1113950]

@tito I'm still blocked though because the toolchain.py script route fails with the same error even after a manual compilation of hostpython3 🤔 I'll give it another shot soon though; perhaps I just need to add a -L option to the Makefile to also use the Makefile's dir files (which includes pyconfig.h).

[s1113950]

I removed my virtualenv and set python3 explicitly and things worked, so I'll rename this ticket to reflect the specific error: unable to build python3 from a venv

[AndreMiras]

We saw that bug recently on p4a with macos host. The fix was to pin virtualenv<20, refs kivy/python-for-android#2063