Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Add python include for compilation #803

Closed

Conversation

illicitonion
Copy link
Contributor

PR Checklist

Please check if your PR fulfills the following requirements:

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

  • Bugfix

What is the current behavior?

When compiling pypi wheels, the rules do not set the correct includes on darwin-arm64. In our case we tried using the psutil package. And got the following error in this repro:

$ bazel build :main
[...]
Failed to build psutil
 (  error: subprocess-exited-with-error

  × Building wheel for psutil (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [50 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-11.0-arm64-cpython-310
      creating build/lib.macosx-11.0-arm64-cpython-310/psutil
      copying psutil/_pswindows.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil
      copying psutil/_common.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil
      copying psutil/__init__.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil
      copying psutil/_psosx.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil
      copying psutil/_psbsd.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil
      copying psutil/_psaix.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil
      copying psutil/_pslinux.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil
      copying psutil/_compat.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil
      copying psutil/_psposix.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil
      copying psutil/_pssunos.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil
      creating build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_contracts.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_connections.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/runner.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_unicode.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_misc.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_posix.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_linux.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_sunos.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/__init__.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_aix.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_process.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_bsd.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_system.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_osx.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_memleaks.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_windows.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/__main__.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/foo.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      copying psutil/tests/test_testutils.py -> build/lib.macosx-11.0-arm64-cpython-310/psutil/tests
      running build_ext
      building 'psutil._psutil_osx' extension
      creating build/temp.macosx-11.0-arm64-cpython-310
      creating build/temp.macosx-11.0-arm64-cpython-310/psutil
      creating build/temp.macosx-11.0-arm64-cpython-310/psutil/arch
      creating build/temp.macosx-11.0-arm64-cpython-310/psutil/arch/osx
      Compiling with an SDK that doesn't seem to exist: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk
      Please check your Xcode installation
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -fPIC -I/var/folders/nx/g31mzz4d2mx2t4fsl1dwhycc0000gn/T/tmps98q1tpw/tools/deps/include -I/var/folders/nx/g31mzz4d2mx2t4fsl1dwhycc0000gn/T/tmps98q1tpw/tools/deps/include/ncursesw -I/var/folders/nx/g31mzz4d2mx2t4fsl1dwhycc0000gn/T/tmps98q1tpw/tools/deps/include/uuid -Werror=unguarded-availability-new -g0 -isysroot /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=591 -DPSUTIL_OSX=1 -I/install/include/python3.10 -c psutil/_psutil_common.c -o build/temp.macosx-11.0-arm64-cpython-310/psutil/_psutil_common.o
      psutil/_psutil_common.c:9:10: fatal error: 'Python.h' file not found
      #include <Python.h>
               ^~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for psutil
ERROR: Failed to build one or more wheels
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_xxx/yyy/external/python3_10_aarch64-apple-darwin/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/private/var/tmp/_bazel_xxx/yyy/external/python3_10_aarch64-apple-darwin/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/private/var/tmp/_bazel_xxx/yyy/external/rules_python/python/pip_install/extract_wheels/extract_single_wheel.py", line 81, in <module>
    main()
  File "/private/var/tmp/_bazel_xxx/yyy/external/rules_python/python/pip_install/extract_wheels/extract_single_wheel.py", line 57, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/private/var/tmp/_bazel_xxx/yyy/external/python3_10_aarch64-apple-darwin/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/tmp/_bazel_xxx/yyy/external/python3_10_aarch64-apple-darwin/bin/python3', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--index=https://pypi.apple.com/simple', '--index-url', 'https://pypi.apple.com/simple', '--index-url', 'https://pypi.apple.com/simple', '-r', '/var/folders/0l/nfqx5j4j08dg3s3nwx51vqjw0000gn/T/tmpslxj3627']' returned non-zero exit status 1.
)

What is the new behavior?

Adds the correct import path to be able to find Python.h

Does this PR introduce a breaking change?

  • Yes
  • No

@groodt
Copy link
Collaborator

groodt commented Aug 27, 2022

Thanks @illicitonion !

Your fix was inspiration for a bigger fix and included here:
#809

@groodt groodt closed this Aug 27, 2022
@illicitonion illicitonion deleted the fix-compile-cc-on-m1 branch August 29, 2022 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants