Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[macOS] psutil compilation fails with the final version of Xcode 12 #1832

Closed
Ceylo opened this issue Sep 21, 2020 · 6 comments
Closed

[macOS] psutil compilation fails with the final version of Xcode 12 #1832

Ceylo opened this issue Sep 21, 2020 · 6 comments

Comments

@Ceylo
Copy link

Ceylo commented Sep 21, 2020

Platform

  • macOS 10.15.6 (19G2021)
  • psutil 5.7.2
  • Python 3.8.2
  • Xcode 12.0 (12A7209)
  • clang --version output:
Apple clang version 12.0.0 (clang-1200.0.32.2)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Applications/Xcode12.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Bug description

psutil setup in venv fails with

Collecting psutil
  Using cached https://files.pythonhosted.org/packages/aa/3e/d18f2c04cf2b528e18515999b0c8e698c136db78f62df34eee89cee205f1/psutil-5.7.2.tar.gz
Installing collected packages: psutil
  Running setup.py install for psutil ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/lsoltic/Development/testxc12/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-install-gxcp64ut/psutil/setup.py'"'"'; __file__='"'"'/private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-install-gxcp64ut/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-record-qtwmxusr/install-record.txt --single-version-externally-managed --compile --install-headers /Users/lsoltic/Development/testxc12/include/site/python3.8/psutil
         cwd: /private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-install-gxcp64ut/psutil/
    Complete output (141 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.14.6-x86_64-3.8
    creating build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_pswindows.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_common.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/__init__.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_psosx.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_psbsd.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_psaix.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_pslinux.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_compat.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_psposix.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    copying psutil/_pssunos.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil
    creating build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_contracts.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_connections.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/runner.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_unicode.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_misc.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_posix.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_linux.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_sunos.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/__init__.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_aix.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_process.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_bsd.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_system.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_osx.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_memleaks.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_windows.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/__main__.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    copying psutil/tests/test_testutils.py -> build/lib.macosx-10.14.6-x86_64-3.8/psutil/tests
    running build_ext
    building 'psutil._psutil_osx' extension
    creating build/temp.macosx-10.14.6-x86_64-3.8
    creating build/temp.macosx-10.14.6-x86_64-3.8/psutil
    creating build/temp.macosx-10.14.6-x86_64-3.8/psutil/arch
    creating build/temp.macosx-10.14.6-x86_64-3.8/psutil/arch/osx
    xcrun -sdk macosx clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=572 -DPSUTIL_OSX=1 -I/Users/lsoltic/Development/testxc12/include -I/Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c psutil/_psutil_common.c -o build/temp.macosx-10.14.6-x86_64-3.8/psutil/_psutil_common.o
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:11:
    In file included from /Applications/Xcode12.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include/limits.h:21:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/limits.h:63:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/cdefs.h:807:2: error: Unsupported architecture
    #error Unsupported architecture
     ^
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:11:
    In file included from /Applications/Xcode12.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include/limits.h:21:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/limits.h:64:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/machine/limits.h:8:2: error: architecture not supported
    #error architecture not supported
     ^
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:71:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_types.h:27:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:33:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/machine/_types.h:34:2: error: architecture not supported
    #error architecture not supported
     ^
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:71:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_types.h:27:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:55:9: error: unknown type name '__int64_t'
    typedef __int64_t       __darwin_blkcnt_t;      /* total blocks */
            ^
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:56:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
    typedef __int32_t       __darwin_blksize_t;     /* preferred block size */
            ^
    note: '__int128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:57:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
    typedef __int32_t       __darwin_dev_t;         /* dev_t */
            ^
    note: '__int128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:60:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_gid_t;         /* [???] process and group IDs */
            ^
    note: '__uint128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:61:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_id_t;          /* [XSI] pid_t, uid_t, or gid_t*/
            ^
    note: '__uint128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:62:9: error: unknown type name '__uint64_t'
    typedef __uint64_t      __darwin_ino64_t;       /* [???] Used for 64 bit inodes */
            ^
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:68:9: error: unknown type name '__darwin_natural_t'
    typedef __darwin_natural_t __darwin_mach_port_name_t; /* Used by mach */
            ^
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:70:9: error: unknown type name '__uint16_t'; did you mean '__uint128_t'?
    typedef __uint16_t      __darwin_mode_t;        /* [???] Some file attributes */
            ^
    note: '__uint128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:71:9: error: unknown type name '__int64_t'
    typedef __int64_t       __darwin_off_t;         /* [???] Used for file sizes */
            ^
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:72:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
    typedef __int32_t       __darwin_pid_t;         /* [???] process and group IDs */
            ^
    note: '__int128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:73:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_sigset_t;      /* [???] signal set */
            ^
    note: '__uint128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:74:9: error: unknown type name '__int32_t'; did you mean '__int128_t'?
    typedef __int32_t       __darwin_suseconds_t;   /* [???] microseconds */
            ^
    note: '__int128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:75:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_uid_t;         /* [???] user IDs */
            ^
    note: '__uint128_t' declared here
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:76:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_useconds_t;    /* [???] microseconds */
            ^
    note: '__uint128_t' declared here
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:71:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_types.h:43:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'?
    typedef __uint32_t      __darwin_wctype_t;
            ^
    note: '__uint128_t' declared here
    In file included from psutil/_psutil_common.c:9:
    In file included from /Applications/Xcode12.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/Python.h:25:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/stdio.h:64:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/_stdio.h:75:
    In file included from /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/sys/_types/_va_list.h:31:
    /Applications/Xcode12.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/machine/types.h:37:2: error: architecture not supported
    #error architecture not supported
     ^
    fatal error: too many errors emitted, stopping now [-ferror-limit=]
    20 errors generated.
    error: command 'xcrun' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/lsoltic/Development/testxc12/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-install-gxcp64ut/psutil/setup.py'"'"'; __file__='"'"'/private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-install-gxcp64ut/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/vm/_ft8xsjd24sbz6lkgs0hck4dfj9v93/T/pip-record-qtwmxusr/install-record.txt --single-version-externally-managed --compile --install-headers /Users/lsoltic/Development/testxc12/include/site/python3.8/psutil Check the logs for full command output.

Steps

python3 -m venv testxc12
cd testxc12
. bin/activate
pip install psutil

Comments

In clang call I see the options -arch arm64 -arch x86_64 but macOS 10.15 SDK is still used (which doesn't support ARM). So either -arch arm64 should be removed or macOS 11 SDK should be used.

After a quick look inside Xcode to view the available SDK, it looks like macOS 11 SDK is missing, so the only option looks to remove -arch arm64 for now.

@darthunix
Copy link

It is not a problem of psutils (it doesn't add -arch options to setup.py) but an Apple's fault. Apple builds python3 with -arch arm64 -arch x86_64 flags that are used as default flags in setup.py that can't be skipped . You can check your python3 flags with:

import sysconfig
print(sysconfig.get_config_vars())

and -arch arm64 should be there.

@Ceylo
Copy link
Author

Ceylo commented Sep 25, 2020

Oh I wasn't aware that changing the active Xcode version was also changing the Python 3 being used (unless using Python 3 from Homebrew I guess), I thought only clang & xcodebuild related tools were affected.

And I'm not familiar with Python module dev so wasn't aware that build flags were taken from Python's config.

Then I suppose final Xcode 12 is just not usable if we need psutil, and we need Xcode 12 beta or Xcode 12.2 beta or any Xcode 11 for now.

@Exitare
Copy link

Exitare commented Sep 25, 2020

Reverting back to 11.7 is fixing the issue, although it is still annoying.

@darthunix
Copy link

BTW, python3 from brew is not affected by -arch arm64. Just install it and add to your $PATH

@japaniel
Copy link

japaniel commented Oct 6, 2020

While I was hitting this when installing a different package (cffi) I was able to work around this by exporting an environment variable
export ARCHFLAGS="-arch x86_64"

yongtang added a commit to yongtang/io that referenced this issue Nov 1, 2020
See giampaolo/psutil#1832

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
kvignesh1420 pushed a commit to tensorflow/io that referenced this issue Nov 1, 2020
…1168)

See giampaolo/psutil#1832

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
@americanhanko
Copy link

americanhanko commented Dec 18, 2020

@japaniel Thanks, Daniel. Your solution worked for me on Catalina and the Xcode 12 command line tools:

❯ clang --version; sw_vers
Apple clang version 12.0.0 (clang-1200.0.32.27)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
ProductName:	Mac OS X
ProductVersion:	10.15.7
BuildVersion:	19H15
$ export ARCHFLAGS="-arch x86_64"; /usr/bin/python3 -m pip install --user --upgrade psutil

...truncated...

Successfully built psutil
Installing collected packages: psutil
Successfully installed psutil-5.7.3

i-ony pushed a commit to i-ony/io that referenced this issue Feb 8, 2021
jsirois added a commit to jsirois/pants that referenced this issue Mar 19, 2021
The interpreter has a bad `-arch arm64` flag on intel that leads to
compile errors for psutil.

See: giampaolo/psutil#1832

[ci skip-rust]
[ci skip-build-wheels]
jsirois added a commit to pantsbuild/pants that referenced this issue Mar 19, 2021
Previously we left them off for internal only PEXes. Leaving them in
does no harm since PEX uses the current interpreter if it matches
constraints and it aids in the debugability of process chroots.

Also work around Xcode 12.x Python interpreter issues hit in CI. These
interpreters have a bad `-arch arm64` flag on intel that leads to
compile errors for psutil.

See: giampaolo/psutil#1832
teskobif7 added a commit to teskobif7/io that referenced this issue Aug 14, 2024
…#1168)

See giampaolo/psutil#1832

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants