Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bug] Error installing cryptography on Big Sur running on Apple Silicon #5291

Closed
acronce opened this issue Jun 30, 2020 · 7 comments
Closed

[bug] Error installing cryptography on Big Sur running on Apple Silicon #5291

acronce opened this issue Jun 30, 2020 · 7 comments
Labels
blocked-on-upstream installation

Comments

@acronce
Copy link

acronce commented Jun 30, 2020

Environment Details:

Operating System+version: macOS Big Sur 11.0 Beta (20A5299w) on Apple A12Z Bionic
Compiler+version: Apple clang version 12.0.0 (clang-1200.0.22.19)
Python version: 3.8.2

Steps to reproduce:

sudo pip3 install cryptography

Logs:

The problem is a compile error building "cryptography". Here's the build log (search below for 'error:' without quotes):

  Building wheel for cryptography (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Applications/Xcode-beta.app/Contents/Developer/usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-install-kkmw5tcb/cryptography/setup.py'"'"'; __file__='"'"'/private/tmp/pip-install-kkmw5tcb/cryptography/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/tmp/pip-wheel-jnxw961w --python-tag cp38
       cwd: /private/tmp/pip-install-kkmw5tcb/cryptography/
  Complete output (150 lines):
  c/_cffi_backend.c:6245:9: warning: 'ffi_prep_closure' is deprecated [-Wdeprecated-declarations]
      if (ffi_prep_closure(closure, &cif_descr->cif,
          ^
  /Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/ffi/ffi.h:350:18: note: 'ffi_prep_closure' has been explicitly marked deprecated here
    __attribute__((deprecated))
                   ^
  1 warning generated.
  c/_cffi_backend.c:6245:9: error: implicit declaration of function 'ffi_prep_closure' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
      if (ffi_prep_closure(closure, &cif_descr->cif,
          ^
  1 error generated.
  Traceback (most recent call last):
    File "/Applications/Xcode-beta.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/unixccompiler.py", line 117, in _compile
      self.spawn(compiler_so + cc_args + [src, '-o', obj] +
    File "/Applications/Xcode-beta.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/ccompiler.py", line 910, in spawn
      spawn(cmd, dry_run=self.dry_run)
    File "/Applications/Xcode-beta.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/spawn.py", line 36, in spawn
      _spawn_posix(cmd, search_path, dry_run=dry_run)
    File "/Applications/Xcode-beta.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/distutils/spawn.py", line 157, in _spawn_posix
      raise DistutilsExecError(
  distutils.errors.DistutilsExecError: command 'xcrun' failed with exit status 1
@acronce acronce mentioned this issue Jun 30, 2020
Closed
@alex
Copy link
Member

alex commented Jun 30, 2020

Looks like as an initial matter this is a CFFI issue, in that it's trying to use a deprecated function from libffi I've ping'd the cffi maintainers about it.

@acronce
Copy link
Author

acronce commented Jul 4, 2020

I hacked around this by modifying ffitarget_arm64.h to enable the legacy closure API:

#define FFI_LEGACY_CLOSURE_API 1

Now CFFI builds and installs, but I see this error building cryptography ('openssl/opensslv.h' file not found):

Building wheels for collected packages: cryptography
  Building wheel for cryptography (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Applications/Xcode-beta.app/Contents/Developer/usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-install-5j1xtg23/cryptography/setup.py'"'"'; __file__='"'"'/private/tmp/pip-install-5j1xtg23/cryptography/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/tmp/pip-wheel-v3rywlqi --python-tag cp38
       cwd: /private/tmp/pip-install-5j1xtg23/cryptography/
  Complete output (109 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.14.6-arm64-3.8
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography
  copying src/cryptography/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography
  copying src/cryptography/utils.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography
  copying src/cryptography/__about__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography
  copying src/cryptography/exceptions.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography
  copying src/cryptography/fernet.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat
  copying src/cryptography/hazmat/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography/x509
  copying src/cryptography/x509/oid.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/x509
  copying src/cryptography/x509/general_name.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/x509
  copying src/cryptography/x509/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/x509
  copying src/cryptography/x509/extensions.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/x509
  copying src/cryptography/x509/name.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/x509
  copying src/cryptography/x509/base.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/x509
  copying src/cryptography/x509/certificate_transparency.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/x509
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends
  copying src/cryptography/hazmat/backends/interfaces.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends
  copying src/cryptography/hazmat/backends/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/cmac.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/constant_time.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/keywrap.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/hmac.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/hashes.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/padding.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/serialization.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/mac.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/bindings
  copying src/cryptography/hazmat/bindings/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/bindings
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/backend.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ec.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ciphers.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/x509.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/aead.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/encode_asn1.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/rsa.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/dh.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/cmac.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/utils.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/dsa.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/decode_asn1.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/hmac.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/x25519.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/hashes.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/backends/openssl
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/scrypt.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/pbkdf2.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/hkdf.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/x963kdf.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/kbkdf.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/concatkdf.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/kdf
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/totp.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/hotp.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/utils.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/twofactor
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/algorithms.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/aead.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/modes.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/base.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/ciphers
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/ec.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/rsa.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/dh.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/utils.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/dsa.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/x25519.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/padding.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/primitives/asymmetric
  creating build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/bindings/openssl
  copying src/cryptography/hazmat/bindings/openssl/__init__.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/bindings/openssl
  copying src/cryptography/hazmat/bindings/openssl/_conditional.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/bindings/openssl
  copying src/cryptography/hazmat/bindings/openssl/binding.py -> build/lib.macosx-10.14.6-arm64-3.8/cryptography/hazmat/bindings/openssl
  running egg_info
  writing src/cryptography.egg-info/PKG-INFO
  writing dependency_links to src/cryptography.egg-info/dependency_links.txt
  writing requirements to src/cryptography.egg-info/requires.txt
  writing top-level names to src/cryptography.egg-info/top_level.txt
  reading manifest file 'src/cryptography.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching 'docs/_build'
  warning: no previously-included files matching '*' found under directory 'vectors'
  writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
  running build_ext
  generating cffi module 'build/temp.macosx-10.14.6-arm64-3.8/_padding.c'
  creating build/temp.macosx-10.14.6-arm64-3.8
  generating cffi module 'build/temp.macosx-10.14.6-arm64-3.8/_constant_time.c'
  generating cffi module 'build/temp.macosx-10.14.6-arm64-3.8/_openssl.c'
  building '_openssl' extension
  creating build/temp.macosx-10.14.6-arm64-3.8/build
  creating build/temp.macosx-10.14.6-arm64-3.8/build/temp.macosx-10.14.6-arm64-3.8
  xcrun -sdk macosx clang -arch x86_64 -arch arm64 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -I/Applications/Xcode-beta.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c build/temp.macosx-10.14.6-arm64-3.8/_openssl.c -o build/temp.macosx-10.14.6-arm64-3.8/build/temp.macosx-10.14.6-arm64-3.8/_openssl.o -Wconversion -Wno-error=sign-conversion
  build/temp.macosx-10.14.6-arm64-3.8/_openssl.c:546:10: fatal error: 'openssl/opensslv.h' file not found
  #include <openssl/opensslv.h>
           ^~~~~~~~~~~~~~~~~~~~
  1 error generated.
  error: command 'xcrun' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for cryptography

@reaperhulk
Copy link
Member

The new error is because you need to build a copy of OpenSSL and provide the path to it for cryptography to link against. We don't include it or build it as part of the compile process.

To keep going down this path you'll probably want to take a look at openssl/openssl#12254, but after that patch run these commands (pick a valid path for that ossl dir of course):

./Configure shared darwin64-arm64-cc no-asm --prefix=/Users/youruser/ossl
make -j$(sysctl -n hw.physicalcpu)
make install_sw install_ssldirs

That will give you a /Users/youruser/ossl directory that you can then compile cryptography against with:

CRYPTOGRAPHY_SUPPRESS_LINK_FLAGS="1" LDFLAGS="/Users/youruser/ossl/lib/libcrypto.a /Users/youruser/ossl/lib/libssl.a" CFLAGS="-I/Users/youruser/ossl/include" pip install cryptography --no-use-wheel cryptography

@acronce
Copy link
Author

acronce commented Jul 4, 2020

Thanks for the detailed response!

I've got openssl 1.1.1 stable patched and building under arm64. But when I try to build cryptography (specifying my installed openssl), I get this error (even after updating to the latest pip3):

no such option: --no-use-wheel

@reaperhulk
Copy link
Member

Oops that’s an old flag. You can drop it entirely for your purposes since no wheel is currently available.

@acronce
Copy link
Author

acronce commented Jul 5, 2020 •
edited
Loading

Thanks. After futzing around a bit, I got cryptography to build. But when I tried to install conan, it wanted to rebuild cryptography again, and I got the same error.

So I passed the same options to the conan install command, hoping that only cryptography would actually be built (or that it would be harmless to provide the openssl link and include options to other packages that might be built):

sudo CRYPTOGRAPHY_SUPPRESS_LINK_FLAGS="1" LDFLAGS="/Users/acronce/Sources/thirdparty/openssl/install/lib/libcrypto.a /Users/acronce/Sources/thirdparty/openssl/install/lib/libssl.a" CFLAGS="-I/Users/acronce/Sources/thirdparty/openssl/install/include" pip3 install conan

This worked! I now have conan installed, which is great.

Thanks again for your help with this work around.

@reaperhulk
Copy link
Member

Glad it works! I'm going to close this for now since there's not much we can do here until a Mac with Apple Silicon is actually available (and it'll be a pain to do things with that until such time as a CI service bites the bullet and buys a huge pile of them).

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
blocked-on-upstream installation
Milestone
No milestone
Development

No branches or pull requests
3 participants
@alex @reaperhulk @acronce