Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

arm64 macOS wheel is broken #20

Closed
messense opened this issue Jul 26, 2022 · 9 comments · Fixed by #21
Closed

arm64 macOS wheel is broken #20

messense opened this issue Jul 26, 2022 · 9 comments · Fixed by #21
Labels
bug Something isn't working

Comments

@messense
Copy link
Member 

In [1]: from ceresdb_client import Client
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 from ceresdb_client import Client

File ~/.pyenv/versions/3.10.4/envs/brm-api/lib/python3.10/site-packages/ceresdb_client/__init__.py:1, in <module>
----> 1 from .ceresdb_client import *
      3 __doc__ = ceresdb_client.__doc__
      4 if hasattr(ceresdb_client, "__all__"):

ImportError: dlopen(/Users/messense/.pyenv/versions/3.10.4/envs/test/lib/python3.10/site-packages/ceresdb_client/ceresdb_client.abi3.so, 0x0002): symbol not found in flat namespace (_aes_hw_cbc_encrypt)

symbol not found in flat namespace (_aes_hw_cbc_encrypt) looks like the same kind of issue as ray-project/ray#14697, I think it might be caused by misconfiguration of grpc or boringssl when cross compiling from x86_64 macOS to arm64 macOS.
@waynexia waynexia added the bug Something isn't working label Jul 26, 2022
@messense
Copy link
Member Author

BTW, boringssl-src dependency has problem cross compiling to Linux aarch64 too. It might be related to this.

error: failed to run custom build command for `grpcio-sys v0.9.1+1.38.0`

Caused by:
  process didn't exit successfully: `/home/runner/work/ceresdb-client-py/ceresdb-client-py/target/release/build/grpcio-sys-f1fff01a9ba9e94e/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=grpc_wrap.cc
  cargo:rerun-if-changed=grpc
  cargo:rerun-if-env-changed=UPDATE_BIND
  cargo:rerun-if-env-changed=CARGO_CFG_TARGET_OS
  cargo:rerun-if-env-changed=GRPCIO_SYS_USE_PKG_CONFIG
  cargo:rerun-if-env-changed=CARGO_CFG_TARGET_OS
  cargo:rerun-if-env-changed=CARGO_CFG_TARGET_OS
  cargo:rerun-if-env-changed=CARGO_CFG_TARGET_OS
  cargo:rerun-if-env-changed=CXX
  OPT_LEVEL = Some("3")
  TARGET = Some("aarch64-unknown-linux-gnu")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_aarch64-unknown-linux-gnu = None
  CC_aarch64_unknown_linux_gnu = Some("aarch64-unknown-linux-gnu-gcc")
  CFLAGS_aarch64-unknown-linux-gnu = None
  CFLAGS_aarch64_unknown_linux_gnu = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = Some("fp,neon,pmuv3")
  CMAKE_TOOLCHAIN_FILE_aarch64-unknown-linux-gnu = None
  CMAKE_TOOLCHAIN_FILE_aarch64_unknown_linux_gnu = None
  TARGET_CMAKE_TOOLCHAIN_FILE = None
  CMAKE_TOOLCHAIN_FILE = None
  CMAKE_GENERATOR_aarch64-unknown-linux-gnu = None
  CMAKE_GENERATOR_aarch64_unknown_linux_gnu = None
  TARGET_CMAKE_GENERATOR = None
  CMAKE_GENERATOR = None
  CMAKE_PREFIX_PATH_aarch64-unknown-linux-gnu = None
  CMAKE_PREFIX_PATH_aarch64_unknown_linux_gnu = None
  TARGET_CMAKE_PREFIX_PATH = None
  CMAKE_PREFIX_PATH = None
  CMAKE_aarch64-unknown-linux-gnu = None
  CMAKE_aarch64_unknown_linux_gnu = None
  TARGET_CMAKE = None
  CMAKE = None
  running: "cmake" "/root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl" "-DCMAKE_INSTALL_PREFIX=/home/runner/work/ceresdb-client-py/ceresdb-client-py/target/aarch64-unknown-linux-gnu/release/build/grpcio-sys-9836eafd78d35a6b/out/boringssl-build" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_C_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_CXX_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_ASM_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" "-DCMAKE_BUILD_TYPE=Release"
  -- The C compiler identification is GNU 6.5.0
  -- The CXX compiler identification is GNU 6.5.0
  -- Check for working C compiler: /usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc
  -- Check for working C compiler: /usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Check for working CXX compiler: /usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++
  -- Check for working CXX compiler: /usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++ -- works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- The ASM compiler identification is GNU
  -- Found assembler: /usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/runner/work/ceresdb-client-py/ceresdb-client-py/target/aarch64-unknown-linux-gnu/release/build/grpcio-sys-9836eafd78d35a6b/out/boringssl-build/build
  running: "cmake" "--build" "." "--target" "ssl" "--config" "Release" "--parallel" "2"
  Scanning dependencies of target ssl
  [  7%] Building CXX object CMakeFiles/ssl.dir/src/ssl/d1_both.cc.o
  [  7%] Building CXX object CMakeFiles/ssl.dir/src/ssl/bio_ssl.cc.o
  [  7%] Building CXX object CMakeFiles/ssl.dir/src/ssl/d1_lib.cc.o
  [  7%] Building CXX object CMakeFiles/ssl.dir/src/ssl/d1_pkt.cc.o
  [ 15%] Building CXX object CMakeFiles/ssl.dir/src/ssl/d1_srtp.cc.o
  [ 15%] Building CXX object CMakeFiles/ssl.dir/src/ssl/dtls_method.cc.o
  [ 15%] Building CXX object CMakeFiles/ssl.dir/src/ssl/dtls_record.cc.o
  [ 23%] Building CXX object CMakeFiles/ssl.dir/src/ssl/encrypted_client_hello.cc.o
  [ 23%] Building CXX object CMakeFiles/ssl.dir/src/ssl/handoff.cc.o
  [ 23%] Building CXX object CMakeFiles/ssl.dir/src/ssl/handshake.cc.o
  [ 30%] Building CXX object CMakeFiles/ssl.dir/src/ssl/handshake_client.cc.o
  [ 30%] Building CXX object CMakeFiles/ssl.dir/src/ssl/handshake_server.cc.o
  [ 30%] Building CXX object CMakeFiles/ssl.dir/src/ssl/s3_both.cc.o
  [ 38%] Building CXX object CMakeFiles/ssl.dir/src/ssl/s3_lib.cc.o
  [ 38%] Building CXX object CMakeFiles/ssl.dir/src/ssl/s3_pkt.cc.o
  [ 38%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_aead_ctx.cc.o
  [ 46%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_asn1.cc.o
  [ 46%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_buffer.cc.o
  [ 46%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_cert.cc.o
  [ 53%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_cipher.cc.o
  [ 53%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_file.cc.o
  [ 53%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_key_share.cc.o
  [ 61%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_lib.cc.o
  [ 61%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_privkey.cc.o
  [ 61%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_session.cc.o
  [ 69%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_stat.cc.o
  [ 69%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_transcript.cc.o
  [ 69%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_versions.cc.o
  [ 76%] Building CXX object CMakeFiles/ssl.dir/src/ssl/ssl_x509.cc.o
  [ 76%] Building CXX object CMakeFiles/ssl.dir/src/ssl/t1_enc.cc.o
  [ 76%] Building CXX object CMakeFiles/ssl.dir/src/ssl/t1_lib.cc.o
  [ 84%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls13_both.cc.o
  [ 84%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls13_client.cc.o
  [ 84%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls13_enc.cc.o
  [ 92%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls13_server.cc.o
  [ 92%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls_method.cc.o
  [ 92%] Building CXX object CMakeFiles/ssl.dir/src/ssl/tls_record.cc.o
  [100%] Linking CXX static library libssl.a
  [100%] Built target ssl
  cargo:root=/home/runner/work/ceresdb-client-py/ceresdb-client-py/target/aarch64-unknown-linux-gnu/release/build/grpcio-sys-9836eafd78d35a6b/out/boringssl-build
  running: "cmake" "/root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl" "-DCMAKE_INSTALL_PREFIX=/home/runner/work/ceresdb-client-py/ceresdb-client-py/target/aarch64-unknown-linux-gnu/release/build/grpcio-sys-9836eafd78d35a6b/out/boringssl-build" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_C_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_CXX_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-g++" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC" "-DCMAKE_ASM_COMPILER=/usr/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" "-DCMAKE_BUILD_TYPE=Release"
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/runner/work/ceresdb-client-py/ceresdb-client-py/target/aarch64-unknown-linux-gnu/release/build/grpcio-sys-9836eafd78d35a6b/out/boringssl-build/build
  running: "cmake" "--build" "." "--target" "crypto" "--config" "Release" "--parallel" "2"
  Scanning dependencies of target crypto
  [  1%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/cipher_extra/aes128gcmsiv-x86_64.S.o
  [  1%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/chacha/chacha-x86_64.S.o
  [  1%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/aesni-gcm-x86_64.S.o
  [  1%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/cipher_extra/chacha20_poly1305_x86_64.S.o
  [  1%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/ghash-ssse3-x86_64.S.o
  [  2%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/aesni-x86_64.S.o
  [  2%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/ghash-x86_64.S.o
  [  3%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/md5-x86_64.S.o
  [  3%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/p[256](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:262)-x86_64-asm.S.o
  [  3%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/p256_beeu-x86_64-asm.S.o
  [  4%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/rdrand-x86_64.S.o
  [  4%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/rsaz-avx2.S.o
  [  4%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/sha1-x86_64.S.o
  [  6%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/sha256-x86_64.S.o
  [  6%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/sha512-x86_64.S.o
  [  6%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/vpaes-x86_64.S.o
  [  7%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/x86_64-mont.S.o
  [  7%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/fipsmodule/x86_64-mont5.S.o
  [  7%] Building ASM object CMakeFiles/crypto.dir/linux-x86_64/crypto/test/trampoline-x86_64.S.o
  [  7%] Building C object CMakeFiles/crypto.dir/err_data.c.o
  [  8%] Building ASM object CMakeFiles/crypto.dir/src/crypto/hrss/asm/poly_rq_mul.S.o
  [  8%] Building C object CMakeFiles/crypto.dir/src/crypto/asn1/a_bitstr.c.o

  --- stderr
  make: warning: -j2 forced in submake: resetting jobserver mode.
  make: warning: -j2 forced in submake: resetting jobserver mode.
  /root/.cargo/registry/src/github.com-1ecc6[299](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:305)db9ec823/boringssl-src-0.3.0+688fc5c/boringssl/src/crypto/hrss/asm/poly_rq_mul.S: Assembler messages:
  /root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl/src/crypto/hrss/asm/poly_rq_mul.S:[320](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:326): Error: unknown pseudo-op: `.att_syntax'
  /root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl/src/crypto/hrss/asm/poly_rq_mul.S:[323](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:329): Error: unknown mnemonic `push' -- `push %rbp'
  /root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl/src/crypto/hrss/asm/poly_rq_mul.S:[325](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:331): Error: bad register expression
  /root/.cargo/registry/src/github.com-1ecc6299db9ec823/boringssl-src-0.3.0+688fc5c/boringssl/src/crypto/hrss/asm/poly_rq_mul.S:[326](https://github.com/messense/ceresdb-client-py/runs/7492958698?check_suite_focus=true#step:7:332): Error: unknown mnemonic `movq' -- `movq %rsp,%rbp'

@messense
Copy link
Member Author

I think rust-lang/cmake-rs#158 could resolve this, need to test it out.

@ShiKaiWi ShiKaiWi mentioned this issue Jul 27, 2022
Closed
@messense
Copy link
Member Author

#21 changed the error to symbol not found in flat namespace (___isPlatformVersionAtLeast)

In [1]: from ceresdb_client import Client
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 from ceresdb_client import Client

File ~/.pyenv/versions/3.10.4/envs/brm-api/lib/python3.10/site-packages/ceresdb_client/__init__.py:1, in <module>
----> 1 from .ceresdb_client import *
      3 __doc__ = ceresdb_client.__doc__
      4 if hasattr(ceresdb_client, "__all__"):

ImportError: dlopen(/Users/messense/.pyenv/versions/3.10.4/envs/brm-api/lib/python3.10/site-packages/ceresdb_client/ceresdb_client.abi3.so, 0x0002): symbol not found in flat namespace (___isPlatformVersionAtLeast)

@messense
Copy link
Member Author

messense commented Jul 29, 2022 •
edited
Loading

symbol not found in flat namespace (___isPlatformVersionAtLeast) looks like the same kind of issue as alexcrichton/curl-rust#279 which was fixed by linking against clang runtime. I think it might need to be addressed in boringssl-src or grpc-sys.

$ objdump --syms /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.1.6/lib/darwin/libclang_rt.osx.a
...
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.1.6/lib/darwin/libclang_rt.osx.a(os_version_check.c.o):

SYMBOL TABLE:
0000000000000000 l     F __TEXT,__text ltmp0
0000000000000bc0 l     O __DATA,__bss __MergedGlobals
00000000000000e8 l     F __TEXT,__text _compatibilityInitializeAvailabilityCheck
00000000000004a0 l     F __TEXT,__text __initializeAvailabilityCheck
000000000000023c l     F __TEXT,__text _initializeAvailabilityCheck
00000000000007d0 l     F __TEXT,__text ___isPlatformOrVariantPlatformVersionAtLeast.cold.1
0000000000000868 l     O __TEXT,__cstring l_.str.3
0000000000000884 l     O __TEXT,__cstring l_.str.4
0000000000000895 l     O __TEXT,__cstring l_.str.5
00000000000008b1 l     O __TEXT,__cstring l_.str.6
00000000000008ce l     O __TEXT,__cstring l_.str.7
00000000000008ee l     O __TEXT,__cstring l_.str.8
000000000000090e l     O __TEXT,__cstring l_.str.9
0000000000000923 l     O __TEXT,__cstring l_.str.10
000000000000092f l     O __TEXT,__cstring l_.str.11
0000000000000941 l     O __TEXT,__cstring l_.str.12
0000000000000954 l     O __TEXT,__cstring l_.str.13
000000000000095e l     O __TEXT,__cstring l_.str.14
000000000000098f l     O __TEXT,__cstring l_.str.15
0000000000000991 l     O __TEXT,__cstring l_.str.16
00000000000009a0 l     O __TEXT,__cstring l_.str.17
00000000000007f4 l     O __TEXT,__cstring l___func__.__isPlatformOrVariantPlatformVersionAtLeast
0000000000000820 l     O __TEXT,__cstring l_.str.1
0000000000000833 l     O __TEXT,__cstring l_.str.2
00000000000007f4 l     O __TEXT,__cstring ltmp1
0000000000000bc0 l     O __DATA,__bss ltmp2
00000000000009b0 l     O __LD,__compact_unwind ltmp3
0000000000000a90 l     O __TEXT,__eh_frame ltmp4
0000000000000000 g     F __TEXT,__text ___isOSVersionAtLeast
0000000000000244 g     F __TEXT,__text ___isPlatformOrVariantPlatformVersionAtLeast
00000000000000f0 g     F __TEXT,__text ___isPlatformVersionAtLeast
0000000000000000         *UND* ___assert_rtn
0000000000000000         *UND* ___stack_chk_fail
0000000000000000         *UND* ___stack_chk_guard
0000000000000000  w      *UND* _dispatch_once_f
0000000000000000         *UND* _dlsym
0000000000000000         *UND* _fclose
0000000000000000         *UND* _fopen
0000000000000000         *UND* _fread
0000000000000000         *UND* _free
0000000000000000         *UND* _fseek
0000000000000000         *UND* _ftell
0000000000000000         *UND* _malloc
0000000000000000         *UND* _rewind
0000000000000000         *UND* _sscanf
...

@messense
Copy link
Member Author

messense commented Jul 29, 2022 •
edited
Loading

I wonder why didn't you use the pure Rust tonic crate for grpc support, grpc-rs build is quite complicated.

@waynexia
Copy link
Member

I've wondered about this too. The key reason was performance. But we haven't compared them for a long time so the situation may have changed.

@messense
Copy link
Member Author

@waynexia https://github.com/LesnyRumcajs/grpc_bench/wiki/2022-04-23-bench-results

@waynexia
Copy link
Member

@waynexia https://github.com/LesnyRumcajs/grpc_bench/wiki/2022-04-23-bench-results

Nice link! Looks like grpcio is still faster than tonic in the multicore machine... I'm afraid we need to keep paying for it for a while

@messense
Copy link
Member Author

messense commented Aug 2, 2022

I've confirmed that it works on M1 macOS with 9a417c4

@messense messense mentioned this issue Aug 4, 2022
Merged
@waynexia waynexia mentioned this issue Sep 7, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: cmake cross compiling fix for arm64 macOS
2 participants
@messense @waynexia