Zig for manylinux compliance without docker #756

konstin · 2021-12-27T09:24:44Z

If this actually works as I imaging it to be, we won't need the manylinux docker anymore!

This allows compiling for a specified manylinux version by using zig ccs custom glibc targeting (https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html). You just add --zig, and it will target the policy you want, defaulting to manylinux2010. Example from ubuntu 20.04:

$ cargo run -- build -m test-crates/pyo3-pure/Cargo.toml -i python --no-sdist 2> /dev/null
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.6
🐍 Not using a specific python interpreter (With abi3, an interpreter is only required on windows)
📖 Found type stub file at pyo3_pure.pyi
📦 Built wheel for abi3 Python ≥ 3.6 to /home/konsti/maturin/test-crates/pyo3-pure/target/wheels/pyo3_pure-2.1.2-cp36-abi3-manylinux_2_24_x86_64.whl
$ cargo run -- build -m test-crates/pyo3-pure/Cargo.toml -i python --no-sdist --zig 2> /dev/null
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.6
🐍 Not using a specific python interpreter (With abi3, an interpreter is only required on windows)
📖 Found type stub file at pyo3_pure.pyi
📦 Built wheel for abi3 Python ≥ 3.6 to /home/konsti/maturin/test-crates/pyo3-pure/target/wheels/pyo3_pure-2.1.2-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

Maturin has to be installed with maturin[zig] to get a compatible zig version.

Still missing:

musl (code is there, but untested)
pyproject.toml integration (do we want/need that?)
Documentation in the user guide

netlify · 2021-12-27T09:24:50Z

✔️ Deploy Preview for maturin-guide canceled.

🔨 Explore the source changes: fa00117

🔍 Inspect the deploy log: https://app.netlify.com/sites/maturin-guide/deploys/61cee893f4c4c20007903920

messense · 2021-12-27T09:32:51Z

Looks interesting, can we add a test for it on CI?

messense · 2021-12-27T09:35:04Z

pyproject.toml integration (do we want/need that?)

This one is tricky, if we allow that user should add [zig] in build-system:

[build-system]
requires = ["maturin[zig]>=0.12,<0.13"]
build-backend = "maturin"

messense · 2021-12-27T09:52:40Z

src/compile.rs

+    custom_linker_file.write_all(
+        format!(
+            r##"#!/bin/bash
+python -m ziglang cc ${{@/-lgcc_s/-lunwind}} -target {}


For musl I'm not whether we should replace libgcc_s with libunwind, for example, Alpine Linux has a patch to use libgcc_eh instead of libunwind:

https://github.com/alpinelinux/aports/blob/396a57da9bff572d15f17f57ff3f80b931d6e008/community/rust/0006-Prefer-libgcc_eh-over-libunwind-for-musl.patch

I've made this to get rid of the manylinux docker and only filled out musllinux because I was there anyway, I've no problem with just disabling musl+zig, either by ignoring --zig on musl or by erroring on the combination

I've tested musllinux with libgcc_s replaced with libunwind and it works fine, so I guess it's safe to do that. And the good thing is that it doesn't require repairing the wheel now for a pure Rust project.

messense · 2021-12-27T10:06:03Z

src/compile.rs

+    custom_linker_file.write_all(
+        format!(
+            r##"#!/bin/bash
+python -m ziglang cc ${{@/-lgcc_s/-lunwind}} -target {}


BTW, I think most distros prefer using existing packages for dependencies, so can we also support plain zig installation which isn't from PyPI?

Yes! Unfortunately we can't just use zig which would also allow using e.g. nightly zig because the zig-pypi deliberately doesn't support it (ziglang/zig-pypi#4 (comment)). We can add a feature flag to switch between zig-pypi and plain zig if distro people confirm that that works for them

I think zig cc shouldn't have many break changes since it mostly acts like a clang compiler thus nighty zig should work fine? Anyway we should prefer to use the python ziglang package and check zig version (like >= 0.9.0) to ensure it's usable.

messense · 2021-12-27T13:29:26Z

musl (code is there, but untested)

I tried it locally with some changes and it doesn't seem to work:

❯ cargo run build -m test-crates/pyo3-pure/Cargo.toml --target x86_64-unknown-linux-musl --zig
   Compiling maturin v0.12.6-beta.2 (/Users/messense/Projects/maturin)
    Finished dev [unoptimized + debuginfo] target(s) in 3.67s
     Running `target/debug/maturin build -m test-crates/pyo3-pure/Cargo.toml --target x86_64-unknown-linux-musl --zig`
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.6
⚠️  Warning: skipped unavailable python interpreter 'python3.8' from pyenv
⚠️  Warning: skipped unavailable python interpreter 'python3.10' from pyenv
🐍 Not using a specific python interpreter (With abi3, an interpreter is only required on windows)
📡 Using build options bindings from pyproject.toml
📦 Built source distribution to /Users/messense/Projects/maturin/test-crates/pyo3-pure/target/wheels/pyo3_pure-2.1.2.tar.gz
   Compiling cfg-if v1.0.0
   Compiling smallvec v1.7.0
   Compiling scopeguard v1.1.0
   Compiling unindent v0.1.7
   Compiling libc v0.2.107
   Compiling paste v0.1.18
   Compiling indoc v0.3.6
   Compiling instant v0.1.12
   Compiling lock_api v0.4.5
   Compiling parking_lot_core v0.8.5
   Compiling parking_lot v0.11.2
   Compiling pyo3 v0.15.1
   Compiling pyo3-pure v2.1.2 (/Users/messense/Projects/maturin/test-crates/pyo3-pure)
error: linking with `/Users/messense/Library/Caches/maturin/0.12.6-beta.2/./zigcc-musl-1-1.sh` failed: exit status: 1
  |
  = note: "/Users/messense/Library/Caches/maturin/0.12.6-beta.2/./zigcc-musl-1-1.sh" "-Wl,--version-script=/var/folders/0r/4fqyt_zs2tj4zvglsb7f4zjw0000gn/T/rustclJMSOE/list" "-m64" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.17jyeu6qkwlvy44g.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.18ux8egx2bzpz8so.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.196hwvt45a8d37pd.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.1bj9cg2jdcibvnyy.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.1lx2vmorw750ivny.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.1u78ltb1og4fp98b.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.1wslxyijiduv56cj.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.1zdlw2jy1h69fpa4.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.2chsv9ejbwfagiqi.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.2ehfhh99t0ogan51.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.2fjn0eo0nd1m6iv5.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.2imr8vscixk9k79d.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.2rj4m7hepztrm0i.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.2vuwz97bq4qvh1vn.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.2y0grd6y1k98ajni.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.35xmcf2ltp6qthds.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3984xz66eoh6djy0.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3ak2ide95doiqh6.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3blgepkcxy2kxjbt.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3fnm2xzyvstngx30.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3ibpiiowkakhvvt1.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3iryb1pc8nej63kl.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3ki7c61xzf439fpi.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3n02c06u0m7im8y8.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3n8uyfl922hyq7b9.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3oyeqahmkbl5502s.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3qf2bc29qssv95aa.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3sdrsk98v11fbfj5.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3uz27pwrvuvszs0z.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3vcvqgwv9eyip0de.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3wbnuq6p7ip9gtkq.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3y1cc0a8dkgcvq80.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.46cyasykn4ilx768.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.4991jmdeb49k38vh.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.4dxyjf7b46y6hkp0.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.4g4ej97a8b4zd6z6.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.4jj2bvlrf3fkislp.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.4ogowr9n8yxki44u.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.4p3hy4fav1naxdqg.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.4r0id0uhzcinn44p.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.4wg793ckp3sdyuqa.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.4x9da3b7gd6fuky0.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.52d89ehwc5hl4qgk.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.5601ok8bxcrnu6tn.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.5a1rb6m2npf3nl6.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.5d5dj9ppvtuublom.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.8yp9ehgubdzlmew.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.g2tb7f467y9sygz.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.j88uaoqie5sgyh.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.kazqtoqzhv5hbuf.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.l8lz5rlue9m022w.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.omy5d3b86u3n7a0.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.tspkpivicgnppyu.rcgu.o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/pyo3_pure.3dfxzraerd3sml8y.rcgu.o" "-Wl,--as-needed" "-L" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps" "-L" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/debug/deps" "-L" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/libpyo3-2876c84f9b694b23.rlib" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/libparking_lot-931f8b71a8bd9f08.rlib" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/libparking_lot_core-b9937a023c625405.rlib" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/libsmallvec-b92f25b0844ad228.rlib" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/liblock_api-70f84b41a4e52907.rlib" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/libscopeguard-33f9775febdd85f3.rlib" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/libinstant-269640bfb90725bc.rlib" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/libcfg_if-1e664cb3f89481e6.rlib" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/liblibc-2ee2e91d536c37e6.rlib" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/libunindent-0984363e93a97d0a.rlib" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/libpaste-4cddc391f20eea5d.rlib" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/libindoc-8fba994c3b684002.rlib" "-Wl,--start-group" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-959fef89f22dd238.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-b1fdbed2ff0e877b.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libminiz_oxide-26f2615ec860cc05.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libadler-3ab202f5008fe9e3.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libobject-534ffdfd2212f40f.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libmemchr-d7ebb1f26aad5ef2.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libaddr2line-222f097bffcc5051.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libgimli-101ef298c21d5f38.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_detect-13e9f6efc030288b.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-cf04d7665c264a5a.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-e80d007771792d61.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-6c47dce420504986.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-8d46cad63bc53896.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-ecbc8c5ae0470bde.rlib" "/var/folders/0r/4fqyt_zs2tj4zvglsb7f4zjw0000gn/T/rustclJMSOE/liblibc-1823dd7e18eaaaeb.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-a735809bf88fef79.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-cd190abde0c329b9.rlib" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-07eeceee5c485af0.rlib" "-Wl,--end-group" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-2f6be22c6f2248c2.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/Users/messense/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-unknown-linux-musl/lib" "-o" "/Users/messense/Projects/maturin/test-crates/pyo3-pure/target/x86_64-unknown-linux-musl/debug/deps/libpyo3_pure.so" "-Wl,--gc-sections" "-shared" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs"
  = note: warning: unsupported linker arg: -znoexecstack
          warning: unsupported linker arg: -zrelro
          warning: unsupported linker arg: -znow
          error: InvalidAbiVersion



error: aborting due to previous error


error: could not compile `pyo3-pure` due to 2 previous errors
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `cargo rustc --message-format json --manifest-path test-crates/pyo3-pure/Cargo.toml --target x86_64-unknown-linux-musl --lib --
❯ cat /Users/messense/Library/Caches/maturin/0.12.6-beta.2/./zigcc-musl-1-1.sh
#!/bin/bash
python -m ziglang cc ${@/-lgcc_s/-lunwind} -target x86_64-linux-musl.1.1

Could it be that it doesn't support target triple like x86_64-linux-musl.1.1 for musl libc?

Edit: it works if we remove the musl libc version suffix, for example, x86_64-linux-musl works.

src/build_options.rs

messense · 2021-12-27T13:33:57Z

src/compile.rs

+    let (zig_linker, target) = match context.platform_tag {
+        None | Some(PlatformTag::Linux) => (
+            "./zigcc-gnu.sh".to_string(),
+            "native-native-gnu".to_string(),


We can also support cross compiling with zig. I just tried locally and it works if we give zig cc the right target triple.

Yes, I've seen https://actually.fyi/posts/zig-makes-rust-cross-compilation-just-work/ and it would awesome if we could port that to maturin! Do you still have to download some built python or specific headers for the target platform to make this work, and if so, what's the easiest way to download them?

Do you still have to download some built python or specific headers for the target platform to make this work

Yes, it's needed for non-abi3 wheels.

and if so, what's the easiest way to download them?

Maybe https://github.com/indygreg/python-build-standalone can be used.

On second thought, for linux maybe we can just collect sysconfigs from manylinux docker images and use them as cross compiling target Python sysconfigdata (something like this).

konstin · 2021-12-27T13:41:18Z

Looks interesting, can we add a test for it on CI?

I'm surprised CI passed at all on the first try 😄 I added a test now, I hope it won't have a problem with the CI cache

src/compile.rs

messense · 2021-12-29T02:18:05Z

src/compile.rs

+        .open(&zig_linker)?;
+    #[cfg(not(target_family = "unix"))]
+    let mut custom_linker_file = File::create(&zig_linker)?;
+    writeln!(&mut custom_linker_file, "#!/bin/bash")?;


Hmm, this doesn't work on Windows apparently.

Here is an idea, can we provide a zig subcommand in maturin instead so it's easier to support non-unix platforms?

maturin zig cc

maturin zig c++

And it should be made compatible with both plain zig and the python ziglang package.

I haven't check if this also applies to windows, but on linux the linker needs to be the name or path to a binary, i.e. no cli arguments, so a subcommand wouldn't help. If there's the same rule on windows we could write a batch file or something, otherwise do CC="python -m ziglang cc" directly. An alternative could be writing python scripts, but again I don't know how well windows like python scripts as linkers.

Windows support would be nice, but isn't a blocker for merging imho

Agreed windows support is not a blocker.

I proposed maturin subcommand to simplify replacing libgcc_s with libunwind because I don't know how to do that in windows batch, a shell/batch script is still needed but it just forwards arguments to a maturin subcommand. 🤪

Oh for that it makes total sense!

tests/common/integration.rs

konstin · 2021-12-31T10:58:46Z

I rebase onto the bad zip file fix, I hope I didn't break anything

This allows compiling for a specified manylinux version by using `zig cc`s custom glibc targeting (https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html). You just add `--zig`, and it will target the policy you want, defaulting to manylinux2010. Example from ubuntu 20.04: ``` $ cargo run -- build -m test-crates/pyo3-pure/Cargo.toml -i python --no-sdist 2> /dev/null 🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.6 🐍 Not using a specific python interpreter (With abi3, an interpreter is only required on windows) 📖 Found type stub file at pyo3_pure.pyi 📦 Built wheel for abi3 Python ≥ 3.6 to /home/konsti/maturin/test-crates/pyo3-pure/target/wheels/pyo3_pure-2.1.2-cp36-abi3-manylinux_2_24_x86_64.whl $ cargo run -- build -m test-crates/pyo3-pure/Cargo.toml -i python --no-sdist --zig 2> /dev/null 🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.6 🐍 Not using a specific python interpreter (With abi3, an interpreter is only required on windows) 📖 Found type stub file at pyo3_pure.pyi 📦 Built wheel for abi3 Python ≥ 3.6 to /home/konsti/maturin/test-crates/pyo3-pure/target/wheels/pyo3_pure-2.1.2-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl ``` Maturin has to be installed with `maturin[zig]` to get a compatible zig version. Still missing: * musl (code is there, but untested) * pyproject.toml integration (do we want/need that?) * Documentation in the user guide

Set `TARGET_CC` and `TARGET_CXX` to support both cc-rs and cmake-rs

Disable Windows zig test for now

messense · 2021-12-31T15:52:41Z

I plan to release a new 0.12.x version from main branch before merging this.

Edit: v0.12.6 released.

messense · 2022-01-04T02:40:05Z

Note to myself:

Cross compile to macOS is blocked by cross compile to x86_64-macos get error: "ld64.lld: error: Unable to find library for -liconv" ziglang/zig#9725
Cross compile to Windows isn't worth the trouble as it requires to link to the Python .lib file that we don't have.

konstin requested a review from messense December 27, 2021 09:24

messense reviewed Dec 27, 2021

View reviewed changes

src/build_options.rs Outdated Show resolved Hide resolved

messense reviewed Dec 27, 2021

View reviewed changes

src/compile.rs Outdated Show resolved Hide resolved

messense reviewed Dec 28, 2021

View reviewed changes

src/compile.rs Outdated Show resolved Hide resolved

messense force-pushed the zig branch 3 times, most recently from 0346551 to 3015c7b Compare December 28, 2021 16:12

messense reviewed Dec 29, 2021

View reviewed changes

tests/common/integration.rs Show resolved Hide resolved

konstin force-pushed the zig branch from c470fee to 48f81e5 Compare December 31, 2021 10:56

konstin and others added 8 commits December 31, 2021 20:24

Add test for zig and small fixes

2fe49d8

Install zig in CI and clear CI cache

24a64b0

Add support for cross compiling with zig

69d079e

Add a zig cross compiling test to CI

def3b59

Add C++ support for --zig

7709b8b

Set `TARGET_CC` and `TARGET_CXX` to support both cc-rs and cmake-rs

Fix compiling on Windows

566abf5

Disable Windows zig test for now

Fixup

fa00117

konstin force-pushed the zig branch from 08e63ae to fa00117 Compare December 31, 2021 11:25

messense merged commit 10331c7 into main Dec 31, 2021

messense deleted the zig branch December 31, 2021 16:44

messense mentioned this pull request Jan 1, 2022

Windows support for zig based cross compiling #761

Closed

messense mentioned this pull request Feb 6, 2022

TODO list for 1.0 #796

Closed

6 tasks

RustyBamboo mentioned this pull request Feb 8, 2022

Remove manylinux docker RustyBamboo/lie#2

Open

messense mentioned this pull request May 4, 2022

Add hardcoded well-known sysconfigs for effortless cross compiling #896

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zig for manylinux compliance without docker #756

Zig for manylinux compliance without docker #756

konstin commented Dec 27, 2021

netlify bot commented Dec 27, 2021 •

edited

Loading

messense commented Dec 27, 2021

messense commented Dec 27, 2021

messense Dec 27, 2021

konstin Dec 27, 2021

messense Dec 29, 2021

messense Dec 27, 2021

konstin Dec 27, 2021

messense Dec 29, 2021

messense commented Dec 27, 2021 •

edited

Loading

messense Dec 27, 2021

konstin Dec 27, 2021

messense Dec 27, 2021

messense Dec 28, 2021

konstin commented Dec 27, 2021

messense Dec 29, 2021

messense Dec 29, 2021

konstin Dec 29, 2021

konstin Dec 29, 2021

messense Dec 29, 2021

konstin Dec 30, 2021

konstin commented Dec 31, 2021

messense commented Dec 31, 2021 •

edited

Loading

messense commented Jan 4, 2022 •

edited

Loading

Zig for manylinux compliance without docker #756

Zig for manylinux compliance without docker #756

Conversation

konstin commented Dec 27, 2021

netlify bot commented Dec 27, 2021 • edited Loading

messense commented Dec 27, 2021

messense commented Dec 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

messense commented Dec 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

konstin commented Dec 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

konstin commented Dec 31, 2021

messense commented Dec 31, 2021 • edited Loading

messense commented Jan 4, 2022 • edited Loading

netlify bot commented Dec 27, 2021 •

edited

Loading

messense commented Dec 27, 2021 •

edited

Loading

messense commented Dec 31, 2021 •

edited

Loading

messense commented Jan 4, 2022 •

edited

Loading