cross compile from linux to win produces a wheel that crashes silently at runtime #2297
Replies: 6 comments 4 replies
-
Have you tried reproduce it directly on Windows w/o cross compilation? |
Beta Was this translation helpful? Give feedback.
-
I did try NOTE: it's a corp windows machine it is quite constrained with no admin rights. This is one reason, besides convenience, I try to buid all wheels from linux. For example Btw thx for the fast reply. |
Beta Was this translation helpful? Give feedback.
-
I keep searching and just found that if I cross compile without
mod black_scholes;
mod implied_vol;
// use pyo3_polars::PolarsAllocator;
use pyo3::types::{PyModule, PyModuleMethods};
use pyo3::{pymodule, Bound, PyResult};
#[pymodule]
fn _rust(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add("__version__", env!("CARGO_PKG_VERSION"))?;
Ok(())
}
// #[global_allocator]
// static ALLOC: PolarsAllocator = PolarsAllocator::new(); It seems PolarAllocator is defined there in repo pyo3-polars. What could be so toxic in it that produces a silent crash ? |
Beta Was this translation helpful? Give feedback.
-
Ref issue pola-rs/pyo3-polars#117 |
Beta Was this translation helpful? Give feedback.
-
Now I'm trying to cross compile from linux to windows with target I do it inside a docker container. Compilation fails with error:
This container runs inside a corp network and internet is only accessible through a proxy. How to let cargo to go fetch data through the proxy defined by these env variables ? if possible ? |
Beta Was this translation helpful? Give feedback.
-
I just tried run the docker container outside corp: FROM python:3.12-slim-bullseye
# ref https://learn.microsoft.com/en-us/vcpkg/users/platforms/mingw#mingw-cross
RUN apt-get update -y && \
apt-get install -y curl wget vim build-essential
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH=/root/.cargo/bin:$PATH
RUN rustup target add x86_64-pc-windows-msvc
RUN python -m pip install --no-cache-dir -U pip && \
python -m pip install --no-cache-dir cargo-xwin && \
python -m pip install --no-cache-dir maturin
ENV XWIN_CACHE_DIR=/io/cache
WORKDIR /io
COPY ./maturin-build-win-msvc.sh .
ENTRYPOINT [ "/io/maturin-build-win-msvc.sh" ] where #! /bin/bash
maturin build -m /io/Cargo.toml --release --target x86_64-pc-windows-msvc --out /io/dist/docker-msvc But I get this error: docker run --rm -v "$(pwd)":/io builder-win-msvc:local
Updating crates.io index
Downloading crates ...
Downloaded ahash v0.8.9
Downloaded allocator-api2 v0.2.16
Downloaded array-init-cursor v0.2.0
Downloaded atoi_simd v0.15.6
Downloaded autocfg v1.1.0
Downloaded argminmax v0.6.2
Downloaded base64 v0.22.1
Downloaded aho-corasick v1.1.2
Downloaded cc v1.1.31
Downloaded bitflags v1.3.2
Downloaded bytemuck_derive v1.5.0
Downloaded byteorder v1.5.0
Downloaded regex-automata v0.4.5
Downloaded shlex v1.3.0
Downloaded streaming-decompression v0.1.2
Downloaded equivalent v1.0.1
Downloaded unindent v0.2.3
Downloaded windows-targets v0.48.5
Downloaded wasm-bindgen-shared v0.2.91
Downloaded windows-targets v0.52.6
Downloaded scopeguard v1.2.0
Downloaded home v0.5.9
Downloaded glob v0.3.1
Downloaded fallible-streaming-iterator v0.1.9
Downloaded heck v0.5.0
Downloaded crossbeam-deque v0.8.5
Downloaded cfg-if v1.0.0
Downloaded strength_reduce v0.2.4
Downloaded iter-read v0.3.1
Downloaded indoc v2.0.4
Downloaded dyn-clone v1.0.16
Downloaded castaway v0.2.3
Downloaded rustversion v1.0.14
Downloaded itoa v1.0.10
Downloaded num-integer v0.1.46
Downloaded target-features v0.1.5
Downloaded static_assertions v1.1.0
Downloaded thiserror v1.0.57
Downloaded version_check v0.9.4
Downloaded bumpalo v3.15.1
Downloaded either v1.13.0
Downloaded thiserror-impl v2.0.0
Downloaded wasm-bindgen-macro v0.2.91
Downloaded wasm-bindgen-macro-support v0.2.91
Downloaded option-pricing v0.1.4
Downloaded streaming-iterator v0.1.9
Downloaded stacker v0.1.15
Downloaded polars-error v0.43.1
Downloaded thiserror-impl v1.0.57
Downloaded polars-schema v0.43.1
Downloaded recursive v0.1.1
Downloaded polars-ffi v0.43.1
Downloaded xxhash-rust v0.8.10
Downloaded pyo3-polars-derive v0.11.0
Downloaded recursive-proc-macro-impl v0.1.1
Downloaded memoffset v0.9.0
Downloaded simdutf8 v0.1.4
Downloaded thiserror v2.0.0
Downloaded getrandom v0.2.12
Downloaded lock_api v0.4.11
Downloaded multiversion-macros v0.7.3
Downloaded log v0.4.20
Downloaded wasi v0.11.0+wasi-snapshot-preview1
Downloaded multiversion v0.7.3
Downloaded bitflags v2.4.2
Downloaded memmap2 v0.7.1
Downloaded bytes v1.8.0
Downloaded wasm-bindgen-backend v0.2.91
Downloaded smallvec v1.13.1
Downloaded pyo3-polars v0.17.0
Downloaded polars-row v0.43.1
Downloaded psm v0.1.23
Downloaded zerocopy-derive v0.7.32
Downloaded unicode-ident v1.0.12
Downloaded strum_macros v0.26.4
Downloaded ryu v1.0.17
Downloaded target-lexicon v0.12.16
Downloaded serde_derive v1.0.214
Downloaded serde v1.0.214
Downloaded rand_chacha v0.3.1
Downloaded pyo3-macros v0.22.5
Downloaded bytemuck v1.14.3
Downloaded once_cell v1.19.0
Downloaded crossbeam-utils v0.8.20
Downloaded fast-float v0.2.0
Downloaded crossbeam-epoch v0.9.18
Downloaded percent-encoding v2.3.1
Downloaded planus v0.3.1
Downloaded polars-compute v0.43.1
Downloaded polars-mem-engine v0.43.1
Downloaded polars-utils v0.43.1
Downloaded ppv-lite86 v0.2.17
Downloaded rand_core v0.6.4
Downloaded pyo3-build-config v0.22.5
Downloaded zerocopy v0.7.32
Downloaded wasm-bindgen v0.2.91
Downloaded compact_str v0.8.0
Downloaded redox_syscall v0.4.1
Downloaded ethnum v1.5.0
Downloaded num-traits v0.2.18
Downloaded parking_lot_core v0.9.9
Downloaded parking_lot v0.12.1
Downloaded polars-arrow-format v0.1.0
Downloaded polars-expr v0.43.1
Downloaded quote v1.0.35
Downloaded parquet-format-safe v0.2.4
Downloaded polars-lazy v0.43.1
Downloaded proc-macro2 v1.0.89
Downloaded python3-dll-a v0.2.9
Downloaded rand_distr v0.4.3
Downloaded rayon-core v1.12.1
Downloaded indexmap v2.2.3
Downloaded pyo3-macros-backend v0.22.5
Downloaded js-sys v0.3.68
Downloaded memchr v2.7.1
Downloaded pyo3-ffi v0.22.5
Downloaded rand v0.8.5
Downloaded num-bigint v0.4.4
Downloaded raw-cpuid v11.0.1
Downloaded polars v0.43.1
Downloaded libm v0.2.8
Downloaded itertools v0.13.0
Downloaded hashbrown v0.14.5
Downloaded syn v1.0.109
Downloaded syn v2.0.87
Downloaded polars-io v0.43.1
Downloaded serde-pickle v1.1.1
Downloaded polars-ops v0.43.1
Downloaded portable-atomic v1.9.0
Downloaded rayon v1.9.0
Downloaded polars-parquet v0.43.1
Downloaded regex-syntax v0.8.5
Downloaded chrono v0.4.34
Downloaded regex v1.10.3
Downloaded polars-plan v0.43.1
Downloaded polars-core v0.43.1
Downloaded polars-arrow v0.43.1
Downloaded windows_aarch64_gnullvm v0.52.6
Downloaded windows_aarch64_gnullvm v0.48.5
Downloaded windows_x86_64_gnullvm v0.52.6
Downloaded windows_x86_64_gnullvm v0.48.5
Downloaded pyo3 v0.22.5
Downloaded libc v0.2.153
Downloaded windows_i686_gnullvm v0.52.6
Downloaded windows_aarch64_msvc v0.48.5
Downloaded windows_aarch64_msvc v0.52.6
Downloaded windows_x86_64_msvc v0.52.6
Downloaded windows_x86_64_msvc v0.48.5
Downloaded windows_x86_64_gnu v0.48.5
Downloaded windows_i686_gnu v0.52.6
Downloaded windows_x86_64_gnu v0.52.6
Downloaded windows_i686_msvc v0.48.5
Downloaded windows_i686_gnu v0.48.5
Downloaded windows_i686_msvc v0.52.6
Downloaded winapi v0.3.9
Downloaded windows-sys v0.52.0
Downloaded winapi-i686-pc-windows-gnu v0.4.0
Downloaded winapi-x86_64-pc-windows-gnu v0.4.0
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.8
🐍 Not using a specific python interpreter (Automatically generating windows import library)
Compiling cfg-if v1.0.0
Compiling libm v0.2.8
Compiling serde v1.0.214
Compiling num-traits v0.2.18
Compiling crossbeam-utils v0.8.20
Compiling libc v0.2.153
Compiling rayon-core v1.12.1
Compiling either v1.13.0
Compiling once_cell v1.19.0
Compiling memchr v2.7.1
Compiling winapi v0.3.9
Compiling psm v0.1.23
Compiling ahash v0.8.9
Compiling windows_x86_64_msvc v0.48.5
The following warnings were emitted during compilation:
warning: psm@0.1.23: Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang-cl` installed?
warning: psm@0.1.23: Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang-cl` installed?
error: failed to run custom build command for `psm v0.1.23`
Caused by:
process didn't exit successfully: `/io/target/release/build/psm-627b4451f7640b8d/build-script-build` (exit status: 1)
--- stdout
OPT_LEVEL = Some(3)
TARGET = Some(x86_64-pc-windows-msvc)
OUT_DIR = Some(/io/target/x86_64-pc-windows-msvc/release/build/psm-fabedde2f730ef42/out)
cargo:rerun-if-env-changed=VCINSTALLDIR
VCINSTALLDIR = None
HOST = Some(x86_64-unknown-linux-gnu)
cargo:rerun-if-env-changed=CC_x86_64-pc-windows-msvc
CC_x86_64-pc-windows-msvc = None
cargo:rerun-if-env-changed=CC_x86_64_pc_windows_msvc
CC_x86_64_pc_windows_msvc = Some(clang-cl)
cargo:rerun-if-env-changed=CC_KNOWN_WRAPPER_CUSTOM
CC_KNOWN_WRAPPER_CUSTOM = None
RUSTC_WRAPPER = None
cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
cargo:warning=Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang-cl` installed?
cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
CRATE_CC_NO_DEFAULTS = None
CARGO_CFG_TARGET_FEATURE = Some(cmpxchg16b,fxsr,sse,sse2,sse3)
DEBUG = Some(false)
cargo:rerun-if-env-changed=CFLAGS_x86_64-pc-windows-msvc
CFLAGS_x86_64-pc-windows-msvc = None
cargo:rerun-if-env-changed=CFLAGS_x86_64_pc_windows_msvc
CFLAGS_x86_64_pc_windows_msvc = Some(--target=x86_64-pc-windows-msvc -Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/root/.cache/cargo-xwin/xwin/crt/include /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/ucrt /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/um /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/shared )
cargo:rerun-if-env-changed=CC_SHELL_ESCAPED_FLAGS
CC_SHELL_ESCAPED_FLAGS = None
cargo:rustc-check-cfg=cfg(switchable_stack,asm,link_asm)
cargo:rustc-cfg=asm
cargo:rustc-cfg=link_asm
cargo:warning=Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `clang-cl` installed?
--- stderr
error occurred: Failed to find tool. Is `clang-cl` installed?
warning: build failed, waiting for other jobs to finish...
💥 maturin failed
Caused by: Failed to build a native library through cargo
Caused by: Cargo build finished with "exit status: 101": `env -u CARGO AR_x86_64_pc_windows_msvc="llvm-lib" BINDGEN_EXTRA_CLANG_ARGS_x86_64_pc_windows_msvc="-I/root/.cache/cargo-xwin/xwin/crt/include -I/root/.cache/cargo-xwin/xwin/sdk/include/ucrt -I/root/.cache/cargo-xwin/xwin/sdk/include/um -I/root/.cache/cargo-xwin/xwin/sdk/include/shared" CARGO_ENCODED_RUSTFLAGS="-C\u{1f}linker-flavor=lld-link\u{1f}-Lnative=/root/.cache/cargo-xwin/xwin/crt/lib/x86_64\u{1f}-Lnative=/root/.cache/cargo-xwin/xwin/sdk/lib/um/x86_64\u{1f}-Lnative=/root/.cache/cargo-xwin/xwin/sdk/lib/ucrt/x86_64" CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER="lld-link" CC_x86_64_pc_windows_msvc="clang-cl" CFLAGS_x86_64_pc_windows_msvc="--target=x86_64-pc-windows-msvc -Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/root/.cache/cargo-xwin/xwin/crt/include /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/ucrt /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/um /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/shared " CL_FLAGS="--target=x86_64-pc-windows-msvc -Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/root/.cache/cargo-xwin/xwin/crt/include /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/ucrt /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/um /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/shared " CMAKE_GENERATOR="Ninja" CMAKE_SYSTEM_NAME="Windows" CMAKE_TOOLCHAIN_FILE_x86_64_pc_windows_msvc="/root/.cache/cargo-xwin/cmake/x86_64-pc-windows-msvc-toolchain.cmake" CXXFLAGS_x86_64_pc_windows_msvc="--target=x86_64-pc-windows-msvc -Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/root/.cache/cargo-xwin/xwin/crt/include /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/ucrt /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/um /imsvc/root/.cache/cargo-xwin/xwin/sdk/include/shared " CXX_x86_64_pc_windows_msvc="clang-cl" PATH="/root/.cargo/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.cache/cargo-xwin" RCFLAGS="-I/root/.cache/cargo-xwin/xwin/crt/include -I/root/.cache/cargo-xwin/xwin/sdk/include/ucrt -I/root/.cache/cargo-xwin/xwin/sdk/include/um -I/root/.cache/cargo-xwin/xwin/sdk/include/shared" TARGET_AR="llvm-lib" TARGET_CC="clang-cl" TARGET_CXX="clang-cl" "cargo" "rustc" "--target" "x86_64-pc-windows-msvc" "--message-format" "json-render-diagnostics" "--manifest-path" "/io/Cargo.toml" "--release" "--lib"`
/io/maturin-build-win-msvc.sh: line 4: docker-msvc: command not found I'm puzzled as I read in the cargo-xwin prerequisite that for a Rust only project (my case) no clang or llvm is necessary. BTW: Unsurprising the corp proxy issue has gone (Cf comment above). So I need find a way to solve these 2 separate issues. |
Beta Was this translation helpful? Give feedback.
-
Bug Description
The cross compile from linux to windows of package polars_plugin_option_pricing produces a wheel.
This wheel can be pip installed on windows.
So far no apparent error or warning.
But at runtime, the program exits silently without error message when reaching the compiled module:
polars_plugin_option_pricing._rust
.I tried to build the wheel on a linux host (ubuntu) and from inside a docker container (reproducible below).
I build the docker container on a ubuntu host and a redhat8 host.
Same result :-(
How to debug this ?
Is there a tested template of docker based cross compilation from linux to win somewhere in the docs ?
Note: the native and manylinux compilations work and produce working wheels.
Your maturin version (
maturin --version
)1.7.4
Your Python version (
python -V
)3.12
Your pip version (
pip -V
)24.3.1
What bindings you're using
pyo3
Does
cargo build
work?If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash
/
)?Steps to Reproduce
Commands on linux host:
docker run logs:
Commands on win host, in conda env with py3.12:
pip install polars_plugin_option_pricing-0.1.0-cp38-abi3-win_amd64.whl # from repo root python test-run-bs.py
This command outputs:
The program should continue (cf demo notebook run-bs.ipynb), but exits without showing any indication what the error was...
Beta Was this translation helpful? Give feedback.
All reactions