Possible problem with candle 0.8.0 - doesn't build on a GTX1650 (CI 75) nor a GTX1070 (CI 61)

[misureaudio]

Minimum reproducible example

cargo build --release --features cuda

ver 0.3.4 #950

Error

Compiling utoipa-swagger-ui v7.1.0
error: failed to run custom build command for candle-kernels v0.8.0 (https://github.com/EricLBuehler/candle.git?rev=e5685ce#e5685cee)
note: To improve backtraces for build dependencies, set the CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.

Caused by:
process didn't exit successfully: C:\Users\misur\Desktop\rustsrc\mistral.rs.0.3.4.0950\target\release\build\candle-kernels-b5b745df527c8066\build-script-build (exit code: 101)
--- stdout
cargo:rerun-if-changed=build.rs
cargo:rerun-if-changed=src/compatibility.cuh
cargo:rerun-if-changed=src/cuda_utils.cuh
cargo:rerun-if-changed=src/binary_op_macros.cuh
cargo:info=["/usr", "/usr/local/cuda", "/opt/cuda", "/usr/lib/cuda", "C:/Program Files/NVIDIA GPU Computing Toolkit", "C:/CUDA"]
cargo:rerun-if-env-changed=CUDA_COMPUTE_CAP
cargo:rustc-env=CUDA_COMPUTE_CAP=75
cargo:info=Builder { cuda_root: Some("C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6"), kernel_paths: ["src\affine.cu", "src\binary.cu", "src\cast.cu", "src\conv.cu", "src\fill.cu", "src\fused_rms_norm.cu", "src\fused_rope.cu", "src\indexing.cu", "src\kvconcat.cu", "src\quantized.cu", "src\reduce.cu", "src\sort.cu", "src\ternary.cu", "src\unary.cu"], watch: [], include_paths: ["src\binary_op_macros.cuh", "src\compatibility.cuh", "src\cuda_utils.cuh"], compute_cap: Some(75), out_dir: "C:\Users\misur\Desktop\rustsrc\mistral.rs.0.3.4.0950\target\release\build\candle-kernels-dce6dbf8224150c5\out", extra_args: [] }
cargo:rustc-env=CUDA_INCLUDE_DIR=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\include
cargo:rerun-if-changed=src\binary_op_macros.cuh
cargo:rerun-if-changed=src\compatibility.cuh
cargo:rerun-if-changed=src\cuda_utils.cuh
cargo:rerun-if-env-changed=NVCC_CCBIN
cargo:rerun-if-changed=src\affine.cu
cargo:rerun-if-changed=src\binary.cu
cargo:rerun-if-changed=src\conv.cu
cargo:rerun-if-changed=src\reduce.cu
cargo:rerun-if-changed=src\indexing.cu
cargo:rerun-if-changed=src\fused_rms_norm.cu
cargo:rerun-if-changed=src\cast.cu
cargo:rerun-if-changed=src\fill.cu
cargo:rerun-if-changed=src\sort.cu
cargo:rerun-if-changed=src\kvconcat.cu
cargo:rerun-if-changed=src\ternary.cu
cargo:rerun-if-changed=src\unary.cu
cargo:rerun-if-changed=src\fused_rope.cu
cargo:rerun-if-changed=src\quantized.cu
C:\Users\misur.cargo\git\checkouts\candle-c6a149c3b35a488f\e5685ce\candle-kernels\src\cast.cu(34): error: identifier "__nv_fp8_e4m3" is undefined
const __nv_fp8_e4m3 *inp,
^

Other information

Please specify:
Windows 11

• GPU or accelerator information

C:\Users\misur\Desktop\rustsrc\mistral.rs.0.3.4.0950>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Sep_12_02:55:00_Pacific_Daylight_Time_2024
Cuda compilation tools, release 12.6, V12.6.77
Build cuda_12.6.r12.6/compiler.34841621_0

C:\Users\misur>nvidia-smi
Sun Dec 1 14:24:53 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 566.14 Driver Version: 566.14 CUDA Version: 12.7 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Driver-Model | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce GTX 1650 ... WDDM | 00000000:01:00.0 Off | N/A |
| N/A 38C P8 3W / 40W | 0MiB / 4096MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

Latest commit or version

Which commit or version you ran with.

[EricLBuehler]

Hi @misureaudio - I just merged #960, which should fix this. I've checked compilation support down to compute cap 5.3!

[misureaudio]

Hi @EricLBuehler -

After updating ro rust 1.8.3, "cargo build --release --features cuda" is ok but "cargo install --path mistralrs-server --features cuda" fails:
"""
Adding utoipa v4.2.3 (available: v5.2.0)
Adding utoipa-swagger-ui v7.1.0 (available: v8.0.3)
Compiling mistralrs-core v0.3.4 (C:\Users\MATTIA\Desktop\rustsrc\mistral.rs.0.3.4.0960\mistralrs-core)
error[E0308]: mismatched types
--> mistralrs-core\src\pipeline\llg.rs:15:67
|
15 | let bt = toktrie_hf_tokenizers::ByteTokenizer::from_tokenizer(tokenizer)
| ---------------------------------------------------- ^^^^^^^^^ expected Tokenizer, found tokenizers::Tokenizer
| |
| arguments to this function are incorrect
|
= note: tokenizers::Tokenizer and Tokenizer have similar names, but are actually distinct types
note: tokenizers::Tokenizer is defined in crate tokenizers
--> C:\Users\MATTIA.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokenizers-0.20.4\src\tokenizer\mod.rs:408:1
|
408 | pub struct Tokenizer(
| ^^^^^^^^^^^^^^^^^^^^
note: Tokenizer is defined in crate tokenizers
--> C:\Users\MATTIA.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokenizers-0.21.0\src\tokenizer\mod.rs:408:1
|
408 | pub struct Tokenizer(
| ^^^^^^^^^^^^^^^^^^^^
= note: perhaps two different versions of crate tokenizers are being used?
note: associated function defined here
--> C:\Users\MATTIA.cargo\git\checkouts\llguidance-fc2ad2787f3486eb\cfef3df\toktrie_hf_tokenizers\src\lib.rs:72:12
|
72 | pub fn from_tokenizer(mut hft: Tokenizer) -> Result {
| ^^^^^^^^^^^^^^

For more information about this error, try rustc --explain E0308.
error: could not compile mistralrs-core (lib) due to 1 previous error
error: failed to compile mistralrs-server v0.3.4 (C:\Users\MATTIA\Desktop\rustsrc\mistral.rs.0.3.4.0960\mistralrs-server), intermediate artifacts can be found at C:\Users\MATTIA\Desktop\rustsrc\mistral.rs.0.3.4.0960\target.
To reuse those artifacts with a future compilation, set the environment variable CARGO_TARGET_DIR to that path.
"""

[EricLBuehler]

@misureaudio #975 should have fixed this, can you please check if it works for you?

[misureaudio]

@misureaudio #975 should have fixed this, can you please check if it works for you?

I'm no more able to complete the install phase:

error[E0422]: cannot find struct, variant or union type cudaExternalMemoryHandleDesc_st__bindgen_ty_1 in module sys
--> C:\Users\MATTIA.cargo\registry\src\index.crates.io-6f17d22bba15001f\cudarc-0.12.2\src\runtime\result.rs:988:26
|
988 | handle: sys::cudaExternalMemoryHandleDesc_st__bindgen_ty_1 {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: a union with a similar name exists: cudaExternalMemoryHandleDesc__bindgen_ty_1
|
::: C:\Users\MATTIA.cargo\registry\src\index.crates.io-6f17d22bba15001f\cudarc-0.12.2\src\runtime\sys\sys_12060.rs:4388:1
|
4388 | pub union cudaExternalMemoryHandleDesc__bindgen_ty_1 {
| ---------------------------------------------------- similarly named union cudaExternalMemoryHandleDesc__bindgen_ty_1 defined here

error[E0422]: cannot find struct, variant or union type cudaExternalMemoryHandleDesc_st__bindgen_ty_1__bindgen_ty_1 in module sys
--> C:\Users\MATTIA.cargo\registry\src\index.crates.io-6f17d22bba15001f\cudarc-0.12.2\src\runtime\result.rs:989:29
|
989 | win32: sys::cudaExternalMemoryHandleDesc_st__bindgen_ty_1__bindgen_ty_1 {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: a struct with a similar name exists: cudaExternalMemoryHandleDesc__bindgen_ty_1__bindgen_ty_1
|
::: C:\Users\MATTIA.cargo\registry\src\index.crates.io-6f17d22bba15001f\cudarc-0.12.2\src\runtime\sys\sys_12060.rs:4395:1
|
4395 | pub struct cudaExternalMemoryHandleDesc__bindgen_ty_1__bindgen_ty_1 {
| ------------------------------------------------------------------- similarly named struct cudaExternalMemoryHandleDesc__bindgen_ty_1__bindgen_ty_1 defined here

error[E0425]: cannot find function cudaImportExternalMemory in module sys
--> C:\Users\MATTIA.cargo\registry\src\index.crates.io-6f17d22bba15001f\cudarc-0.12.2\src\runtime\result.rs:997:14
|
997 | sys::cudaImportExternalMemory(external_memory.as_mut_ptr(), &handle_description)
| ^^^^^^^^^^^^^^^^^^^^^^^^ not found in sys

Compiling tokio-util v0.7.13
Compiling ref-cast v1.0.23
Compiling tokio-rustls v0.26.1
Compiling derive_builder_macro v0.20.2
Compiling serde_spanned v0.6.8
Compiling indexmap v2.7.0
Compiling either v1.13.0
Compiling toml_datetime v0.6.8
Compiling icu_normalizer v1.5.0
Compiling rayon v1.10.0
Compiling toml_edit v0.22.22
Compiling idna_adapter v1.2.0
Compiling idna v1.0.3
Compiling h2 v0.4.7
Compiling url v2.5.4
Compiling serde_urlencoded v0.7.1
Compiling proc-macro-crate v3.2.0
Compiling ug v0.0.2
Compiling gemm-common v0.17.1
Some errors have detailed explanations: E0422, E0425.
For more information about an error, try rustc --explain E0422.
error: could not compile cudarc (lib) due to 3 previous errors
warning: build failed, waiting for other jobs to finish...
Building [===================> ] 528/635: mistralrs-quant(build)