Build and test with CUDA 13.0.0 #1273

jameslamb · 2025-08-22T03:22:16Z

Contributes to rapidsai/build-planning#208

uses CUDA 13.0.0 to build and test
adds CUDA 13 devcontainers
moves some dependency pins:
- cuda-python: >=12.9.2 (CUDA 12), >=13.0.1 (CUDA 13)
- cupy: >=13.6.0

Contributes to rapidsai/build-planning#68

updates to CUDA 13 dependencies in fallback entries in dependencies.yaml matrices (i.e., the ones that get written to pyproject.toml in source control)

Notes for Reviewers

This switches GitHub Actions workflows to the cuda13.0 branch from here: rapidsai/shared-workflows#413

A future round of PRs will revert that back to branch-25.10, once all of RAPIDS supports CUDA 13.

What about Go, Java, and Rust?

This PR expands building / testing for those bindings to cover CUDA 13, but more changes probably need to be made to support distributing packages for those.

Proposing deferring that to follow-ups:

copy-pr-bot · 2025-08-22T03:22:20Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

jameslamb · 2025-08-22T03:58:08Z

/ok to test

jameslamb · 2025-08-22T04:38:59Z

The first runs all failed with some form of "need RAFT packages", so nothing else learned from this yet.

build link: https://github.com/rapidsai/cuvs/actions/runs/17145663792/job/48641578548

Will update this once rapidsai/raft#2787 is merged and we have RAFT packages.

jameslamb · 2025-08-22T17:02:39Z

/ok to test

jameslamb · 2025-08-22T19:41:48Z

Happy to say that wheels are looking happy, so libcuvs.so CUDA 13 support is probably close!

But some benchmarks are failing to compile, like this:

 │ │ [680/773] Building CXX object bench/ann/CMakeFiles/HNSWLIB_ANN_BENCH.dir/src/hnswlib/hnswlib_benchmark.cpp.o
 │ │ FAILED: [code=1] bench/ann/CMakeFiles/HNSWLIB_ANN_BENCH.dir/src/hnswlib/hnswlib_benchmark.cpp.o 
 │ │ sccache $BUILD_PREFIX/bin/aarch64-conda-linux-gnu-c++ -DANN_BENCH_BUILD_MAIN> -DBENCHMARK_STATIC_DEFINE -DCUVS_ANN_BENCH_USE_HNSWLIB=CUVS_ANN_BENCH_USE_HNSWLIB -I$SRC_DIR/cpp/include -I$SRC_DIR/cpp/build/_deps/benchmark-src/include -I$SRC_DIR/cpp/build/_deps/hnswlib-src -isystem $BUILD_PREFIX/targets/sbsa-linux/include/cccl -isystem /opt/conda/include -isystem $BUILD_PREFIX/include -fvisibility-inlines-hidden -fmessage-length=0 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include   -I$PREFIX/targets/sbsa-linux/include -I$PREFIX/targets/sbsa-linux/include/cccl -I$BUILD_PREFIX/targets/sbsa-linux/include -I$BUILD_PREFIX/targets/sbsa-linux/include/cccl -O3 -DNDEBUG -std=gnu++17 -fPIE -pthread -fopenmp "-ffile-prefix-map=$PREFIX/=''" -MD -MT bench/ann/CMakeFiles/HNSWLIB_ANN_BENCH.dir/src/hnswlib/hnswlib_benchmark.cpp.o -MF bench/ann/CMakeFiles/HNSWLIB_ANN_BENCH.dir/src/hnswlib/hnswlib_benchmark.cpp.o.d -o bench/ann/CMakeFiles/HNSWLIB_ANN_BENCH.dir/src/hnswlib/hnswlib_benchmark.cpp.o -c $SRC_DIR/cpp/bench/ann/src/hnswlib/hnswlib_benchmark.cpp
 │ │ In file included from $SRC_DIR/cpp/bench/ann/src/hnswlib/hnswlib_wrapper.h:20,
 │ │                  from $SRC_DIR/cpp/bench/ann/src/hnswlib/hnswlib_benchmark.cpp:18:
 │ │ $SRC_DIR/cpp/bench/ann/src/hnswlib/../common/util.hpp: In function 'auto cuvs::bench::cuda_info()':
 │ │ $SRC_DIR/cpp/bench/ann/src/hnswlib/../common/util.hpp:451:64: error: 'struct cudaDeviceProp' has no member named 'clockRate'
 │ │   451 |   props.emplace_back("gpu_sm_freq", std::to_string(device_prop.clockRate * 1e3));
 │ │       |                                                                ^~~~~~~~~
 │ │ $SRC_DIR/cpp/bench/ann/src/hnswlib/../common/util.hpp:452:65: error: 'struct cudaDeviceProp' has no member named 'memoryClockRate'
 │ │   452 |   props.emplace_back("gpu_mem_freq", std::to_string(device_prop.memoryClockRate * 1e3));
 │ │       |                                                                 ^~~~~~~~~~~~~~~

(conda-cpp-build link)

jameslamb · 2025-08-22T20:44:07Z

Updated to latest branch-25.10 and pushed this patch from @robertmaynard : d122856

Let's see how that goes. Should get faster feedback now that we've mostly filled up the sccache caches for CUDA 13 builds here 😁

jameslamb · 2025-08-22T20:44:11Z

/ok to test

jameslamb · 2025-08-22T21:50:44Z

cpp/bench/ann/src/common/util.hpp

+  int clockRate       = 0;
+  int memoryClockRate = 0;
+  err_code            = cudaDeviceGetAttribute(&clockRate, cudaDevAttrClockRate, dev);
+  err_code            = cudaDeviceGetAttribute(&memoryClockRate, cudaDevAttrMemoryClockRate, dev);


The Java builds are failing like this:

[INFO] ------------------------------------------------------------- Error: /__w/cuvs/cuvs/java/cuvs-java/src/main/java22/com/nvidia/cuvs/internal/common/Util.java:[192,23] cannot find symbol symbol: method cudaGetDeviceProperties_v2(java.lang.foreign.MemorySegment,int) location: class com.nvidia.cuvs.internal.common.Util [INFO] 1 error [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 13.340 s [INFO] Finished at: 2025-08-22T21:20:16Z [INFO] ------------------------------------------------------------------------ Error: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.13.0:compile (compile-java-22) on project cuvs-java: Compilation failure Error: /__w/cuvs/cuvs/java/cuvs-java/src/main/java22/com/nvidia/cuvs/internal/common/Util.java:[192,23] cannot find symbol Error: symbol: method cudaGetDeviceProperties_v2(java.lang.foreign.MemorySegment,int) Error: location: class com.nvidia.cuvs.internal.common.Util

(build link)

@mythrocks do the Java bindings need to be updated? If so could you do that and push them to this PR?

I don't have Java set up in my development environment.

Hmm. This is exactly the sort of error that I'm running into in my work on the portable fat-jar PR. Except, I'm building on 12.9.

I think this is an artifact of some strange jextract behaviour.

Thanks for the confirmation; so I'm not imagining it.

I'll dig into this more.

As a first step I would try to see if cudaGetDeviceProperties_v2(java.lang.foreign.MemorySegment,int) is defined and what signature it has.
If this passes on 12.9 but fails consistently in 13.0, I suspect this is a header issue.
We generate functions bindings for Java using jextract, but this may or may not be the issue. I suspect CUDA headers #define cudaGetDeviceProperties as "something", and that "something" as a different signature in 12 vs 13. Just a guess, let me see if I can validate it.

(Java, as the other non-C langs, can bind to C functions but of course not via macros - that's why we need to call cudaGetDeviceProperties_v2 directly -- again, modulo my guesswork being correct and cudaGetDeviceProperties is indeed a macro)

Yep, I see #define cudaGetDeviceProperties cudaGetDeviceProperties_v2 in my cuda_runtime_api.h. Now let me see how it looks like in 13.0

Looks like this was broken by #1267

Error: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.13.0:compile (compile-java-22) on project cuvs-java: Compilation failure Error: /__w/cuvs/cuvs/java/cuvs-java/src/main/java22/com/nvidia/cuvs/internal/GPUInfoProviderImpl.java:[55,25] cannot find symbol Error: symbol: method cudaGetDeviceProperties_v2(java.lang.foreign.MemorySegment,int) Error: location: class com.nvidia.cuvs.internal.GPUInfoProviderImpl.AvailableGpuInitializer Error: Error: -> [Help 1]

(conda-java-tests link)

I'll try to push a fix.

I tried something here: 104508b

Would appreciate your input on it @ldematte @mythrocks . If you look and that seems wrong, please feel free to push to my branch here. I'd really like to get this PR in as soon as possible, so we can move on to other projects in RAPIDS that depend on cuvs.

Ok one more commit (8530e22) because I missed that that was a static method (sorry, I haven't written anything for the JVM in a while 😅)

And looks like that work!

I see this as a minor extension of logic you all were already ok with putting in temporarily, so I'm planning to merge this once CI passes. Thanks again for all the help!

I'm not sure this was borked by #1267, exactly. I thought your change here would've dealt with it.

That said, your fix seems right. +1 to checking this in on passing CI.

Thanks!

I think it was.... we found in this thread that cudaGetDeviceProperties_v2() could not be used unconditionally, and #1267 introduces code that uses it unconditionally:

cuvs/java/cuvs-java/src/main/java22/com/nvidia/cuvs/internal/GPUInfoProviderImpl.java

Lines 55 to 56 in e02b7d1

returnValue = cudaGetDeviceProperties_v2(deviceProp, i);

checkCudaError(returnValue, "cudaGetDeviceProperties_v2");

jameslamb · 2025-08-25T13:54:02Z

/ok to test

jameslamb · 2025-08-25T13:56:22Z

/ok to test

docs/source/build.rst

python/cuvs/pyproject.toml

msarahan

approving for code-owners

jameslamb · 2025-08-27T22:08:02Z

Merging in recent changes, I now see build failures for conda-cpp-build jobs on CUDA 13:

 │ │ $SRC_DIR/cpp/include/cuvs/neighbors/common.hpp(465): error #20011-D: calling a __host__ function("cuvs::neighbors::filtering::base_filter::~base_filter()") from a __host__ __device__ function("cuvs::neighbors::filtering::base_filter::~base_filter [subobject]") is not allowed
 │ │ Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"
 │ │ 1 error detected in the compilation of "$SRC_DIR/cpp/src/neighbors/cagra_search_uint8.cu".
 │ │ [4/766] Building CUDA object CMakeFiles/cuvs-cagra-search.dir/src/neighbors/cagra_search_int8.cu.o
 │ │ FAILED: [code=1] CMakeFiles/cuvs-cagra-search.dir/src/neighbors/cagra_search_int8.cu.o

(build link)

I strongly suspect that's a result of #1211. @jinsolp could you please take a look? There are CUDA 13 devcontainers on this branch that you might find helpful for investigating.

mythrocks

Approving for Java-owners. @jameslamb: Good work, mate.

divyegala

Just a single comment, pre-approving. Thanks!

docs/source/build.rst

robertmaynard · 2025-08-28T14:56:32Z

Merging in recent changes, I now see build failures for conda-cpp-build jobs on CUDA 13:

 │ │ $SRC_DIR/cpp/include/cuvs/neighbors/common.hpp(465): error #20011-D: calling a __host__ function("cuvs::neighbors::filtering::base_filter::~base_filter()") from a __host__ __device__ function("cuvs::neighbors::filtering::base_filter::~base_filter [subobject]") is not allowed
 │ │ Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"
 │ │ 1 error detected in the compilation of "$SRC_DIR/cpp/src/neighbors/cagra_search_uint8.cu".
 │ │ [4/766] Building CUDA object CMakeFiles/cuvs-cagra-search.dir/src/neighbors/cagra_search_int8.cu.o
 │ │ FAILED: [code=1] CMakeFiles/cuvs-cagra-search.dir/src/neighbors/cagra_search_int8.cu.o

(build link)

I strongly suspect that's a result of #1211. @jinsolp could you please take a look? There are CUDA 13 devcontainers on this branch that you might find helpful for investigating.

Yes, absolutely caused by #1211, since it removed the changes I made in #1219 to allow for CUDA 13 compilation.

jameslamb · 2025-08-28T14:58:59Z

Thanks for looking and confirming @robertmaynard ! If you know how to fix this, could you push a patch to my branch here?

mythrocks · 2025-08-29T04:22:02Z

CI seems to have borked on an intermittent test failure. I'm going to re-kick it.

jameslamb · 2025-08-29T13:37:28Z

Thanks very much @mythrocks for re-running CI and to you and everyone else for your help on this!!!!

I'm going to merge this, @ me any time if it seems like it caused issues.

jameslamb · 2025-08-29T13:37:33Z

/merge

Contributes to rapidsai/build-planning#208 * uses CUDA 13.0.0 to build and test (using the same patterns from the `cuvs-java` tests, in rapidsai/cuvs#1273) ## Notes for Reviewers This switches GitHub Actions workflows to the `cuda13.0` branch from here: rapidsai/shared-workflows#413 A future round of PRs will revert that back to `branch-25.10`, once all of RAPIDS supports CUDA 13. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Jake Awe (https://github.com/AyodeAwe) - Ben Frederickson (https://github.com/benfred) - rhdong (https://github.com/rhdong) URL: #20

…based on symbol presence (#1323) In #1273 we addressed a signature change in CUDA 13 by binding different symbols based on a environment variable, `RAPIDS_CUDA_MAJOR`. That works, but forces users of cuvs-java with CUDA 12 to define this environment variable. This PR improves on it by making the symbol lookup dynamic, looking for the CUDA 13 symbol name, and falling back to the CUDA 12 exported name if we fail to locate the first. Authors: - Lorenzo Dematté (https://github.com/ldematte) Approvers: - MithunR (https://github.com/mythrocks) URL: #1323

Build and test with CUDA 13.0.0

5cc2c3a

github-project-automation bot moved this to Todo in Vector Search, ML, & Data Mining Release Board Aug 22, 2025

github-project-automation bot added this to Vector Search, ML, & Data Mining Release Board Aug 22, 2025

jameslamb added non-breaking Introduces a non-breaking change improvement Improves an existing functionality labels Aug 22, 2025

update dependencies.yaml, add more CUDA 13 jobs

4b22d51

jameslamb mentioned this pull request Aug 22, 2025

Add CUDA 13.0 Support across all of RAPIDS rapidsai/build-planning#208

Open

jameslamb mentioned this pull request Aug 22, 2025

Build and test with CUDA 13.0.0 rapidsai/cuml#7128

Merged

jameslamb added 2 commits August 22, 2025 11:23

Merge branch 'branch-25.10' into cuda-13.0.0

f9e36e1

fix docs, add comments on Go, Java, and Rust workflows

be8e3b6

This was referenced Aug 22, 2025

[Go] CUDA 13 support #1275

Closed

[Java] CUDA 13 support #1276

Open

[Rust] CUDA 13 support #1277

Open

jameslamb added 2 commits August 22, 2025 15:38

Merge branch 'branch-25.10' of github.com:rapidsai/cuvs into cuda-13.0.0

a4cdd76

use different code paths for clockRate

d122856

jameslamb commented Aug 22, 2025

View reviewed changes

ldematte mentioned this pull request Aug 25, 2025

[WIP][JAVA] Fix cudaGetDeviceProperties symbol name to bind correctly for CUDA > 12 #1280

Closed

jameslamb added 3 commits August 25, 2025 08:50

do not fail-fast builds

d3a905f

Merge branch 'branch-25.10' of github.com:rapidsai/cuvs into cuda-13.0.0

8442834

try cudaGetDeviceProperties() patch

5ec0d48

define RAPIDS_CUDA_MAJOR

492852f

This was referenced Aug 26, 2025

Build and test with CUDA 13.0.0 rapidsai/integration#795

Merged

Build and test with CUDA 13.0.0 rapidsai/docker#782

Merged

divyegala reviewed Aug 27, 2025

View reviewed changes

docs/source/build.rst Show resolved Hide resolved

python/cuvs/pyproject.toml Show resolved Hide resolved

jameslamb added 2 commits August 27, 2025 00:41

Merge branch 'branch-25.10' into cuda-13.0.0

d190409

fix docs

3b6b8d6

jameslamb requested a review from divyegala August 27, 2025 05:44

msarahan approved these changes Aug 27, 2025

View reviewed changes

fix conflicts

5fab31d

mythrocks approved these changes Aug 27, 2025

View reviewed changes

benfred approved these changes Aug 27, 2025

View reviewed changes

divyegala approved these changes Aug 28, 2025

View reviewed changes

docs/source/build.rst Show resolved Hide resolved

jameslamb and others added 6 commits August 28, 2025 13:50

fix CUDA 13 compilation issue

16a133f

fix compile error

7a9851b

update docs

d4e3154

Merge branch 'branch-25.10' into cuda-13.0.0

36508fa

use patched cudaGetDeviceProperties()

104508b

oops, cudaGetDeviceProperties() is static

8530e22

rapids-bot bot merged commit 55985fe into rapidsai:branch-25.10 Aug 29, 2025
156 of 164 checks passed

github-project-automation bot moved this from Todo to Done in Vector Search, ML, & Data Mining Release Board Aug 29, 2025

jameslamb deleted the cuda-13.0.0 branch August 29, 2025 13:37

jameslamb mentioned this pull request Sep 2, 2025

Build and test with CUDA 13.0 rapidsai/cuvs-lucene#20

Merged

ldematte mentioned this pull request Sep 9, 2025

[Java] Make cudaGetDeviceProperties compatible with CUDA 12 and 13 based on symbol presence #1323

Merged

	returnValue = cudaGetDeviceProperties_v2(deviceProp, i);
	checkCudaError(returnValue, "cudaGetDeviceProperties_v2");

Build and test with CUDA 13.0.0 #1273

Build and test with CUDA 13.0.0 #1273

Uh oh!

Conversation

jameslamb commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes for Reviewers

What about Go, Java, and Rust?

Uh oh!

copy-pr-bot bot commented Aug 22, 2025

Uh oh!

jameslamb commented Aug 22, 2025

Uh oh!

jameslamb commented Aug 22, 2025

Uh oh!

jameslamb commented Aug 22, 2025

Uh oh!

jameslamb commented Aug 22, 2025

Uh oh!

jameslamb commented Aug 22, 2025

Uh oh!

jameslamb commented Aug 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mythrocks Aug 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jameslamb Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jameslamb commented Aug 25, 2025

Uh oh!

jameslamb commented Aug 25, 2025

Uh oh!

Uh oh!

Uh oh!

msarahan left a comment

Choose a reason for hiding this comment

Uh oh!

jameslamb commented Aug 27, 2025

Uh oh!

mythrocks left a comment

Choose a reason for hiding this comment

Uh oh!

divyegala left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

robertmaynard commented Aug 28, 2025

Uh oh!

jameslamb commented Aug 28, 2025

Uh oh!

mythrocks commented Aug 29, 2025

Uh oh!

jameslamb commented Aug 29, 2025

Uh oh!

jameslamb commented Aug 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

jameslamb commented Aug 22, 2025 •

edited

Loading

mythrocks Aug 23, 2025 •

edited

Loading

jameslamb Aug 29, 2025 •

edited

Loading