Relax `libcufile` dependency for old CTK 12 on ARM #930

jakirkham · 2025-08-22T04:25:01Z

Following up on this thread: #905 (comment)

The CUDA Toolkit (CTK) added cuFile for ARM in CTK 12.2. As a result always requiring libcufile with libcucim can make it uninstallable on ARM with CTK 12.0 & 12.1.

To fix this, build two variants of the libcucim package. One with libcufile and one without. Constrain cuda-version accordingly so that libcufile is installed whenever possible. Only when libcufile cannot be installed, fallback to the libcucim package where the libcufile dependency is left off.

The x86_64 CTK has had cuFile since the CTK 11.4. As RAPIDS now requires CUDA 12.0+, there is no issue always requiring libcufile on x86_64.

This issue will not be present in CUDA 13+. So always require libcufile in that case.

The CUDA Toolkit (CTK) added cuFile for ARM in CTK 12.2. As a result always requiring `libcufile` with `libcucim` can make it uninstallable on ARM with CTK 12.0 & 12.1. To fix this, build two variants of the `libcucim` package. One with `libcufile` and one without. Constrain `cuda-version` accordingly so that `libcufile` is installed whenever possible. Only when `libcufile` cannot be installed, fallback to the `libcucim` package where the `libcufile` dependency is left off. The `x86_64` CTK has had cuFile since the CTK 11.4. As RAPIDS now requires CUDA 12.0+, there is no issue always requiring `libcufile` on `x86_64`. This issue will not be present in CUDA 13+. So always require `libcufile` in that case.

The CUDA Toolkit will either have `cufile.h` or not. If it has it, we can build support for cuFile. If it doesn't, we can skip it entirely.

jakirkham · 2025-08-22T05:35:24Z

gds/include/cufile_stub.h

-#else
-    #include "cufile_stub_types.h"
-#endif
+#include <cufile.h>


Trying to #include "cufile_stub_types.h" causes the following error on CI:

In file included from /opt/conda/conda-bld/work/gds/src/cufile_stub.cpp:17: /opt/conda/conda-bld/work/gds/include/cufile_stub.h:23:14: fatal error: cufile_stub_types.h: No such file or directory 23 | #include "cufile_stub_types.h" | ^~~~~~~~~~~~~~~~~~~~~ compilation terminated.

Given we either have a new enough CUDA Toolkit and it has cuFile or it doesn't, we can simply check for cuFile and build or not

Unsure what we gain by having this in-between mode. Though would be happy to learn if I'm missing something

copy-pr-bot · 2025-08-22T06:24:56Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

`libcucim` is not installable at the moment on arm64 systems with CUDA < 12.2: * https://github.com/rapidsai/cucim/pull/905/files#r2292609197 * rapidsai/cucim#930 Which is blocking builds of RAPIDS docker images: * rapidsai/docker#782 (comment) This proposes **temporarily** excluding `cucim` from the dependencies of arm64 CUDA 12 `rapids` packages, to unblock `docker` CI (and therefore publication of the first CUDA 13 nightlies of those images) until that `cucim` issue is resolved. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Gil Forsyth (https://github.com/gforsyth) URL: #796

jameslamb · 2025-09-08T16:46:05Z

/ok to test

grlee77 · 2025-09-09T15:03:45Z

ci/build_cpp.sh

+cat > extra_variants.yaml <<EOF
+has_cufile:
+  - True
+EOF
+if [[ "$(arch)" == "aarch64" ]] && [[ "${RAPIDS_CUDA_VERSION%%.*}" == "12" ]]
+then
+  cat >> extra_variants.yaml <<EOF
+  - False
+EOF
+fi


Looks like the current logic will have both in the aarch64+CUDA12 case?

has_cufile: - True - False

Suggested change

cat > extra_variants.yaml <<EOF

has_cufile:

- True

EOF

if [[ "$(arch)" == "aarch64" ]] && [[ "${RAPIDS_CUDA_VERSION%%.*}" == "12" ]]

then

cat >> extra_variants.yaml <<EOF

- False

EOF

fi

if [[ "$(arch)" == "aarch64" ]] && [[ "${RAPIDS_CUDA_VERSION%%.*}" == "12" ]]; then

cat > extra_variants.yaml <<EOF

has_cufile:

- False

EOF

else

cat > extra_variants.yaml <<EOF

has_cufile:

- True

EOF

fi

Let me commit this and see if it helps

Yes this is intended. We want to build once for each of these values, which is what the original logic does

Also for CUDA 13+ or x86_64 builds we want this to only be True

The CI failure is due to something else unrelated to this syntax. Please see this comment: #930 (comment)

Have gone ahead and restored the original logic. Though happy to chat more if you have additional questions 🙂

grlee77 · 2025-09-09T15:05:38Z

/ok to test

jakirkham · 2025-09-17T14:51:29Z

The error in the CI log points to cuFile being used in the build and not being found

In file included from /opt/conda/conda-bld/work/gds/src/cufile_stub.cpp:17:
 /opt/conda/conda-bld/work/gds/include/cufile_stub.h:19:10: fatal error: cufile.h: No such file or directory
Z    19 | #include <cufile.h>
4Z       |          ^~~~~~~~~~
76Z compilation terminated.
111Z

Will either need to...

Always build with libcufile-dev and simply selectively drop the runtime dependency on libcufile
Rework the build logic to handle cuFile being absent

Personally would lean towards 1 as that should be less effort. Plus the effort in 2 becomes irrelevant once CUDA 12.0 & 12.1 are dropped from support

jakirkham

Have added option 1 below as a suggestion

Option 2 likely involves more changes possibly to parts we haven't yet touched

conda/recipes/libcucim/meta.yaml

jakirkham · 2025-09-18T00:22:26Z

/ok to test

jakirkham · 2025-09-18T00:23:07Z

Pushed the option 1 approach. Let's see how that goes

jakirkham · 2025-09-18T01:33:04Z

/ok to test

This reverts commit 4dbfd5a.

jakirkham · 2025-09-18T02:31:21Z

/ok to test

jakirkham · 2025-09-18T02:34:21Z

/ok to test

jakirkham · 2025-09-18T03:33:41Z

/ok to test

To ensure the different cuCIM packages with and without cuFile are both used, they need different package strings. As the new variant, `has_cufile`, is included in the package hash, make sure to include the package hash in the package string. That way each package is named differently and they don't clobber each other. Thus ensuring both packages are uploaded and not whichever one was built last.

jakirkham · 2025-09-18T04:02:46Z

/ok to test

jakirkham · 2025-09-18T07:33:31Z

At this point this PR has the needed pieces to address the issue

High-level cuCIM is designed to always build with cuFile and optionally use it at runtime (only if it can find and use the cuFile library)

As RAPIDS takes the approach of building with the latest CUDA minor version for any major version (like 12.9 for the 12 series), cuFile is available at build time for all architectures. So building cuCIM with cuFile on CUDA 12.9 works unconditionally

At runtime, cuCIM has two modes of using cuFile:

Statically linked cuFile
dlopen to load the library

In RAPIDS we only ever build cuCIM with 2 (dlopen)

cucim/gds/src/cufile_stub.cpp

Lines 64 to 68 in 16c07c6

    
           handle_ = cucim::dynlib::load_library( 
        
               { 
        
                   "libcufile.so.0", 
        
               }, 
        
               RTLD_LAZY | RTLD_LOCAL | RTLD_NODELETE);

With this approach, cuCIM will only use cuFile if it can dlopen it

cucim/cpp/src/filesystem/cufile_driver.cpp

Lines 313 to 316 in 16c07c6

    
           CUfileError_t status = cuFileDriverOpen(); 
        
           if (status.err == CU_FILE_SUCCESS) 
        
           { 
        
               is_available_ = true;

However if cuCIM cannot dlopen cuFile, it falls back to normal POSIX IO

cucim/cpp/src/filesystem/cufile_driver.cpp

Lines 332 to 339 in 16c07c6

    
           else 
        
           { 
        
               is_available_ = false; 
        
               max_device_cache_size_ = DEFAULT_MAX_CACHE_SIZE; 
        
               max_host_cache_size_ = DEFAULT_MAX_CACHE_SIZE; 
        
               // fmt::print(stderr, "[Warning] CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.\n"); 
        
           }

So all we ever need do for old CUDA 12 ARM builds is drop the cuFile dependency at runtime, which is what this PR does

Opening this up for review

jakirkham

Added a few notes for context below

ci/build_cpp.sh

jakirkham · 2025-09-18T07:39:48Z

conda/recipes/libcucim/meta.yaml

 build:
  number: {{ GIT_DESCRIBE_NUMBER }}
-  string: cuda{{ cuda_major }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }}
+  string: cuda{{ cuda_major }}_{{ date_string }}_gh{{ GIT_DESCRIBE_HASH }}_gn{{ GIT_DESCRIBE_NUMBER }}}_ph{{ PKG_HASH }}


Since we are building for two different variants of has_cufile, the package hash will change. However the rest of these values stay the same for each variant build. As a result the package name would be the same unless we add the package hash, which we do here

jakirkham · 2025-09-18T07:42:08Z

conda/recipes/libcucim/meta.yaml

    - {{ compiler('cuda') }}
    - cuda-cudart-dev
-    - libcufile-dev  # [linux64]
+    - libcufile-dev


We want to ignore the run_exports_from libcufile-dev in all cases where it is used as we already constrain the version via cuda-version

This avoids overly tight version constraints that otherwise cause solver errors when testing CUDA 12.9 packages on CUDA 12.0

jakirkham · 2025-09-18T07:44:34Z

conda/recipes/libcucim/meta.yaml

+    {% if aarch64 and cuda_major == "12" %}
+    # Relax `libcufile` dependency for old CUDA 12 ARM installs
+    - {{ pin_compatible('cuda-version', lower_bound='12.0', upper_bound='12.2') }}  # [not has_cufile]
+    - {{ pin_compatible('cuda-version', lower_bound='12.2', upper_bound='13.0') }}  # [has_cufile]
+    - libcufile                                                                     # [has_cufile]
+    {% else %}


This ensures ARM packages that support CUDA 12.2+ still depend io cuFile whereas packages that require CUDA 12.0 or 12.1 packages on ARM don't require cuFile (as it is not given in this case)

jameslamb

Thank you @jakirkham ! The changes and comments all make sense to me, I'd be happy to see this go in.

Once it does, we can do rapidsai/integration#797 (adding the cucim dependency back to the rapids metapackage unconditionally)

bdice

Generally looks fine to me.

jakirkham · 2025-09-18T18:27:19Z

Thanks all! 🙏

jakirkham · 2025-09-18T18:27:41Z

/merge

jakirkham · 2025-09-18T19:39:49Z

Saw a flaky networking issue in one CI job:

Get:22 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted arm64 Packages [5000 kB]
Fetched 38.1 MB in 32s (1181 kB/s)
Reading package lists...
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/jammy-security/main/binary-arm64/Packages.gz  File has unexpected size (3038653 != 3054887). Mirror sync in progress? [IP: 91.189.91.104 80]
   Hashes of expected file:
    - Filesize:3054887 [weak]
    - SHA256:04e8cacff94dccf44b0d14079f4f896262844739a764aa6333f42a0678c1195a
    - SHA1:447449b9caef6590e3ad69ce93872cebed2c302a [weak]
    - MD5Sum:451978ccf71438e21b7cbaf35ca1390d [weak]
   Release file created at: Thu, 18 Sep 2025 13:17:15 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

Mentioned offline. Went ahead and restarted this job

Previously cuCIM was dropped from CUDA 12 ARM as the packages unconditionally required cuFile, which caused issues when testing with CUDA 12.0 & 12.1. To address this, the cuCIM package was changed to only require cuFile in CUDA 12.2+ ARM packages while creating a separate package for ARM CUDA 12.0 & 12.1, which left out the cuFile dependency. This work was completed in PR: rapidsai/cucim#930 With this change now in place, it should be possible to readd cuCIM to the RAPIDS metapackage generally, which is what this PR does <hr> Reverts #796 Fixes #797 Authors: - https://github.com/jakirkham Approvers: - James Lamb (https://github.com/jameslamb) URL: #802

jakirkham requested review from a team as code owners August 22, 2025 04:25

jakirkham requested a review from jameslamb August 22, 2025 04:25

jakirkham added bug Something isn't working non-breaking Introduces a non-breaking change labels Aug 22, 2025

jakirkham mentioned this pull request Aug 22, 2025

Remove CUDA 11 references #905

Merged

jakirkham force-pushed the fix_dep_cufile_arm branch 3 times, most recently from a29ea1b to 15e235d Compare August 22, 2025 05:18

jakirkham force-pushed the fix_dep_cufile_arm branch from 15e235d to b0e17af Compare August 22, 2025 05:19

Simply use cufile.h

f5f3ae9

The CUDA Toolkit will either have `cufile.h` or not. If it has it, we can build support for cuFile. If it doesn't, we can skip it entirely.

jakirkham commented Aug 22, 2025

View reviewed changes

Require cuFile component in GDS stub

584e88c

jakirkham requested a review from a team as a code owner August 22, 2025 05:59

jakirkham marked this pull request as draft August 22, 2025 06:24

This was referenced Sep 5, 2025

Build and test with CUDA 13.0.0 rapidsai/docker#782

Merged

temporarily drop cucim from 'rapids' CUDA 12 package rapidsai/integration#796

Merged

Add CUDA 13.0 Support across all of RAPIDS rapidsai/build-planning#208

Open

jameslamb mentioned this pull request Sep 5, 2025

Restore cucim dependency for CUDA 12 arm64 packages rapidsai/integration#797

Closed

Merge branch 'branch-25.10' into fix_dep_cufile_arm

f6f2d7d

grlee77 reviewed Sep 9, 2025

View reviewed changes

Update ci/build_cpp.sh

4dbfd5a

jakirkham commented Sep 17, 2025

View reviewed changes

conda/recipes/libcucim/meta.yaml Outdated Show resolved Hide resolved

Always build with libcufile-dev

2063e7c

Ignore run_exports of libcufile-dev on ARM too

61ab317

Revert "Update ci/build_cpp.sh"

e447f6b

This reverts commit 4dbfd5a.

Tweak comment about selectively dropping cuFile

9dd7bff

jakirkham added 2 commits September 17, 2025 19:36

Align selectors

99975e2

Show what extra_variants.yaml contains in CI log

73dd613

jakirkham marked this pull request as ready for review September 18, 2025 07:33

jakirkham commented Sep 18, 2025

View reviewed changes

jameslamb approved these changes Sep 18, 2025

View reviewed changes

bdice approved these changes Sep 18, 2025

View reviewed changes

Note ARM CUDA 12.2+ requires cuFile

5dbbb43

rapids-bot bot merged commit 42cf58f into rapidsai:branch-25.10 Sep 18, 2025
114 of 115 checks passed

jakirkham deleted the fix_dep_cufile_arm branch September 18, 2025 20:10

jakirkham mentioned this pull request Sep 18, 2025

Restore cuCIM on CUDA 12 ARM rapidsai/integration#802

Merged

Relax libcufile dependency for old CTK 12 on ARM #930

Relax libcufile dependency for old CTK 12 on ARM #930

Uh oh!

Conversation

jakirkham commented Aug 22, 2025

Uh oh!

jakirkham Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

copy-pr-bot bot commented Aug 22, 2025

Uh oh!

jameslamb commented Sep 8, 2025

Uh oh!

grlee77 Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

grlee77 Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

jakirkham Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grlee77 commented Sep 9, 2025

Uh oh!

jakirkham commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jakirkham commented Sep 18, 2025

Uh oh!

jakirkham commented Sep 18, 2025

Uh oh!

jakirkham commented Sep 18, 2025

Uh oh!

jakirkham commented Sep 18, 2025

Uh oh!

jakirkham commented Sep 18, 2025

Uh oh!

jakirkham commented Sep 18, 2025

Uh oh!

jakirkham commented Sep 18, 2025

Uh oh!

jakirkham commented Sep 18, 2025

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jakirkham Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

jakirkham Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

jakirkham Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

jameslamb left a comment

Choose a reason for hiding this comment

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

jakirkham commented Sep 18, 2025

Uh oh!

jakirkham commented Sep 18, 2025

Uh oh!

jakirkham commented Sep 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Relax `libcufile` dependency for old CTK 12 on ARM #930

Relax `libcufile` dependency for old CTK 12 on ARM #930

jakirkham Sep 18, 2025 •

edited

Loading

jakirkham commented Sep 17, 2025 •

edited

Loading