Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce libcuml wheels #6199

Merged
merged 77 commits into from
Jan 24, 2025
Merged
Changes from 1 commit
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
e42f3d1
initial libcuml wheel
jameslamb Dec 24, 2024
13db0a9
remove uses of ASSOCIATED_TARGETS, lots of other refactoring
jameslamb Dec 30, 2024
f33e5ed
Merge branch 'branch-25.02' into libcuml-wheels
jameslamb Dec 30, 2024
368faea
more changes
jameslamb Dec 31, 2024
bbc4742
Merge branch 'branch-25.02' of github.com:rapidsai/cuml into libcuml-…
jameslamb Dec 31, 2024
e99a2b7
use static treelite
jameslamb Dec 31, 2024
b6eeb73
merge branch-25.02
jameslamb Dec 31, 2024
8082c45
one more merge conflict
jameslamb Dec 31, 2024
8434d77
more fiddling
jameslamb Jan 2, 2025
9bda048
more changes
jameslamb Jan 2, 2025
eb14ede
more changes
jameslamb Jan 6, 2025
c5d385b
Merge branch 'branch-25.02' into libcuml-wheels
jameslamb Jan 6, 2025
8c1f38d
got importing minimally working
jameslamb Jan 6, 2025
7c09598
use conda packages from RAFT PR
jameslamb Jan 6, 2025
c7ad450
pre-commit
jameslamb Jan 6, 2025
8d058e6
copyright
jameslamb Jan 6, 2025
cbf26a2
fix scripts
jameslamb Jan 6, 2025
dc8a5a4
fix wheel downloads
jameslamb Jan 7, 2025
9773c81
clone cumlprims_mg in CI
jameslamb Jan 7, 2025
a56e3d4
fix pins, fix script
jameslamb Jan 7, 2025
dba121f
pull in changes from logger targets
jameslamb Jan 8, 2025
2290fb7
more changes
jameslamb Jan 9, 2025
e6f4d73
update handling of cumlprims_mg
jameslamb Jan 9, 2025
a224fc1
more todo
jameslamb Jan 9, 2025
afe7c4f
Merge branch 'branch-25.02' of github.com:rapidsai/cuml into libcuml-…
jameslamb Jan 9, 2025
96b90f3
merge branch-25.02
jameslamb Jan 10, 2025
05f8fbf
start unwinding CPM debugging changes
jameslamb Jan 10, 2025
8f505e3
undo exporting
jameslamb Jan 10, 2025
829d714
add debugging prints, update RAFT commit
jameslamb Jan 13, 2025
8151e62
this is working
jameslamb Jan 13, 2025
cae868e
revert some debugging changes
jameslamb Jan 13, 2025
def5a4d
use static treelite
jameslamb Jan 13, 2025
9ae5941
Merge branch 'branch-25.02' into libcuml-wheels
jameslamb Jan 13, 2025
83b28d9
bigger pydistcheck threshold, remove cuvs from build env
jameslamb Jan 13, 2025
e6cf739
Merge branch 'libcuml-wheels' of github.com:jameslamb/cuml into libcu…
jameslamb Jan 13, 2025
1d5326f
fix validation working dir
jameslamb Jan 13, 2025
2ef32ea
try to fix more builds
jameslamb Jan 13, 2025
e35df32
use dynamic treelite in conda builds
jameslamb Jan 13, 2025
7853681
refine build dependencies
jameslamb Jan 14, 2025
7e94d7d
merge branch-25.02
jameslamb Jan 14, 2025
ba3387f
more dependency fixes
jameslamb Jan 14, 2025
1f9ac89
fix scripts
jameslamb Jan 14, 2025
82e3f16
update RAFT commit, build cuml without build isolation
jameslamb Jan 14, 2025
53c8b35
try fixing math wheels linkage for CUDA 11
jameslamb Jan 14, 2025
002c0f3
add use_cuda_wheels
jameslamb Jan 14, 2025
35c88ac
fix build_wheel_cuml.sh again
jameslamb Jan 14, 2025
0e62018
try to fix build_wheel_cuml.sh again
jameslamb Jan 14, 2025
f5b6590
misc. changes
jameslamb Jan 16, 2025
e4c1fad
Merge branch 'branch-25.02' of github.com:rapidsai/cuml into libcuml-…
jameslamb Jan 16, 2025
c79617b
rapids_cuda_init_architectures()
jameslamb Jan 16, 2025
57db9f7
Revert libraft testing changes.
bdice Jan 17, 2025
e7109c4
simplify CMake
jameslamb Jan 17, 2025
e2ae2dc
Merge branch 'libcuml-wheels' of github.com:jameslamb/cuml into libcu…
jameslamb Jan 17, 2025
b17f613
fix typo in rapids_cuda_init_architectures
jameslamb Jan 17, 2025
2623cc9
fix SINGLEGPU arg
jameslamb Jan 17, 2025
77c8f14
updates
jameslamb Jan 17, 2025
668e46b
Merge branch 'branch-25.02' of github.com:rapidsai/cuml into libcuml-…
jameslamb Jan 17, 2025
3a02bf5
use libcuvs wheels
jameslamb Jan 21, 2025
bd8de88
CUVS_COMMIT
jameslamb Jan 21, 2025
610970a
fix depends_on_libcuvs
jameslamb Jan 21, 2025
2859fd8
Merge branch 'branch-25.02' into libcuml-wheels
jameslamb Jan 22, 2025
73f20b9
use CUDA wheels on CUDA 11
jameslamb Jan 22, 2025
9c615b1
update cuVS commit
jameslamb Jan 22, 2025
71ce00d
rearrange CI-artifact-downloading
jameslamb Jan 22, 2025
8c9dd70
align paths for constraints
jameslamb Jan 22, 2025
d016c28
this constraints file is giving me agita
jameslamb Jan 22, 2025
9d5100e
newer cuvs commit
jameslamb Jan 22, 2025
b1b9d08
adjust wheel-size thresholds, clean up some debugging stuff
jameslamb Jan 22, 2025
21fe7c7
more cleanup
jameslamb Jan 22, 2025
057a4a9
Update ci/build_wheel_cuml.sh
jameslamb Jan 23, 2025
c451a47
Update python/libcuml/libcuml/load.py
jameslamb Jan 23, 2025
93ddda5
merge branch-25.02
jameslamb Jan 23, 2025
02ac476
Merge branch 'branch-25.02' into libcuml-wheels
bdice Jan 23, 2025
fb83625
remove download-cuvs-ci-artifacts stuff
jameslamb Jan 23, 2025
304c52d
Merge branch 'libcuml-wheels' of github.com:jameslamb/cuml into libcu…
jameslamb Jan 23, 2025
1f09c32
Remove testing changes now that libcuvs is available.
bdice Jan 23, 2025
90b4d81
Merge branch 'libcuml-wheels' of github.com:jameslamb/cuml into libcu…
bdice Jan 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use conda packages from RAFT PR
jameslamb committed Jan 6, 2025
commit 7c095984bceef359b84cf123ea931b6e8ab108b6
3 changes: 3 additions & 0 deletions ci/build_cpp.sh
Original file line number Diff line number Diff line change
@@ -9,6 +9,9 @@ source rapids-configure-sccache

source rapids-date-string

# TODO(jameslamb): remove this when https://github.com/rapidsai/raft/pull/2531 is merged
source ci/use_conda_packages_from_prs.sh

export CMAKE_GENERATOR=Ninja

rapids-print-env
3 changes: 3 additions & 0 deletions ci/build_docs.sh
Original file line number Diff line number Diff line change
@@ -8,6 +8,9 @@ rapids-logger "Create test conda environment"
RAPIDS_VERSION="$(rapids-version)"
export RAPIDS_VERSION_MAJOR_MINOR="$(rapids-version-major-minor)"

# TODO(jameslamb): remove this when https://github.com/rapidsai/raft/pull/2531 is merged
source ci/use_conda_packages_from_prs.sh

rapids-dependency-file-generator \
--output conda \
--file-key docs \
3 changes: 3 additions & 0 deletions ci/build_python.sh
Original file line number Diff line number Diff line change
@@ -9,6 +9,9 @@ source rapids-configure-sccache

source rapids-date-string

# TODO(jameslamb): remove this when https://github.com/rapidsai/raft/pull/2531 is merged
source ci/use_conda_packages_from_prs.sh

export CMAKE_GENERATOR=Ninja

rapids-print-env
3 changes: 3 additions & 0 deletions ci/build_wheel_cuml.sh
Original file line number Diff line number Diff line change
@@ -8,6 +8,9 @@ package_dir="python/cuml"

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

# TODO(jameslamb): remove this when https://github.com/rapidsai/raft/pull/2531 is merged
source ./ci/use_wheels_from_prs.sh

# TODO(jameslamb): split this out into build_wheel_{cuml,libcuml}.sh
# TODO(jameslamb): add libcuml++.so to cuml exclusions
case "${RAPIDS_CUDA_VERSION}" in
3 changes: 3 additions & 0 deletions ci/build_wheel_libcuml.sh
Original file line number Diff line number Diff line change
@@ -8,6 +8,9 @@ package_dir="python/libcuml"

rapids-logger "Generating build requirements"

# TODO(jameslamb): remove this when https://github.com/rapidsai/raft/pull/2531 is merged
source ./ci/use_wheels_from_prs.sh

rapids-dependency-file-generator \
--output requirements \
--file-key "py_build_${package_name}" \
3 changes: 3 additions & 0 deletions ci/test_cpp.sh
Original file line number Diff line number Diff line change
@@ -11,6 +11,9 @@ cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")"/../
rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)

# TODO(jameslamb): remove this when https://github.com/rapidsai/raft/pull/2531 is merged
source ci/use_conda_packages_from_prs.sh

rapids-logger "Generate C++ testing dependencies"
rapids-dependency-file-generator \
--output conda \
3 changes: 3 additions & 0 deletions ci/test_notebooks.sh
Original file line number Diff line number Diff line change
@@ -8,6 +8,9 @@ rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

# TODO(jameslamb): remove this when https://github.com/rapidsai/raft/pull/2531 is merged
source ci/use_conda_packages_from_prs.sh

rapids-logger "Generate Notebook testing dependencies"
rapids-dependency-file-generator \
--output conda \
3 changes: 3 additions & 0 deletions ci/test_python_common.sh
Original file line number Diff line number Diff line change
@@ -9,6 +9,9 @@ rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

# TODO(jameslamb): remove this when https://github.com/rapidsai/raft/pull/2531 is merged
source ci/use_conda_packages_from_prs.sh

rapids-logger "Generate Python testing dependencies"
rapids-dependency-file-generator \
--output conda \
1 change: 0 additions & 1 deletion ci/test_wheel.sh
Original file line number Diff line number Diff line change
@@ -20,7 +20,6 @@ EXITCODE=0
trap "EXITCODE=1" ERR
set +e


rapids-logger "pytest cuml single GPU"
./ci/run_cuml_singlegpu_pytests.sh \
--numprocesses=8 \
10 changes: 10 additions & 0 deletions ci/use_conda_packages_from_prs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/bin/bash

# TODO(jameslamb): remove this when https://github.com/rapidsai/raft/pull/2531 is merged
RAFT_COMMIT="345f0e556b602ec65b5eebe825ffd000d61706fe"

RAFT_CPP_CHANNEL=$(rapids-get-pr-conda-artifact raft 2531 cpp "${RAFT_COMMIT:0:7}")
RAFT_PYTHON_CHANNEL=$(rapids-get-pr-conda-artifact raft 2531 python "${RAFT_COMMIT:0:7}")

conda config --system --add channels "${RAFT_CPP_CHANNEL}"
conda config --system --add channels "${RAFT_PYTHON_CHANNEL}"
18 changes: 18 additions & 0 deletions ci/use_wheels_from_prs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# TODO(jameslamb): remove this when https://github.com/rapidsai/raft/pull/2531 is merged
RAFT_COMMIT="345f0e556b602ec65b5eebe825ffd000d61706fe"
LIBRAFT_CHANNEL=$(
RAPIDS_PY_WHEEL_NAME="libraft_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact raft 2531 cpp "${RAFT_COMMIT:0:7}"
)
PYLIBRAFT_CHANNEL=$(
RAPIDS_PY_WHEEL_NAME="libraft_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact raft 2531 python "${RAFT_COMMIT:0:7}"
)
RAFT_DASK_CHANNEL=$(
RAPIDS_PY_WHEEL_NAME="raft_dask_${RAPIDS_PY_CUDA_SUFFIX}" rapids-get-pr-wheel-artifact raft 2531 python "${RAFT_COMMIT:0:7}"
)
cat > ./constraints.txt <<EOF
libraft-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBRAFT_CHANNEL}/libraft_*.whl)
pylibraft-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${PYLIBRAFT_CHANNEL}/pylibraft_*.whl)
raft-dask-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${RAFT_DASK_CHANNEL}/raft_dask_*.whl)
EOF

export PIP_CONSTRAINT=$(pwd)/constraints.txt