Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dask] dask cudf inplace prediction. #5512

Merged
merged 22 commits into from
Apr 15, 2020
Merged

[dask] dask cudf inplace prediction. #5512

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
8689170
dask cudf inplace prediction.
trivialfis Apr 10, 2020
12b31c5
Change CI script.
trivialfis Apr 12, 2020
bd50317
Remove Dockerfile.release, since it's not used anywhere
hcho3 Apr 14, 2020
e94fe52
Use latest Miniconda version
hcho3 Apr 14, 2020
e044d8f
Use Conda exclusively in CUDF and GPU containers
hcho3 Apr 14, 2020
1576332
Merge remote-tracking branch 'upstream/master' into dask-cudf-inplace
hcho3 Apr 14, 2020
73f073b
Correct pytest mark.
trivialfis Apr 14, 2020
7eca412
Merge branch 'dask-cudf-inplace' of https://github.com/trivialfis/xgb…
trivialfis Apr 14, 2020
76cbef7
Use `python-kubernetes`.
trivialfis Apr 14, 2020
9e20bb7
Use kubernetes with pip
hcho3 Apr 14, 2020
269c9f1
Do not install 'cudatoolkit' conda package for GPU container, as cuDF…
hcho3 Apr 14, 2020
4288462
Strict device id checks.
trivialfis Apr 14, 2020
e1ebde0
Fix latest cudf.
trivialfis Apr 14, 2020
c652cc0
Add skip marks.
trivialfis Apr 14, 2020
f8ed0c4
Don't use raw memcpy.
trivialfis Apr 14, 2020
30f792e
Revert installation.
trivialfis Apr 14, 2020
d503aac
type mapping.
trivialfis Apr 14, 2020
cfbc3cd
Just run all of them.
trivialfis Apr 14, 2020
07467aa
File name.
trivialfis Apr 14, 2020
bc2aa18
Add back cuDF tests that need multi-GPU
hcho3 Apr 15, 2020
58ca057
Add mgpu-cudf category
hcho3 Apr 15, 2020
3dd595d
Just run all of test_gpu_with_dask.py
hcho3 Apr 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions python-package/xgboost/dask.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -623,8 +623,6 @@ def mapped_predict(data, is_df):
if is_df:
if lazy_isinstance(data, 'cudf.core.dataframe', 'DataFrame'):
import cudf # pylint: disable=import-error
# There's an error with cudf saying `concat_cudf` got an
# expected argument `ignore_index`. So this is not yet working.
prediction = cudf.DataFrame({'prediction': prediction},
dtype=numpy.float32)
else:
Expand Down
6 changes: 4 additions & 2 deletions tests/ci_build/Dockerfile.cudf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,10 @@ ENV PATH=/opt/python/bin:$PATH

# Create new Conda environment with cuDF and dask
RUN \
conda create -n cudf_test -c rapidsai -c nvidia -c numba -c conda-forge -c anaconda \
cudf=0.9 python=3.7 anaconda::cudatoolkit=$CUDA_VERSION dask dask-cuda cupy
conda create -n cudf_test python=3.7 && \
source activate cudf_test && \
conda install -c rapidsai -c nvidia -c numba -c conda-forge -c anaconda \
anaconda::cudatoolkit=$CUDA_VERSION dask dask-cuda dask-cudf cupy cudf

# Install other Python packages
RUN \
Expand Down
2 changes: 1 addition & 1 deletion tests/ci_build/Dockerfile.gpu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ ENV PATH=/opt/python/bin:$PATH
RUN \
pip install numpy pytest scipy scikit-learn pandas matplotlib wheel kubernetes urllib3 graphviz && \
pip install "dask[complete]" && \
conda install -c rapidsai -c nvidia -c numba -c conda-forge -c anaconda dask-cuda
conda install -c rapidsai -c conda-forge -c anaconda dask-cuda
hcho3 marked this conversation as resolved.
Show resolved Hide resolved

ENV GOSU_VERSION 1.10

Expand Down
3 changes: 2 additions & 1 deletion tests/ci_build/test_python.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ case "$suite" in
cudf)
source activate cudf_test
install_xgboost
pytest -v -s --fulltrace -m "not mgpu" tests/python-gpu/test_from_cudf.py tests/python-gpu/test_from_cupy.py
pytest -v -s --fulltrace -m "not mgpu" tests/python-gpu/test_from_cudf.py tests/python-gpu/test_from_cupy.py \
tests/python-gpu/test_gpu_with_dask.py::TestDistributedGPU::test_dask_dataframe
;;

cpu)
Expand Down
10 changes: 4 additions & 6 deletions tests/python-gpu/test_gpu_with_dask.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,16 +51,14 @@ def test_dask_dataframe(self):
predictions = dxgb.predict(client, out, dtrain).compute()
assert isinstance(predictions, np.ndarray)

# There's an error with cudf saying `concat_cudf` got an
# expected argument `ignore_index`. So the test here is just
# place holder.

# series_predictions = dxgb.inplace_predict(client, out, X)
# assert isinstance(series_predictions, dd.Series)
series_predictions = dxgb.inplace_predict(client, out, X)
assert isinstance(series_predictions, dd.Series)
series_predictions = series_predictions.compute()

single_node = out['booster'].predict(
xgboost.DMatrix(X.compute()))
cupy.testing.assert_allclose(single_node, predictions)
cupy.testing.assert_allclose(single_node, series_predictions)

@pytest.mark.skipif(**tm.no_cupy())
def test_dask_array(self):
Expand Down