WIP: install 'hdbscan' from PyPI

[jameslamb]

Description

Proposes:

• getting test dependency hdbscan from PyPI instead of GitHub
  • (there was a release 3 weeks ago: https://pypi.org/project/hdbscan/)
• ensuring rapids-dependency-file-generator --output requirements generates a full set of test dependencies

Benefits of these changes

• ensures devcontainers environment includes all of cuml's test dependencies
• removes a dependency on building from the tip of a remote git repo, and all the potential for instability and security issues that comes with that

Notes for Reviewers

I tried testing a devcontainers change today (rapidsai/devcontainers#325), like this:

# clean previous build outputs
uninstall-all -j
clean-all -j

# work on latest branches of everything
rapids-checkout-same-branch --omit ucxx
git -C ./ucxx checkout branch-0.39
rapids-pull-repositories

# install all the other dependencies
rapids-make-pip-env --force

# build all the packages
build-all -j

# run the cuml unit tests
cd ./cuml
./ci/run_cuml_singlegpu_pytests.sh

Found that the cuml unit tests failed immediately with an error complaining about hdbscan not being available. Looks like that's because it's used unconditionally here:

cuml/python/cuml/tests/test_hdbscan.py

Line 20 in 6342914

import hdbscan

This hasn't shown up in CI because in CI, the cuml wheel's [test] extra is used to figure out what dependencies to install:

cuml/ci/test_wheel.sh

Line 16 in 6342914

python -m pip install $(echo ./dist/cuml*.whl)[test]

But RAPIDS devcontainers use rapids-dependency-file-generator --output requirements:

https://github.com/rapidsai/devcontainers/blob/0ec3cbd47af5ffdc1399af2cb9e33e34617230c3/features/src/rapids-build-utils/opt/rapids-build-utils/bin/make-pip-dependencies.sh#L90-L96

[dantegd]

Thanks @jameslamb, @divyegala was working on this in this PR #5906. The issue we ran into is that hdbscan requires cython >=0.27,<3 with the potential options at runtime, which is the same issue this PR has already ran into in the pip devcontainer job

[jameslamb]

🤦🏻 I should have checked the open PRs, was a little too quick here. Thanks @dantegd and sorry for the noise, I'll close this.