Support for precomputed distance matrix in DBSCAN

[Nyrio]

Closes #3302

Notes about performance

If we don't count the cost of pre-computing the distance matrix (which is done by the user), the single-GPU code runs slightly faster when the distance matrix is pre-computed. (note: this is 2d, greater speedups expected for larger dimensions!)

As I have stated in a comment in the code, it works with two kernels: one that uses a coalesced reduction to compute the vertex degrees from the distance matrix, and one that uses a 2D copy fused with an unary operation to get the boolean neighborhood matrix.

Note: the performance of this step could be better if adj was a row-major B*N matrix instead of column-major. We could fuse everything into one efficient kernel. It is something to keep in mind when we re-write csr_adj_graph_batched.

Notes about MNMG

Cf #3615

[tfeher]

Thanks @Nyrio for the PR! It looks good in general, I have just marked a few smaller issues.

Please file an issue about the MNMG case with distributed data, to promote discussion on its priority.

[tfeher]

Thanks @Nyrio for the update. Changes look good in general, there are only some minor things left:

• Fix the failing test_base_children_get_param_names unit test by adding 'metric' to get_param_names() in dbscan.pyx.
• The PR description will be added as a commit message of the merge commit. I think the TODO section can be removed from the description.
• Consider linking the original feature request with the appropriate keyword.
• Please file an issue about "distance matrix that is scattered across the nodes".

[tfeher]

Thanks Louis for the update! The PR looks good to me.

[teju85]

For official-sake, approving this PR based on the reviews done by @tfeher (he's not yet a cpp-codeowner). @dantegd or @JohnZed can we get python-side approval from one of you?

[dantegd]

Sorry for the delayed review, long week of reviews. Just had one question and one very minor suggestion to improve the python docstring, then it looks good in the Python side as well.

[JohnZed]

Blocked waiting on RAFT change

[Nyrio]

@dantegd @teju85 I made the change to use raft::distance::DistanceType but I'm having a very weird issue in dbscan.pyx that I don't understand. If I use DistanceType.Precomputed, I get a compiler error:

UnicodeEncodeError: 'latin-1' codec can't encode character '\u0308' in position 15936: ordinal not in range(256)

If I comment line 259, it compiles. The problem doesn't seem to be a special character, I've tried rewriting this line from scratch, also it compiles when commented. I don't see why the enum value would cause that either, the value is 100, which is under 256.

I'm quite confused. Does someone know what's happening here?

[dantegd]

@Nyrio that's quite an odd issue, I would suggest merging branch-0.19 into the branch of the PR (to solve the copyright issues) and then we can see if the unicode error persists.

[Nyrio]

I've merged branch-0.19 but it unleashed a dependency nightmare. The latest changes seem to require Faiss 1.7.0 and the latest Rapids Docker images released a few hours ago come with Faiss 1.6.3. Forcing the update of the libfaiss package creates linking errors with other libraries...

Edit: it works with 11.2. The dependency issue was with 11.0

[Nyrio]

Update: I managed to solve my dependency issues with 11.2 but the problem of the odd codec error persists.

I've also tried using contiguous values in the enum, which didn't seem to work.

[tfeher]

I confirm the problem with the enum. Python install fails as long as we have the Precomputed name here.

A workaround is to rename the enum to Precalculated. With that it compiles for me, but that requires another RAFT PR.

[Nyrio]

It turns out it was an encoding bug, with an invisible character that didn't appear in the editor. Fixed.

[codecov-io]

Codecov Report

Merging #3585 (99797c5) into branch-0.19 (fd9ec89) will decrease coverage by 35.36%.
The diff coverage is 57.07%.

@@               Coverage Diff                @@
##           branch-0.19    #3585       +/-   ##
================================================
- Coverage        80.70%   45.34%   -35.37%     
================================================
  Files              227      224        -3     
  Lines            17615    17189      -426     
================================================
- Hits             14217     7794     -6423     
- Misses            3398     9395     +5997     

Flag	Coverage Δ
dask	45.34% <57.07%> (+0.35%)	⬆️
non-dask	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
python/cuml/cluster/kmeans.pyx	58.29% <ø> (-33.67%)	⬇️
python/cuml/common/memory_utils.py	65.85% <0.00%> (-13.27%)	⬇️
python/cuml/fil/fil.pyx	63.77% <ø> (-28.07%)	⬇️
python/cuml/model_selection/_split.py	5.58% <0.00%> (-84.78%)	⬇️
python/cuml/neighbors/__init__.py	100.00% <ø> (ø)
python/cuml/neighbors/ann.pyx	8.04% <0.00%> (-53.59%)	⬇️
python/cuml/neighbors/nearest_neighbors.pyx	41.47% <0.00%> (-51.18%)	⬇️
python/cuml/pipeline/__init__.py	0.00% <0.00%> (ø)
python/cuml/preprocessing/encoders.py	88.04% <ø> (-7.04%)	⬇️
python/cuml/solvers/qn.pyx	17.31% <7.14%> (-80.32%)	⬇️
... and 150 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0883026...99797c5. Read the comment docs.

[dantegd]

@gpucibot merge