[WIP] Expose extend() in C API

[ajit283]

Add functionality to add additional vectors after build to C API

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[benfred]

/ok to test

[benfred]

thanks for adding this functionality! This change looks good to me -

[cjnolet]

/ok to test

[cjnolet]

@ajit283 I think we discussed generating a bigger dataset for the test case here. You could even use raft::random::make_blobs to accomplish this.

[ajit283]

👍 will look into this tomorrow

[cjnolet]

/ok to test

[cjnolet]

@ajit283 can you fix the conflicts in this PR?

[ajit283]

Done! We should look into the tests for this one.

• The previous approach was to test against a set of precomputed queries, expected neighbors and distances.
• This does not work anymore as we generate the dataset dynamically using make_blobs(). Right now I am adding a static set of vectors to the dataset using extend(), then query these vectors and check if I get them back with distance 0. This shows that extend() works, but is generally a regression compared to the previous tests.
• My ideas for improving the tests would be either to remove make_blobs() and use a larger static dataset, or to calculate expected neighbors and distances at runtime using brute-force.

What do you think?

[cjnolet]

/ok to test

[cjnolet]

/ok to test

[ajit283]

@cjnolet I added a test using the pairwise distance matrix as discussed in the call

[cjnolet]

Overall, I think this PR is taking shape and looking great! I think the remaining items are minimal- mostly just doing the diligence of capturing Github issues.