Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] CAGRA avoid unnecessary device memory copies of dataset / knn graph #1479

Closed
Tracked by #1392
tfeher opened this issue May 1, 2023 · 0 comments · Fixed by #1494
Closed
Tracked by #1392

[FEA] CAGRA avoid unnecessary device memory copies of dataset / knn graph #1479

tfeher opened this issue May 1, 2023 · 0 comments · Fixed by #1494
Labels
feature request New feature or request

Comments

@tfeher
Copy link
Contributor

tfeher commented May 1, 2023

Is your feature request related to a problem? Please describe.

When data / graph is in device memory, consider just storing a reference in the index for the data. See discussion #1375 (comment)

Tracker #1392

@tfeher tfeher added the feature request New feature or request label May 1, 2023
rapids-bot bot pushed a commit that referenced this issue Jul 20, 2023
This PR aims to improve the workflow when dealing with large datasets. When experimenting with different versions of the knn-graph, we might want to construct indices with the same dataset (see #1435 for further discussion).

If the dataset is already in device memory (and rows are properly aligned / padded), then we only store a reference to the dataset, therefore multiple indices can refer to the same dataset. Similarly, when `knn_graph` is a device array, then store only a reference. 

Additionally, this PR adds `update_dataset` and `update_graph` methods to the index.
Closes #1479

Authors:
  - Tamas Bela Feher (https://github.com/tfeher)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)

URL: #1494
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
Status: No status
Development

Successfully merging a pull request may close this issue.

1 participant