Adds section on product quantization for docs #6926

jmazanec15 · 2024-04-09T00:40:16Z

Description

Adds section in vector quantization docs for product quantization. In it, it contains tips for using it as well as memory estimations. Along with this, changed some formatting to make docs easier to write.

I decided to include completely accurate memory estimate for formula with a note about the typical number of segments.

We added a section on scalar quantization in 2.13 - but it did not include product quantization. Related comment here: https://github.com/opensearch-project/documentation-website/pull/6249/files#r1529479186. This should be backported for 2.13

Issues Resolved

List any issues this PR will resolve, e.g. Closes [...].

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

_search-plugins/knn/knn-vector-quantization.md

Adds section in vector quantization docs for product quantization. In it, it contains tips for using it as well as memory estimations. Along with this, changed some formatting to make docs easier to write. Signed-off-by: John Mazanec <jmazane@amazon.com>

vamshin

LGTM! Thanks

Fix formatting Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Define abbreviation on first mention Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli

Doc review complete. Please let me know if you have any questions about my changes. Once you've addressed my feedback, I'll approve the PR as ready for editorial. Thank you.

_search-plugins/knn/knn-index.md

vagimeli · 2024-04-10T18:01:44Z

_search-plugins/knn/knn-vector-quantization.md

@@ -10,22 +10,42 @@ has_math: true

 # k-NN vector quantization

-By default, the k-NN plugin supports the indexing and querying of vectors of type `float`, where each dimension of the vector occupies 4 bytes of memory. For use cases that require ingestion on a large scale, keeping `float` vectors can be expensive because OpenSearch needs to construct, load, save, and search graphs (for native `nmslib` and `faiss` engines). To reduce the memory footprint, you can use vector quantization.
+By default, the k-NN plugin supports the indexing and querying of vectors of type `float`, where each dimension of the 


Please fix the line break formatting of lines 13--16.

I made the line breaks so that editing would be easier and it doesnt impact rendering (i.e. it wouldnt be one line that rolls out of the screen). Is this incorrect to do?

Yes, it's incorrect to enter line breaks. The site and OpenSearch Project doc team follow a specific formatting guide. I'll handle formatting the doc before moving it into editorial. https://github.com/opensearch-project/documentation-website/blob/main/FORMATTING_GUIDE.md

_search-plugins/knn/knn-vector-quantization.md

Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: John Mazanec <jmazane@amazon.com>

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

natebower

@jmazanec15 @vagimeli Please see my comments and changes and let me know if you have any questions. Thanks!

_search-plugins/knn/knn-index.md

natebower · 2024-04-16T10:54:09Z

_search-plugins/knn/knn-vector-quantization.md

+
+In OpenSearch, the training vectors need to be present in an index. In general, the amount of training data will depend on which ANN algorithm will be used and how much data will go into the index. For IVF-based indices, a good number of training vectors to use is `max(1000*nlist, 2^code_size * 1000)`. For HNSW-based indexes, a good number is `2^code_size*1000` training vectors. See [Faiss's documentation](https://github.com/facebookresearch/faiss/wiki/FAQ#how-many-training-points-do-i-need-for-k-means) for more details about the methodology behind calculating these figures.
+
+For PQ, the two parameters that need to be selected are _m_ and _code_size_. _m_ determines how many sub-vectors the vectors should be split to encode separately. Consequently, the _dimension_ needs to be divisible by _m_. _code_size_ determines how many bits each sub-vector will be encoded with. In general, a good place to start is setting `code_size = 8` and then tuning _m_ to get the desired trade-off between memory footprint and recall.


I'm not following the second sentence here. Do we mean something like "m determines the number of subvectors into which vectors should be split for separate encoding"? In the fourth sentence, is "with" the correct preposition, or should it be "into"?

Yes, your rewrite is correct. I revised the following sentence to read: _code_size_ determines the number of bits used to encode each subvector.

_search-plugins/knn/knn-vector-quantization.md

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

_search-plugins/knn/knn-vector-quantization.md

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

_search-plugins/knn/knn-vector-quantization.md

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Address editorial feedback Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli · 2024-04-16T15:00:27Z

@jmazanec15 @vagimeli Please see my comments and changes and let me know if you have any questions. Thanks!

@natebower Thank you for the review. I accepted your edits and addressed the rewrite comments.

vagimeli

Doc review and editorial review completed

vagimeli · 2024-04-16T15:01:45Z

@jmazanec15 @vagimeli Please see my comments and changes and let me know if you have any questions. Thanks!

@natebower Thank you for the review. I accepted your edits and addressed the rewrite comments.

* Adds section on product quantization for docs Adds section in vector quantization docs for product quantization. In it, it contains tips for using it as well as memory estimations. Along with this, changed some formatting to make docs easier to write. Signed-off-by: John Mazanec <jmazane@amazon.com> * Update knn-vector-quantization.md Fix formatting Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update knn-vector-quantization.md Define abbreviation on first mention Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: John Mazanec <jmazane@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-index.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update knn-index.md Formatting and copyedits Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-index.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-index.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-index.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-index.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-index.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _search-plugins/knn/knn-vector-quantization.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update knn-vector-quantization.md Address editorial feedback Signed-off-by: Melissa Vagi <vagimeli@amazon.com> --------- Signed-off-by: John Mazanec <jmazane@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com> (cherry picked from commit 9a6bb8a) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

jmazanec15 requested review from hdhalter, kolchfa-aws, Naarcha-AWS, vagimeli, AMoo-Miki, natebower, dlvenable, stephen-crawford and epugh as code owners April 9, 2024 00:40

jmazanec15 force-pushed the knn-pq-improved-docs branch from 6bdae77 to 58058f6 Compare April 9, 2024 00:42

vamshin reviewed Apr 9, 2024

View reviewed changes

_search-plugins/knn/knn-vector-quantization.md Outdated Show resolved Hide resolved

jmazanec15 force-pushed the knn-pq-improved-docs branch from 58058f6 to 4f1bd63 Compare April 9, 2024 17:32

jmazanec15 requested a review from vamshin April 9, 2024 17:32

vamshin approved these changes Apr 9, 2024

View reviewed changes

hdhalter added 4 - Doc review PR: Doc review in progress backport 2.13 PR: Backport label for 2.13 labels Apr 10, 2024

Naarcha-AWS self-assigned this Apr 10, 2024

vagimeli added 3 commits April 10, 2024 12:11

Update knn-vector-quantization.md

0370b85

Fix formatting Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update knn-vector-quantization.md

548805b

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update knn-vector-quantization.md

167cb96

Define abbreviation on first mention Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli reviewed Apr 10, 2024

View reviewed changes

jmazanec15 and others added 8 commits April 10, 2024 14:13

Update _search-plugins/knn/knn-vector-quantization.md

be1c836

Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: John Mazanec <jmazane@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

255a25b

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-index.md

08194cb

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

0a0e105

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

83503a8

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

1e413bc

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

050064e

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

f2f42ee

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

natebower reviewed Apr 16, 2024

View reviewed changes

vagimeli and others added 6 commits April 16, 2024 08:47

Update _search-plugins/knn/knn-index.md

5231b87

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-index.md

be1e447

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-index.md

c78bc75

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-index.md

036db27

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-index.md

08a9857

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

55bf87c

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli reviewed Apr 16, 2024

View reviewed changes

_search-plugins/knn/knn-vector-quantization.md Outdated Show resolved Hide resolved

vagimeli and others added 2 commits April 16, 2024 08:49

Update _search-plugins/knn/knn-vector-quantization.md

38aca8d

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

5591d88

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli reviewed Apr 16, 2024

View reviewed changes

_search-plugins/knn/knn-vector-quantization.md Outdated Show resolved Hide resolved

vagimeli and others added 11 commits April 16, 2024 08:49

Update _search-plugins/knn/knn-vector-quantization.md

2b51288

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

2c389a3

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

5b0f258

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

d054f3f

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

9cc412f

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

ff0bebc

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

5b68721

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

b15ac62

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

072f1e6

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _search-plugins/knn/knn-vector-quantization.md

ead048e

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update knn-vector-quantization.md

2721679

Address editorial feedback Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli approved these changes Apr 16, 2024

View reviewed changes

vagimeli removed the 5 - Editorial review PR: Editorial review in progress label Apr 16, 2024

vagimeli merged commit 9a6bb8a into opensearch-project:main Apr 16, 2024
7 checks passed

opensearch-trigger-bot bot mentioned this pull request Apr 16, 2024

[Backport 2.13] Adds section on product quantization for docs #6959

Merged

github-actions bot pushed a commit that referenced this pull request Apr 16, 2024

Adds section on product quantization for docs (#6926) (#6959)

764bfb9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds section on product quantization for docs #6926

Adds section on product quantization for docs #6926

jmazanec15 commented Apr 9, 2024 •

edited

Loading

vamshin left a comment

vagimeli left a comment

vagimeli Apr 10, 2024

jmazanec15 Apr 10, 2024

vagimeli Apr 12, 2024 •

edited

Loading

natebower left a comment

natebower Apr 16, 2024

vagimeli Apr 16, 2024

vagimeli commented Apr 16, 2024

vagimeli left a comment

vagimeli commented Apr 16, 2024


		In OpenSearch, the training vectors need to be present in an index. In general, the amount of training data will depend on which ANN algorithm will be used and how much data will go into the index. For IVF-based indices, a good number of training vectors to use is `max(1000nlist, 2^code_size 1000)`. For HNSW-based indexes, a good number is `2^code_size*1000` training vectors. See [Faiss's documentation](https://github.com/facebookresearch/faiss/wiki/FAQ#how-many-training-points-do-i-need-for-k-means) for more details about the methodology behind calculating these figures.

		For PQ, the two parameters that need to be selected are _m_ and _code_size_. _m_ determines how many sub-vectors the vectors should be split to encode separately. Consequently, the _dimension_ needs to be divisible by _m_. _code_size_ determines how many bits each sub-vector will be encoded with. In general, a good place to start is setting `code_size = 8` and then tuning _m_ to get the desired trade-off between memory footprint and recall.

Adds section on product quantization for docs #6926

Adds section on product quantization for docs #6926

Conversation

jmazanec15 commented Apr 9, 2024 • edited Loading

Description

Issues Resolved

Checklist

vamshin left a comment

Choose a reason for hiding this comment

vagimeli left a comment

Choose a reason for hiding this comment

vagimeli Apr 10, 2024

Choose a reason for hiding this comment

jmazanec15 Apr 10, 2024

Choose a reason for hiding this comment

vagimeli Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

natebower Apr 16, 2024

Choose a reason for hiding this comment

vagimeli Apr 16, 2024

Choose a reason for hiding this comment

vagimeli commented Apr 16, 2024

vagimeli left a comment

Choose a reason for hiding this comment

vagimeli commented Apr 16, 2024

jmazanec15 commented Apr 9, 2024 •

edited

Loading

vagimeli Apr 12, 2024 •

edited

Loading