[WIP]sparse-quantizer #374
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Roxanne0321
requested review from
LHT129,
ShawnShawnYou,
inabao,
wxyucs and
jiaweizone
as code owners
ShawnShawnYou
force-pushed
the
support-sparse-dataset
branch
from
5a6c1ef to
35a8807
Compare
src/quantization/sparse_computer.h
Outdated
| } | ||
|
|
||
| inline void | ||
| ComputeDist(uint32_t u, float* dists) {//u是quant中存储的第u个稀疏向量 |
There was a problem hiding this comment.
computer and quantizer only consider compute distance from code to code, don't consider the storage
src/quantization/sparse_quantizer.h
Outdated
|
|
||
| bool | ||
| SparseQuantizer::InsertAll(const SparseVectors& sv) { | ||
| // 首先释放旧的内存 |
There was a problem hiding this comment.
ditto, for test the quantizer, this func of code can be moved to sparse_quantizer_test
src/quantization/sparse_quantizer.h
Outdated
| } | ||
|
|
||
| inline float Compute(int32_t nnz1, const uint32_t* ids1, const float* vals1, | ||
| int32_t nnz2, const uint32_t* ids2, const float* vals2) const { |
There was a problem hiding this comment.
implement encode and decode
src/quantization/sparse_quantizer.h
Outdated
| inline void | ||
| SparseQuantizer::ComputeDist(SparseComputer& computer, uint32_t u, float* dists) const { | ||
|
|
||
| if (u >= num_) { |
src/quantization/sparse_quantizer.h
Outdated
|
|
||
| inline float | ||
| SparseQuantizer::ComputeDist(SparseComputer& computer, uint32_t u) const { | ||
|
|
src/quantization/sparse_quantizer.h
Outdated
| } | ||
|
|
||
| inline void | ||
| SparseQuantizer::ProcessQuery(int32_t nnz, const uint32_t* ids, const float* vals, SparseComputer& computer) const { |
There was a problem hiding this comment.
implement in Encode
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implemented SparseQuantizer and SparseComputer classes, and completed functions such as sparse vector inner product calculation