feat(hyperloglog): add support of the Hyperloglog data structure #2142

tutububug · 2024-03-07T09:27:00Z

storage format description: https://github.com/apache/kvrocks-website/pull/207/files.

Only dense is supported now, and Merge is moved in d3b2978 / 22923f9 and would be back in coming patches

mapleFU

I'll take a careful review this week

src/types/redis_hyperloglog.cc

torwig · 2024-03-07T11:52:45Z

@tutububug Thank you for your contribution. Running ./x.py format should eliminate the linting issues.

Yangsx-1 · 2024-03-07T12:02:05Z

Thank you for your contribution! Maybe you need to add a command parser to use the data structure with actual command like redis.

PragmaTwice · 2024-03-07T13:31:47Z

Thank you for your contribution!

Could you include your design in the PR description? For example, explain how to encode the metadata and HLL data (subkeys), similar to what is shown on https://kvrocks.apache.org/community/data-structure-on-rocksdb.

src/storage/redis_db.cc

src/storage/redis_metadata.h

tutububug · 2024-03-07T15:11:58Z

Thank you for your contribution! Maybe you need to add a command parser to use the data structure with actual command like redis.

Yes, I will give the commit later.

Thank you for your contribution!

Could you include your design in the PR description? For example, explain how to encode the metadata and HLL data (subkeys), similar to what is shown on https://kvrocks.apache.org/community/data-structure-on-rocksdb.

OK.

tutububug · 2024-03-08T10:31:33Z

@PragmaTwice I create a PR(apache/kvrocks-website#207) for describe hyperloglog storage format.

PragmaTwice · 2024-03-09T06:51:25Z

@PragmaTwice I create a PR(apache/kvrocks-website#207) for describe hyperloglog storage format.

Thank you!
Currently, it is only necessary to include your design in the PR description, not on the website as the design is not finalized yet.

Regarding your design, I have some questions:

Based on the current design, typically one Redis key will introduce a maximum of 16384 RocksDB keys (registers). Each value corresponding to a RocksDB key contains only one integer. This may be inefficient; merging multiple registers onto one key could reduce the number of keys introduced. WDYT?
I noticed that integers are stored as string representations (std::to_string) rather than their binary form (e.g., subkey and register value). What is the reason for this approach?
Having a constant size seems illogical since the number of subkeys linked to this Redis key varies. (However, if every write operation modifies the metadata, it may lead to a decrease in performance. I don't have a clear idea about this aspect.)

Concerning the code, although I haven't reviewed it thoroughly yet, there are some points worth mentioning:

The complete source code of MurmurHash can be placed in the vendor directory.
It appears that using PFADD in the code leads to an increasing number of RocksDB keys (registers). There seems to be no operation that reduces these keys until deleting this Redis key. How can we prevent an increase in RocksDB keys without a mechanism to decrease them?

cc @git-hulk @mapleFU

git-hulk · 2024-03-09T07:21:32Z

@tutububug As @PragmaTwice mentioned in #2142 (comment), it's unnecessary to use a static number of 16384 since it may heavily affect the read performance while using PFMERGE, I guess a smaller one like 16 is enough and every subkey has 1000 integers.

src/types/redis_hyperloglog.cc

src/common/status.h

PragmaTwice · 2024-03-09T07:39:26Z

I suggest that we can store the number of registers in one rocksdb key in the metadata (for example, referred to as register_number_in_one_key), so that even if adjustments are made later for performance reasons, the compatibility of kvrocks data can be maintained.

Currently, a fixed value for register_number_in_one_key can be hardcoded in the code.

cc @tutububug

src/types/redis_hyperloglog.cc

mapleFU · 2024-03-09T08:13:23Z

Based on the current design, typically one Redis key will introduce a maximum of 16384 RocksDB keys (registers). Each value corresponding to a RocksDB key contains only one integer. This may be inefficient; merging multiple registers onto one key could reduce the number of keys introduced. WDYT?

There're two parts of things.

Bitmap type can be used to do the dense logic, which would reducing the keycount
sparse hll type can be introduced to save some overhead here.

PragmaTwice · 2024-03-09T08:24:34Z

Based on the current design, typically one Redis key will introduce a maximum of 16384 RocksDB keys (registers). Each value corresponding to a RocksDB key contains only one integer. This may be inefficient; merging multiple registers onto one key could reduce the number of keys introduced. WDYT?

There're two parts of things.

Bitmap type can be used to do the dense logic, which would reducing the keycount

sparse hll type can be introduced to save some overhead here.

So I think the question is, should we also introduce two mode of hll encoding (sparse and dense layout) and an auto switching policy between these two layout?

mapleFU · 2024-03-09T08:35:38Z

So I think the question is, should we also introduce two mode of hll encoding (sparse and dense layout) and an auto switching policy between these two layout?

I prefer to do this :-) But we can regard it as a further optimization

src/types/redis_hyperloglog.cc

src/types/redis_hyperloglog.h

src/types/redis_hyperloglog.cc

mapleFU · 2024-03-09T12:24:46Z

src/types/redis_hyperloglog.cc

+
+/* Store the value of the register at position 'regnum' into variable 'target'.
+ * 'p' is an array of unsigned bytes. */
+#define HLL_DENSE_GET_REGISTER(target, p, regnum)             \


can you change it to a function?

src/types/redis_hyperloglog.cc

mapleFU · 2024-03-09T12:45:51Z

Other Looks ok to me

tutububug · 2024-03-09T17:06:22Z

Regarding your design, I have some questions:

1. Based on the current design, typically one Redis key will introduce a maximum of 16384 RocksDB keys (registers). Each value corresponding to a RocksDB key contains only one integer. This may be inefficient; merging multiple registers onto one key could reduce the number of keys introduced. WDYT?

Not really. The register(subkey) only be stored which its count is not zero. This point is different from the memory implementation with static array as dense encode. On disk, I think its sparse encode naturally.

2. I noticed that integers are stored as string representations (`std::to_string`) rather than their binary form (e.g., subkey and register value). What is the reason for this approach?

The number of consecutive 0s is calculated from the last 50 digits of the hash value, so the maximum value is 50, and the maximum value stored in a string is 2 bytes. It should not waste a lot of space, and at the same time save the overhead of integer encoding and decoding. For keys, it may be more efficient, but the largest index is only 5 bytes (16383).

3. Having a constant `size` seems illogical since the number of subkeys linked to this Redis key varies. (However, if every write operation modifies the metadata, it may lead to a decrease in performance. I don't have a clear idea about this aspect.)

Currently, the ‘size’ variable has no practical purpose; the only requirement is that it be non-zero. Due to the implementation of kvrocks getmetadata, non-string type data structures with a size of 0 are judged to be expired, and the HLL add parameter that redis has implemented allows no parameters but the key will be stored. For compatibility, size is used as a constant to prevent expiration.

Concerning the code, although I haven't reviewed it thoroughly yet, there are some points worth mentioning:
1. The complete source code of MurmurHash can be placed in the `vendor` directory.

OK.

2. It appears that using `PFADD` in the code leads to an increasing number of RocksDB keys (registers). There seems to be no operation that reduces these keys until deleting this Redis key. How can we prevent an increase in RocksDB keys without a mechanism to decrease them?

For an HLL user key, the maximum register value is 16384, and it cannot be larger. In fact, I think this should be considered controllable compared to data structures such as hash, set, list, etc. whose size is determined by user input.

@PragmaTwice cc @git-hulk @mapleFU

mapleFU · 2024-03-09T17:48:04Z

The number of consecutive 0s is calculated from the last 50 digits of the hash value, so the maximum value is 50, and the maximum value stored in a string is 2 bytes. It should not waste a lot of space, and at the same time save the overhead of integer encoding and decoding. For keys, it may be more efficient, but the largest index is only 5 bytes (16383).

For rocksdb value, it's should be 1-2 bytes payload, the value size is also included. So, it introduce an extremly huge overhead. So I prefer the impl of bitmap/bitfield.

Get 2^14 times in rocksdb would also be heavy, and might break some statistics in rocksdb, which making it tent to compaction more or caching the unneccessary blocks.

PragmaTwice · 2024-07-30T16:13:01Z

src/storage/redis_metadata.h

@@ -49,6 +49,8 @@ enum RedisType : uint8_t {
  kRedisStream = 8,
  kRedisBloomFilter = 9,
  kRedisJson = 10,
+  kRedisSearch = 11,


need to delete

src/storage/redis_metadata.h

PragmaTwice · 2024-07-30T16:32:04Z

src/commands/cmd_hll.cc

+};
+
+REDIS_REGISTER_COMMANDS(MakeCmdAttr<CommandPfAdd>("pfadd", -2, "write", 1, 1, 1),
+                        MakeCmdAttr<CommandPfCount>("pfcount", 2, "read-only", 1, 1, 1), );


How about PFMERGE?

Would be add in next patch

src/storage/redis_metadata.cc

mapleFU · 2024-07-30T16:35:00Z

src/types/hyperloglog.cc

+  DCHECK_LT(index, kHyperLogLogRegisterCount);
+  hash >>= kHyperLogLogRegisterCountPow; /* Remove bits used to address the register. */
+  hash |= (static_cast<uint64_t>(1U) << kHyperLogLogHashBitCount);
+  uint8_t ctz = __builtin_ctzll(hash) + 1;


This is a little difference from valkey code, I've send a pr to them

src/types/redis_hyperloglog.cc

src/types/redis_hyperloglog.h

src/storage/redis_metadata.h

git-hulk · 2024-07-31T02:03:42Z

I think it's good to merge after fixing the lint issues.

src/commands/cmd_hll.cc

mapleFU · 2024-07-31T05:34:37Z

Merged, thanks all!

sonarcloud · 2024-07-31T05:40:24Z

Quality Gate passed

Issues
55 New issues
0 Accepted issues

Measures
0 Security Hotspots
59.6% Coverage on New Code
1.1% Duplication on New Code

See analysis details on SonarCloud

Support Hyperloglog

3e92895

tutububug mentioned this pull request Mar 7, 2024

Add support of the new data structure HyperLogLog #1307

Open

mapleFU reviewed Mar 7, 2024

View reviewed changes

src/types/redis_hyperloglog.cc Outdated Show resolved Hide resolved

src/types/redis_hyperloglog.cc Outdated Show resolved Hide resolved

add origin copyright

e8cdf69

PragmaTwice reviewed Mar 7, 2024

View reviewed changes

src/storage/redis_db.cc Outdated Show resolved Hide resolved

PragmaTwice reviewed Mar 7, 2024

View reviewed changes

src/storage/redis_metadata.h Outdated Show resolved Hide resolved

yangxiao added 2 commits March 8, 2024 12:11

code tuning

d4b174d

code format

c780c84

PragmaTwice changed the title ~~Support Hyperloglog~~ Add support of the Hyperloglog data structure Mar 9, 2024

git-hulk self-requested a review March 9, 2024 07:25

git-hulk reviewed Mar 9, 2024

View reviewed changes

src/types/redis_hyperloglog.cc Outdated Show resolved Hide resolved

src/common/status.h Outdated Show resolved Hide resolved

git-hulk reviewed Mar 9, 2024

View reviewed changes

src/types/redis_hyperloglog.cc Outdated Show resolved Hide resolved

mapleFU reviewed Mar 9, 2024

View reviewed changes

src/types/redis_hyperloglog.cc Outdated Show resolved Hide resolved

tutububug closed this Mar 9, 2024

tutububug reopened this Mar 9, 2024

mapleFU marked this pull request as ready for review July 30, 2024 15:52

Update vendor lib

008ec3f

mapleFU changed the title ~~Add support of the Hyperloglog data structure~~ feat(HLL): Add support of the Hyperloglog data structure Jul 30, 2024

Trying to fix lint

afdb7a7

PragmaTwice reviewed Jul 30, 2024

View reviewed changes

src/storage/redis_metadata.h Outdated Show resolved Hide resolved

PragmaTwice reviewed Jul 30, 2024

View reviewed changes

src/storage/redis_metadata.h Outdated Show resolved Hide resolved

PragmaTwice reviewed Jul 30, 2024

View reviewed changes

src/storage/redis_metadata.cc Outdated Show resolved Hide resolved

mapleFU reviewed Jul 30, 2024

View reviewed changes

remove bad conflict resolve

d8cc9a1

PragmaTwice reviewed Jul 30, 2024

View reviewed changes

src/types/redis_hyperloglog.cc Outdated Show resolved Hide resolved

PragmaTwice reviewed Jul 30, 2024

View reviewed changes

src/types/redis_hyperloglog.h Outdated Show resolved Hide resolved

update comments

22923f9

PragmaTwice reviewed Jul 30, 2024

View reviewed changes

src/storage/redis_metadata.h Outdated Show resolved Hide resolved

Change HLL to 11

e4f3e77

PragmaTwice previously approved these changes Jul 30, 2024

View reviewed changes

PragmaTwice requested a review from git-hulk July 30, 2024 17:08

trying to fix lint

b6fc4a1

mapleFU dismissed PragmaTwice’s stale review via b6fc4a1 July 31, 2024 02:50

PragmaTwice reviewed Jul 31, 2024

View reviewed changes

src/commands/cmd_hll.cc Show resolved Hide resolved

git-hulk approved these changes Jul 31, 2024

View reviewed changes

PragmaTwice approved these changes Jul 31, 2024

View reviewed changes

PragmaTwice changed the title ~~feat(HLL): Add support of the Hyperloglog data structure~~ feat(hyperloglog): Add support of the Hyperloglog data structure Jul 31, 2024

PragmaTwice changed the title ~~feat(hyperloglog): Add support of the Hyperloglog data structure~~ feat(hyperloglog): add support of the Hyperloglog data structure Jul 31, 2024

mapleFU mentioned this pull request Jul 31, 2024

Tracking issue: HyperLogLog #2454

Open

9 tasks

mapleFU merged commit 727bec2 into apache:unstable Jul 31, 2024
31 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(hyperloglog): add support of the Hyperloglog data structure #2142

feat(hyperloglog): add support of the Hyperloglog data structure #2142

tutububug commented Mar 7, 2024 •

edited by mapleFU

Loading

mapleFU left a comment

torwig commented Mar 7, 2024

Yangsx-1 commented Mar 7, 2024

PragmaTwice commented Mar 7, 2024

tutububug commented Mar 7, 2024 •

edited by aleksraiden

Loading

tutububug commented Mar 8, 2024

PragmaTwice commented Mar 9, 2024 •

edited

Loading

git-hulk commented Mar 9, 2024

PragmaTwice commented Mar 9, 2024 •

edited

Loading

mapleFU commented Mar 9, 2024

PragmaTwice commented Mar 9, 2024

mapleFU commented Mar 9, 2024 •

edited

Loading

mapleFU Mar 9, 2024

mapleFU commented Mar 9, 2024

tutububug commented Mar 9, 2024

mapleFU commented Mar 9, 2024 •

edited

Loading

PragmaTwice Jul 30, 2024

PragmaTwice Jul 30, 2024

mapleFU Jul 30, 2024

mapleFU Jul 30, 2024

git-hulk commented Jul 31, 2024

mapleFU commented Jul 31, 2024

sonarcloud bot commented Jul 31, 2024

feat(hyperloglog): add support of the Hyperloglog data structure #2142

feat(hyperloglog): add support of the Hyperloglog data structure #2142

Conversation

tutububug commented Mar 7, 2024 • edited by mapleFU Loading

mapleFU left a comment

Choose a reason for hiding this comment

torwig commented Mar 7, 2024

Yangsx-1 commented Mar 7, 2024

PragmaTwice commented Mar 7, 2024

tutububug commented Mar 7, 2024 • edited by aleksraiden Loading

tutububug commented Mar 8, 2024

PragmaTwice commented Mar 9, 2024 • edited Loading

git-hulk commented Mar 9, 2024

PragmaTwice commented Mar 9, 2024 • edited Loading

mapleFU commented Mar 9, 2024

PragmaTwice commented Mar 9, 2024

mapleFU commented Mar 9, 2024 • edited Loading

mapleFU Mar 9, 2024

Choose a reason for hiding this comment

mapleFU commented Mar 9, 2024

tutububug commented Mar 9, 2024

mapleFU commented Mar 9, 2024 • edited Loading

PragmaTwice Jul 30, 2024

Choose a reason for hiding this comment

PragmaTwice Jul 30, 2024

Choose a reason for hiding this comment

mapleFU Jul 30, 2024

Choose a reason for hiding this comment

mapleFU Jul 30, 2024

Choose a reason for hiding this comment

git-hulk commented Jul 31, 2024

mapleFU commented Jul 31, 2024

sonarcloud bot commented Jul 31, 2024

Quality Gate passed

tutububug commented Mar 7, 2024 •

edited by mapleFU

Loading

tutububug commented Mar 7, 2024 •

edited by aleksraiden

Loading

PragmaTwice commented Mar 9, 2024 •

edited

Loading

PragmaTwice commented Mar 9, 2024 •

edited

Loading

mapleFU commented Mar 9, 2024 •

edited

Loading

mapleFU commented Mar 9, 2024 •

edited

Loading