Support BITOP command

[ColinChamber]

No description provided.

[git-hulk]

cool, thanks for your hard work! @ColinChamber

This implementation may cause performance issue when the bitmap was sparsely, but it's more clear and simple than using the seek operator. So it looks good to me. HDYT @ShooterIT

[git-hulk]

cool, thanks for your hard work! @ColinChamber

This implementation may cause performance issue when the bitmap was too sparsely, but it's more clear and simple than using the seek operator. So it looks good to me. HDYT @ShooterIT

[ColinChamber]

This implementation may cause performance

I think so.
Currently, the bitop scene can't simply use the seek operator. Kvrocks use the default ordering function for key, which orders bytes lexicographically. For example, the order of the bitmap subkey after iteration is key|version|1024, key|version|10240, key|version|2048, not key|version|1024, key|version|2048, key|version|10240, which will make the operation of the same offset fragment between different keys unattainable. I think this is also the reason why bitcount and bitpos can't use the seek operator. In the future, we can do some refactoring, modify the encode and decode of the bitmap, and fill in the same number of digits. @git-hulk

[git-hulk]

@ColinChamber Yeah, the key order was easy to change, what i think was to seek out all fragments and apply the operator on those fragments without depending on the key order.

[ColinChamber]

Do you mean GetString and then do the operation? That seems not a suitable method that will use too much memory and is inefficient when operating on multiple keys.

[git-hulk]

@ColinChamber RocksDB write batch would also cache those fragments on memory before writing indeed, so it should be not different.

[ColinChamber]

RocksDB write batch would also cache those fragments on memory before writing indeed.

That is the memory consumed by destkey.
Seeking out all fragments will increase the memory consumed by srckey.

• BITOP AND destkey srckey1 srckey2 srckey3 ... srckeyN
• BITOP OR destkey srckey1 srckey2 srckey3 ... srckeyN
• BITOP XOR destkey srckey1 srckey2 srckey3 ... srckeyN
• BITOP NOT destkey srckey

[git-hulk]

Yeah, we can apply those fragments one by one, so the number of memory was same as current solution.

[ColinChamber]

It will depend on the subkey order. If we want to seek, then iterate one by one.

[git-hulk]

Sorry that I didn't explain clearly. What I mean was below:

1. alloc the dest bitmap with the max size
2. seek key fragments then apply the operator and fragment into the dest bitmap fragment with the subkey(it's ok that subkey was lexical order)
3. repeat step 2 if we have many keys

[ColinChamber]

Sorry for my late understanding. Your method will improve performance, but the number of calculations will increase. I think the best way is to iterate and calculate all srckeys simultaneously, which relies on getting subkeys in index order.

Getting subkeys in index order can also improve the performance of bitcount. What is the reason we don't do that? Is it to maintain compatibility with the previous version?

[git-hulk]

For why we didn't use seek operator at bitcount command, it's a design mistake because we only focus on hot commands like getbit and setbit, and didn't carefully design the key format before implementing the bitcount.

A compatible way is to use the extra bits to flag the new key format, but it's too tricky which I don't think we should do it now. So current implementation was good to me and can refactor it if those commands performance issue was the key point for us.

[git-hulk]

I have no advice on this PR. @ShooterIT @ChrisZMF

[ChrisZMF]

LGTM @ShooterIT