Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

find the suitable size for SST #118

Closed
4 tasks done
git-hulk opened this issue Aug 21, 2020 · 2 comments · Fixed by #120
Closed
4 tasks done

find the suitable size for SST #118

git-hulk opened this issue Aug 21, 2020 · 2 comments · Fixed by #120
Assignees
@karelrooted
Labels
enhancement type enhancement feature type new feature

Comments

@git-hulk
Copy link
Member

git-hulk commented Aug 21, 2020 •
edited by karelrooted
Loading

the key-value size may so be quite different in many scenes, and use 256MiB as SST file size may cause data loading(large index/filter block) ineffective when the key-value was too small. kvrocks supports user-defined SST file in config(rocksdb.target_file_size_base), but it still too trivial and inconvenient to adjust the different sizes for different instances. so we want to periodic auto-adjust the SST size in-flight with user avg key-value size. for example:

case 1: 0 ~ 128 bytes，Block Size = 2k, SST size = 16 M

case 2:  128 ~ 1024 bytes, Block Size = 8k, SST size = 64 M

case 3: 1k ~ 32k, Block Size = 32k,   SST size = 128 M

case 4: 32k ~ 256k, Block Size = 256k, SST size = 256 M

case 5: 256k ~ 512k, Block Size = 512k , SST size = 512 M

case 6: > 512 k, Block Size = 1M, SST size = 1 G

TODO:

  • develop the feature

  • code review

  • benchmark between fixed and dynamic SST size(32bytes/128bytes/512bytes/1k/16k/64k/256k/1M)

  • how about dynamic block size? which may affect the performance of the block cache hit rate
@git-hulk git-hulk added enhancement type enhancement feature type new feature labels Aug 21, 2020
@karelrooted karelrooted mentioned this issue Aug 24, 2020
Merged
@karelrooted
Copy link
Contributor

karelrooted commented Sep 11, 2020 •
edited
Loading

Benchmark(String):

Hardware

  • Macbook Pro (15-inch, 2017)

Software

  • Centos 7 VM
    • Memory: 4 GiB
  • Kvrocks
    • options->use_direct_reads = true (don't use os's page cache)
    • rocksdb.metadata_block_cache_size 60M

Result

* redis-benchmark -h 10.211.55.11 -p 6666  -r 10000000 -n 10000000 -a foobared -t set -d 32
* redis-benchmark -h 10.211.55.11 -p 6666  -r 10000000 -n 10000000 -a foobared -t get

Size        FIXED(256M)                                                         Dynamic(16M)
            
            Set QPS   99.99%                                                    Set QPS      99.99%      
32byte      42520.62  <= 68ms                                                   48253        <= 22ms    


            Get QPS   99.99%     Avg      Slowlog_Avg(last_128）                Get QPS     99.99%      Avg           Slowlog_Avg(last_128)
            7647.54   <= 137ms   0.954ms   23.5449ms                            24759.83    <= 25ms     0.270ms      18.265ms (total 80)   
            
 
* redis-benchmark -h 10.211.55.11 -p 6666  -r 10000000 -n 10000000 -a foobared -t set -d 256
* redis-benchmark -h 10.211.55.11 -p 6666  -r 10000000 -n 100000 -a foobared -t get

Size        FIXED(256M)                                                         Dynamic(64M)
 
            Set QPS   99.99%                                                    Set QPS      99.99%      
256byte     38705.54  <= 35ms                                                   37189.38     <= 36ms    


            Get QPS   99.99%     Avg       Slowlog_Avg(last_128）               Get QPS     99.99%      Avg           Slowlog_Avg(last_128)
            282.55    <= 488ms   28.131ms   34.8564ms                           1203.5    <= 154ms      6.576ms        13.3628ms
      
    
* redis-benchmark -h 10.211.55.11 -p 6666  -r 10000000 -n 10000000 -a foobared -t set -d 2048
* redis-benchmark -h 10.211.55.11 -p 6666  -r 10000000 -n 10000 -a foobared -t get

Size        FIXED(256M)                                                         Dynamic(128M)
 
            Set QPS   99.99%                                                    Set QPS      99.99%      
2K          14737.63  <= 1707ms                                                 13520.99     <= 242ms    


            Get QPS   99.99%      Avg       Slowlog_Avg(last_128）              Get QPS     99.99%         Avg           Slowlog_Avg(last_128)
            41.9      <= 3234ms   188.94ms   198.149ms                          211.33      <= 1296ms      37.602ms       87.2035ms     
            
            
* redis-benchmark -h 10.211.55.11 -p 6666  -r 200000 -n 200000 -a foobared -t set -d 49152
* redis-benchmark -h 10.211.55.11 -p 6666  -r 200000 -n 200000 -a foobared -t get

Size                                                                           Dynamic(256M)
 
                                                                               Set QPS      99.99%      
48K                                                                            1633.44      <= 7032ms    


                                                                               Get QPS     99.99%         Avg           Slowlog_Avg(last_128)
                                                                               3914.2      <= 76ms        1.856ms         11.443ms     
                                                                               
                                                                               
                                                                               
* redis-benchmark -h 10.211.55.11 -p 6666  -r 10000 -n 10000 -a foobared -t set -d 319488
* redis-benchmark -h 10.211.55.11 -p 6666  -r 10000 -n 10000 -a foobared -t get

Size        FIXED(256M)                                                         Dynamic(512M)
 
            Set QPS   99.99%                                                    Set QPS      99.99%      
312K        348.78    <= 4961ms                                                 433.88       <= 1595ms    


            Get QPS   99.97%      Avg       Slowlog_Avg(last_128）              Get QPS     99.87%         Avg           Slowlog_Avg(last_128)
            2260.4    <= 63ms    2.399ms     14.2772ms                          2481.39     <= 55ms        2.294ms        12.6723ms (total 75)
            
            
* redis-benchmark -h 10.211.55.11 -p 6666  -r 10000 -n 10000 -a foobared -t set -d 524288
* redis-benchmark -h 10.211.55.11 -p 6666  -r 10000 -n 10000 -a foobared -t get

Size        FIXED(256M)                                                         Dynamic(1G)
 
            Set QPS   99.99%                                                    Set QPS      99.99%      
512K        160.72    <= 6580ms                                                 200.73       <= 4407ms    


            Get QPS   99.89%      Avg       Slowlog_Avg(last_128）              Get QPS     99.99%         Avg           Slowlog_Avg(last_128)
            1594.39   <= 94ms    3.228ms     13.5389ms                          1700.10     <= 85ms       2.711ms         13.3244ms

@karelrooted
Copy link
Contributor

karelrooted commented Oct 16, 2020 •
edited
Loading

related issue: facebook/rocksdb#7385
related PR: facebook/rocksdb#7936

@git-hulk git-hulk mentioned this issue Jun 2, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@karelrooted karelrooted

Labels
enhancement type enhancement feature type new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

dynamic resize block and sst apache/kvrocks
2 participants
@karelrooted @git-hulk