Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix config set compression type didn't take effect #1576

Merged
merged 11 commits into from
Jul 16, 2023

Conversation

xiaobiaozhao
Copy link
Contributor

@xiaobiaozhao xiaobiaozhao commented Jul 10, 2023

In this PR, we also refactor the c-style ConfigEnum array into the std::vector, the main reason is to make it easier to adopt dynamic config enums like the compression types. And for the test case, it's a bit hard to test if the compression type takes effect in underlying storage because we only enabled it when the levels >= 2. But will check it manually to make sure if works.

This closes #1560

@xiaobiaozhao xiaobiaozhao force-pushed the dynamic-compression branch from 201c9bd to 9fe9911 Compare July 10, 2023 16:13
src/storage/storage.cc Outdated Show resolved Hide resolved
@xiaobiaozhao xiaobiaozhao force-pushed the dynamic-compression branch from 9fe9911 to f82a9c7 Compare July 11, 2023 06:08
@aleksraiden aleksraiden changed the title Enable dynamin compression Enable dynamic compression Jul 11, 2023
@git-hulk git-hulk changed the title Enable dynamic compression Fix config set compression type didn't take effect Jul 11, 2023
src/storage/storage.cc Outdated Show resolved Hide resolved
src/config/config.cc Outdated Show resolved Hide resolved
src/config/config.h Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
Copy link
Member

@PragmaTwice PragmaTwice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code does not look well to me, some fixes need to be pushed

@xiaobiaozhao xiaobiaozhao force-pushed the dynamic-compression branch from 2957f68 to 46eb2ea Compare July 12, 2023 17:49
src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
@caipengbo
Copy link
Contributor

I think the SetCompressionOption parameter value is the name of the compression algorithm. We get rocksdb::CompressionType by map[name](name->CompressionType), and then pass it to SetOptions, in SetOptions static_cast it to rocksdb::CompressionType.

@git-hulk
Copy link
Member

@PragmaTwice @xiaobiaozhao @caipengbo I minor refactor the PR, PTAL again.

@git-hulk git-hulk marked this pull request as draft July 14, 2023 11:57
@git-hulk
Copy link
Member

git-hulk commented Jul 14, 2023

I have tested if the config set command works on my side:

Set the compression to LZ4:

2023/07/14-20:45:08.518174 6158610432 [db/db_impl/db_impl.cc:1195] SetOptions() on column family [default], inputs:
2023/07/14-20:45:08.518210 6158610432 [db/db_impl/db_impl.cc:1198] compression_per_level: kNoCompression:kNoCompression:kLZ4Compression:kLZ4Compression:kLZ4Compression:kLZ4Compression:kLZ4Compression

then use redis-benchmark -p 6666 -n 1000000 -r 5000000 -t set -d 1024 to write the data, and check the db LOG to see if the compression was used:

2023/07/14-20:50:15.181736 6156890112 (Original Log Time 2023/07/14-20:50:15.181594) [db/compaction/compaction_job.cc:888] [metadata] compacted to: files[0 1 13 0 0 0 0] 
max score 0.93, MB/sec: 39.1 rd, 4.5 wr, level 2, files in(1, 1) out(1 +0 blob) MB 
in(58.0, 2.6 +0.0 blob) out(7.1 +0.0 blob), read-write-amplify(1.2) write-amplify(0.1) OK, 
records in: 89927, records dropped: 3779 output_compression: LZ4

@git-hulk git-hulk marked this pull request as ready for review July 14, 2023 12:53
@git-hulk git-hulk self-requested a review July 14, 2023 14:46
src/config/config.cc Outdated Show resolved Hide resolved
src/storage/storage.cc Outdated Show resolved Hide resolved
git-hulk and others added 2 commits July 16, 2023 14:33
Co-authored-by: Twice <twice@apache.org>
caipengbo
caipengbo previously approved these changes Jul 16, 2023
Copy link
Contributor

@caipengbo caipengbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@git-hulk git-hulk requested a review from PragmaTwice July 16, 2023 09:00
@git-hulk git-hulk merged commit b67f268 into apache:unstable Jul 16, 2023
git-hulk added a commit to git-hulk/kvrocks that referenced this pull request Jul 30, 2023
Co-authored-by: git-hulk <hulk.website@gmail.com>
Co-authored-by: Twice <twice@apache.org>
@git-hulk git-hulk mentioned this pull request Jul 31, 2023
git-hulk added a commit that referenced this pull request Aug 1, 2023
Co-authored-by: git-hulk <hulk.website@gmail.com>
Co-authored-by: Twice <twice@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants