Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lightning: optimize the performance of compression when sending kv pairs to tikv #41970

Closed
sleepymole opened this issue Mar 7, 2023 · 0 comments · Fixed by #41974
Closed
Labels
component/lightning This issue is related to Lightning of TiDB. type/enhancement The issue or PR belongs to an enhancement.

Comments

@sleepymole
Copy link
Contributor

sleepymole commented Mar 7, 2023

Enhancement

In #41164, lightning supports compression when sending kv pairs to tikv. The default compression algorithm is gzip. However, after a large scale test, we found that the performance of the gzip compression algorithm is not ideal. In the test, we import 562GiB data into TiKV, with compression enabled, the total time is 1h37m56s, while with compression disabled, the total time is 33m9s.

After investigation, we found the compression is the bottleneck of the import speed. From the flame graph, most of the time is spent on the Write function of the gzip.Writer.
image

https://github.com/klauspost/compress/tree/master/gzip is a faster implementation of gzip compression algorithm. From the benchmark result https://github.com/klauspost/pgzip#compression-1, it is 7.52x faster than the standard library. So we can use it to replace the standard library to improve the performance of compression.

@sleepymole sleepymole added type/enhancement The issue or PR belongs to an enhancement. component/lightning This issue is related to Lightning of TiDB. labels Mar 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/lightning This issue is related to Lightning of TiDB. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant