lightning: optimize the performance of compression when sending kv pairs to tikv #41970
Labels
component/lightning
This issue is related to Lightning of TiDB.
type/enhancement
The issue or PR belongs to an enhancement.
Enhancement
In #41164, lightning supports compression when sending kv pairs to tikv. The default compression algorithm is gzip. However, after a large scale test, we found that the performance of the gzip compression algorithm is not ideal. In the test, we import 562GiB data into TiKV, with compression enabled, the total time is 1h37m56s, while with compression disabled, the total time is 33m9s.
After investigation, we found the compression is the bottleneck of the import speed. From the flame graph, most of the time is spent on the
Write
function of thegzip.Writer
.https://github.com/klauspost/compress/tree/master/gzip is a faster implementation of gzip compression algorithm. From the benchmark result https://github.com/klauspost/pgzip#compression-1, it is 7.52x faster than the standard library. So we can use it to replace the standard library to improve the performance of compression.
The text was updated successfully, but these errors were encountered: