Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstd: Improve Best compression #404

Merged
merged 7 commits into from
Aug 2, 2021
Merged

zstd: Improve Best compression #404

merged 7 commits into from
Aug 2, 2021

Conversation

klauspost
Copy link
Owner

@klauspost klauspost commented Jul 6, 2021

Compare expected literal encoding to predefined table code side and increase table sizes.

Before/after:

enwik8	zskp	4	100000000	31421593	4230	22.54
enwik8	zskp	4	100000000	29988041	9278	10.28
 => 4.6% better

silesia.tar	zskp	4	211947520	60995370	8825	22.90
silesia.tar	zskp	4	211947520	60104364	18082	11.18
 => 1.5% better

gob-stream	zskp	4	1911399616	167273881	30734	59.31
gob-stream	zskp	4	1911399616	165609838	50369	36.19
 => 1.0% better

github-ranks-backup.bin	zskp	4	1862623243	393605175	66308	26.79
github-ranks-backup.bin	zskp	4	1862623243	383959113	127082	13.98
 => 2.5% better

nyc-taxi-data-10M.csv	zskp	4	3325605752	490907191	80377	39.46
nyc-taxi-data-10M.csv	zskp	4	3325605752	476268884	135958	23.33
 => 3.0% better

@klauspost klauspost marked this pull request as ready for review August 2, 2021 15:03
@klauspost klauspost merged commit 6c2ac28 into master Aug 2, 2021
@klauspost klauspost deleted the zstd-improve-best branch August 2, 2021 16:18
mostynb added a commit to mostynb/zstdpool-syncpool that referenced this pull request Aug 8, 2021
This includes the following zstd improvements since v1.13.0:

* Add configurable Decoder window size
  klauspost/compress#394
* Add stream content size
  klauspost/compress#401
* Simplify hashing functions
  klauspost/compress#402
* use SpeedBestCompression for level >= 10
  klauspost/compress#410
* Fix WriteTo error forwarding
  klauspost/compress#411
* Improve Best compression
  klauspost/compress#404
mostynb added a commit to mostynb/zstdpool-syncpool that referenced this pull request Aug 8, 2021
This includes the following zstd improvements since v1.13.0:

* Add configurable Decoder window size
  klauspost/compress#394
* Add stream content size
  klauspost/compress#401
* Simplify hashing functions
  klauspost/compress#402
* use SpeedBestCompression for level >= 10
  klauspost/compress#410
* Fix WriteTo error forwarding
  klauspost/compress#411
* Improve Best compression
  klauspost/compress#404
mostynb added a commit to mostynb/go-grpc-compression that referenced this pull request Aug 8, 2021
This includes the following zstd improvements since v1.13.0:

* Add configurable Decoder window size
  klauspost/compress#394
* Add stream content size
  klauspost/compress#401
* Simplify hashing functions
  klauspost/compress#402
* use SpeedBestCompression for level >= 10
  klauspost/compress#410
* Fix WriteTo error forwarding
  klauspost/compress#411
* Improve Best compression
  klauspost/compress#404
mostynb added a commit to mostynb/go-grpc-compression that referenced this pull request Aug 8, 2021
This includes the following zstd improvements since v1.13.0:

* Add configurable Decoder window size
  klauspost/compress#394
* Add stream content size
  klauspost/compress#401
* Simplify hashing functions
  klauspost/compress#402
* use SpeedBestCompression for level >= 10
  klauspost/compress#410
* Fix WriteTo error forwarding
  klauspost/compress#411
* Improve Best compression
  klauspost/compress#404
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant