Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

switch to lz4_flex #445

Merged
merged 1 commit into from
Sep 4, 2023
Merged

switch to lz4_flex #445

merged 1 commit into from
Sep 4, 2023

Conversation

PSeitz
Copy link
Contributor

@PSeitz PSeitz commented Jun 20, 2023

The release of lz4_flex 0.11 fixes two issue that caused problems
before:

  • The new auto-finish flushes on drop
  • FrameEncoder writes now empty frames. Previously nothing was written,
    which caused issues with double compressed empty data.

The release of lz4_flex 0.11 fixes two issue that caused problems
before:
- The new `auto-finish` flushes on drop
- FrameEncoder writes now empty frames. Previously nothing was written,
  which caused issues with double compressed empty data.
Copy link
Member

@marcospb19 marcospb19 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay.

I did some benchmarks before merging.

The more I chain decoders, the better results I get:

  1. .lz4 -> 1.3x
  2. .lz4.lz4 -> 1.6x
  3. .lz4.lz4.lz4.lz4 -> 2.0x
Summary
  ./ouch-new compress input.txt output.lz4.lz4 ran
    1.63 ± 0.02 times faster than ./ouch-previous compress input.txt output.lz4.lz4

I also saw improvements when decompressing.

Input file was generated with the following command:

dd if=/dev/random of=input.txt bs=1M count=500

All tests done in /tmp, because it is a filesystem that lives in-memory (doesn't add disk IO to the mix).

Benchmark command used:

hyperfine -w 3 -p 'rm output.lz4.lz4' './ouch-previous compress input.txt output.lz4.lz4' './ouch-new compress input.txt output.lz4.lz4'

In my machine (using nightly-x86_64-unknown-linux-gnu), I also saw a reduction in binary size of 123kb (from 3589 to 3466) (yes I did rebase this PR to account for other changes).

Thanks for your contribution!

@marcospb19 marcospb19 merged commit fe6fa92 into ouch-org:main Sep 4, 2023
@figsoda
Copy link
Member

figsoda commented Sep 4, 2023

Thank you for benchmarking this @marcospb19, and thank you for implementing this and fixing the bugs @PSeitz! I didn't have to run the benchmarks initially and kinda forgot about it (sorry), this looks great!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants