Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flate: Add limited window compression #843

Merged
merged 1 commit into from
Aug 9, 2023
Merged

Conversation

klauspost
Copy link
Owner

Adds a medium compressor that can operate with limited window size.

Exposed in gzip outside deflate for now.

Example sizes:

=== RUN   TestFileWindow/32
    gzip_test.go:349: size: 82504 bytes
=== RUN   TestFileWindow/64
    gzip_test.go:349: size: 75350 bytes
=== RUN   TestFileWindow/128
    gzip_test.go:349: size: 70668 bytes
=== RUN   TestFileWindow/256
    gzip_test.go:349: size: 69276 bytes
=== RUN   TestFileWindow/512
    gzip_test.go:349: size: 68327 bytes
=== RUN   TestFileWindow/1024
    gzip_test.go:349: size: 67876 bytes
=== RUN   TestFileWindow/2048
    gzip_test.go:349: size: 40900 bytes
=== RUN   TestFileWindow/4096
    gzip_test.go:349: size: 38684 bytes
=== RUN   TestFileWindow/8192
    gzip_test.go:349: size: 36263 bytes
=== RUN   TestFileWindow/16384
    gzip_test.go:349: size: 35434 bytes
=== RUN   TestFileWindow/32768
    gzip_test.go:349: size: 34654 bytes
--- PASS: TestFileWindow (0.03s)

Limited testing done.

Adds a medium compressor that can operate with limited window size.

Exposed in gzip outside deflate for now.

Example sizes:

```
=== RUN   TestFileWindow/32
    gzip_test.go:349: size: 82504 bytes
=== RUN   TestFileWindow/64
    gzip_test.go:349: size: 75350 bytes
=== RUN   TestFileWindow/128
    gzip_test.go:349: size: 70668 bytes
=== RUN   TestFileWindow/256
    gzip_test.go:349: size: 69276 bytes
=== RUN   TestFileWindow/512
    gzip_test.go:349: size: 68327 bytes
=== RUN   TestFileWindow/1024
    gzip_test.go:349: size: 67876 bytes
=== RUN   TestFileWindow/2048
    gzip_test.go:349: size: 40900 bytes
=== RUN   TestFileWindow/4096
    gzip_test.go:349: size: 38684 bytes
=== RUN   TestFileWindow/8192
    gzip_test.go:349: size: 36263 bytes
=== RUN   TestFileWindow/16384
    gzip_test.go:349: size: 35434 bytes
=== RUN   TestFileWindow/32768
    gzip_test.go:349: size: 34654 bytes
--- PASS: TestFileWindow (0.03s)
```

Limited testing done.
@klauspost
Copy link
Owner Author

@rojer - are you able to test if this does what you need?

@rojer
Copy link

rojer commented Aug 4, 2023

not immediately, but will do

@rojer
Copy link

rojer commented Aug 9, 2023

ok, i've run tests and happy to report that it works.
i didn't go as far as integrating it into our app, but i did the next closes thing: we use miniz in embedded applications. i patched tinfl_decompress_mem_to_callback to take dict size as well, coupled it with flate.NewWriterWindow and ran compress/decompress tests for all window sizes, from 32 to 32768. it works.
https://github.com/rojer/compress/commit/5a2c99d7f2c68bbb45e2257cc046ea44107131f3 is the test, i don't know if you'd like to adopt it for the library or not, but this is what i used:

$ go test -v -asan ./flate/miniz/...
=== RUN   Test_miniz_inflate
--- PASS: Test_miniz_inflate (0.10s)
PASS
ok      github.com/klauspost/compress/flate/miniz       0.107s

@klauspost
Copy link
Owner Author

@rojer Great! I will merge it then.

It seems like the next version will be a feature release, so it may take a bit to get out.

@klauspost klauspost merged commit b404607 into master Aug 9, 2023
18 checks passed
@klauspost klauspost deleted the limited-window-comp branch August 9, 2023 11:00
@rojer
Copy link

rojer commented Aug 9, 2023

thank you @klauspost , much appreciated!

kodiakhq bot referenced this pull request in cloudquery/filetypes Oct 1, 2023
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [github.com/klauspost/compress](https://togithub.com/klauspost/compress) | indirect | minor | `v1.16.7` -> `v1.17.0` |

---

### Release Notes

<details>
<summary>klauspost/compress (github.com/klauspost/compress)</summary>

### [`v1.17.0`](https://togithub.com/klauspost/compress/releases/tag/v1.17.0)

[Compare Source](https://togithub.com/klauspost/compress/compare/v1.16.7...v1.17.0)

#### What's Changed

-   Add dictionary builder by [@&#8203;klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/853](https://togithub.com/klauspost/compress/pull/853)
-   Add xerial snappy read/writer by [@&#8203;klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/838](https://togithub.com/klauspost/compress/pull/838)
-   flate: Add limited window compression by [@&#8203;klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/843](https://togithub.com/klauspost/compress/pull/843)
-   s2: Do 2 overlapping match checks by [@&#8203;klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/839](https://togithub.com/klauspost/compress/pull/839)
-   flate: Add amd64 assembly matchlen by [@&#8203;klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/837](https://togithub.com/klauspost/compress/pull/837)
-   gzip: Copy bufio.Reader on Reset by [@&#8203;thatguystone](https://togithub.com/thatguystone) in [https://github.com/klauspost/compress/pull/860](https://togithub.com/klauspost/compress/pull/860)
-   zstd: Remove offset from bitReader by [@&#8203;greatroar](https://togithub.com/greatroar) in [https://github.com/klauspost/compress/pull/854](https://togithub.com/klauspost/compress/pull/854)
-   fse, huff0, zstd: Remove always-nil error returns by [@&#8203;greatroar](https://togithub.com/greatroar) in [https://github.com/klauspost/compress/pull/857](https://togithub.com/klauspost/compress/pull/857)
-   tests: unnecessary use of fmt.Sprintf by [@&#8203;testwill](https://togithub.com/testwill) in [https://github.com/klauspost/compress/pull/836](https://togithub.com/klauspost/compress/pull/836)
-   tests: Fix OSS fuzzer t.Run by [@&#8203;klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/852](https://togithub.com/klauspost/compress/pull/852)
-   tests: Use Go 1.21.x by [@&#8203;klauspost](https://togithub.com/klauspost) in [https://github.com/klauspost/compress/pull/851](https://togithub.com/klauspost/compress/pull/851)

#### New Contributors

-   [@&#8203;testwill](https://togithub.com/testwill) made their first contribution in [https://github.com/klauspost/compress/pull/836](https://togithub.com/klauspost/compress/pull/836)
-   [@&#8203;thatguystone](https://togithub.com/thatguystone) made their first contribution in [https://github.com/klauspost/compress/pull/860](https://togithub.com/klauspost/compress/pull/860)

**Full Changelog**: klauspost/compress@v1.16.7...v1.17.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi4xMDkuNCIsInVwZGF0ZWRJblZlciI6IjM2LjEwOS40IiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants