Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use faster library for decompression when index is available #19

Closed
mxmlnkn opened this issue Jun 15, 2023 · 1 comment
Closed

Use faster library for decompression when index is available #19

mxmlnkn opened this issue Jun 15, 2023 · 1 comment
Labels
performance Something is slower than it could be

Comments

@mxmlnkn
Copy link
Owner

mxmlnkn commented Jun 15, 2023

When the index is available, we can use existing libraries, at least zlib works. zlib-ng might also work but performance improvements will be marginal according to simple tests #9 (comment). On the other hand, libdeflate might bring some significant performance improvements. However, it is questionable whether libdeflate works in this use case because we need to very specialized interface functions: "set window" and "inflate prime" in order to start in the middle of a byte from a deflate block directly. If something like inflatePrime is not offered, then we might have to modify libdeflate or we might be able to do some tricks like pigz does in order to byte-align separate deflate streams by inserting zero-size deflate blocks. However, our case might be harder to do. Byte-aligning is easy with empty non-compressed blocks but aligning to some subbit will require empty Fixed Huffman or Dynamic Huffman blocks.

I might even be able to use igzip although it might be troublesome to implement the NASM-dependent build process in setuptools, so I probably should only use igzip for building the wheels and not by default. The same question about how to implement the inflatePrime call as for libdeflate applies here. And a comparison benchmark would need to be added.

@mxmlnkn mxmlnkn added the performance Something is slower than it could be label Jun 15, 2023
@mxmlnkn
Copy link
Owner Author

mxmlnkn commented Aug 5, 2023

Implemented in 0.8.1. The NASM build process was indeed troublesome to integrate into setuptools for all three platforms.

@mxmlnkn mxmlnkn closed this as completed Aug 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Something is slower than it could be
Projects
None yet
Development

No branches or pull requests

1 participant