You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When the index is available, we can use existing libraries, at least zlib works. zlib-ng might also work but performance improvements will be marginal according to simple tests #9 (comment). On the other hand, libdeflate might bring some significant performance improvements. However, it is questionable whether libdeflate works in this use case because we need to very specialized interface functions: "set window" and "inflate prime" in order to start in the middle of a byte from a deflate block directly. If something like inflatePrime is not offered, then we might have to modify libdeflate or we might be able to do some tricks like pigz does in order to byte-align separate deflate streams by inserting zero-size deflate blocks. However, our case might be harder to do. Byte-aligning is easy with empty non-compressed blocks but aligning to some subbit will require empty Fixed Huffman or Dynamic Huffman blocks.
I might even be able to use igzip although it might be troublesome to implement the NASM-dependent build process in setuptools, so I probably should only use igzip for building the wheels and not by default. The same question about how to implement the inflatePrime call as for libdeflate applies here. And a comparison benchmark would need to be added.
The text was updated successfully, but these errors were encountered:
When the index is available, we can use existing libraries, at least zlib works. zlib-ng might also work but performance improvements will be marginal according to simple tests #9 (comment). On the other hand, libdeflate might bring some significant performance improvements. However, it is questionable whether libdeflate works in this use case because we need to very specialized interface functions: "set window" and "inflate prime" in order to start in the middle of a byte from a deflate block directly. If something like
inflatePrime
is not offered, then we might have to modify libdeflate or we might be able to do some tricks like pigz does in order to byte-align separate deflate streams by inserting zero-size deflate blocks. However, our case might be harder to do. Byte-aligning is easy with empty non-compressed blocks but aligning to some subbit will require empty Fixed Huffman or Dynamic Huffman blocks.I might even be able to use igzip although it might be troublesome to implement the NASM-dependent build process in setuptools, so I probably should only use igzip for building the wheels and not by default. The same question about how to implement the inflatePrime call as for libdeflate applies here. And a comparison benchmark would need to be added.
The text was updated successfully, but these errors were encountered: