Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bad output when compressing empty array, with Compression::none() #474

Open
xNxExOx opened this issue Feb 28, 2025 · 4 comments
Open

Bad output when compressing empty array, with Compression::none() #474

xNxExOx opened this issue Feb 28, 2025 · 4 comments

Comments

@xNxExOx
Copy link

xNxExOx commented Feb 28, 2025

I think this test example is enough to demonstrate the issue: Rust Playground

You can uncomment dbg!(&compressed_table); to see the huge result, that can not be decompressed afterwards.

Priority low, I resolved the issue on my side by using flate2::Compression::default() for empty tables, which works fine.

This is really weird edge case, so no need to fix it any time soon, because not many people is compressing empty stuff, and even less does so with flate2::Compression::none().

@oyvindln
Copy link
Contributor

Which version of miniz_oxide is it using? There was a bug introduced in miniz_oxide 0.8.3 that I fixed in 0.8.5 that caused corrupt output in this case so maybe rust playground hasn't updated yet. I was considering yanking the two versions though it seemed like an extreme edge case to happen in the wild though I guess if it does occur maybe I should as people can be slow to update.

You very rarely want to intentionally use no compression as zlib and other compressors already switch to it automatically if the data is not compressible (though there are probably edge cases where you would to intentionally not compress it for whatever reason). An "empty" file will be smaller with any compression level higher than none as the block header metadata for a deflate compressed block with static codes is smaller than for an uncompressed block.

@xNxExOx
Copy link
Author

xNxExOx commented Feb 28, 2025

Well my reason for using none might be stupid, but I sometimes read the files in hex editors, and dealing with double, or even triple compressed files is annoying, but that is weird edge case on its own. It have nothing to do with real size. Also having it without compression make it clear when the size grows over few GB, which compression hides very well to few MB, because that part of data is too well compressible.

For me locally I have: miniz_oxide: version = "0.8.4" based on Cargo.lock. Even after updating to flate2 version = "1.1.0" from version = "1.0.35". So I am not sure, if that is the issue.

And finding out that one of the tables is empty was surprise for me, so that is huge edge case too.

@xNxExOx
Copy link
Author

xNxExOx commented Feb 28, 2025

Even main branch is still on that version: https://github.com/rust-lang/flate2-rs/blob/main/Cargo.toml#L27

@oyvindln
Copy link
Contributor

If you run cargo update it should update to the latest version.

oyvindln added a commit to oyvindln/flate2-rs that referenced this issue Feb 28, 2025
0.8.4 has an edge case bug, see rust-lang#474
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants