-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement the xz Delta filter (filter 3) #101
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding some notes inline to aid review:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the contribution, and apologies for the delay as I was caught up in many other things this year. Overall this looks good, I left a few comments.
Great feedback! I think the new diff is much cleaner and clearer than the previous. Since I was touching decomp_big_file anyway, I took the liberty of replacing read_all_file with the new fs::read builtin while I was there. Please give it another look, at your convenience. |
Pull Request Overview
This pull request adds a delta compression filter and fixes multi-filter decompression; this appears to be the first test case making use of multiple filters.
Terrain height data tends to have lots of periodic repeating patterns, so GeoTiffs compressed with GDAL's liblzma support tend to have lots of tiles that make use of delta filtering. I need a native rust implementation of lzma decompression for use in OpenFA, so that I can port it to run on the web.
Testing Strategy
This pull request was tested by...
.xz
file with a delta filter).cargo bench
to verify there were no performance regressions.Supporting Documentation and References
Code for the delta filter was adapted from the C code in xz-file-format.txt.