Support Zip64 when compressing iterables and strings #25
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I use this to flush partial zips as files are streamed into them from requests, and then at the end add manifest and error files to the end of the archive I've also added a related test and example of use
|
Would it be possible to merge this? I ran into this issue because I need to stream very large amounts of data. |
|
Hi @matthewatabet, it seems the original developer of this package is not active anymore. So I've forked the package as zipstream-new: #33 |
Add partial flushing of ZipStreams
Find compression type.
When flushing, stream out the iterators in First in first out order. Python `pop()` with no arguments would take the last path but I think it makes sense to stream the first things first. We ran into this issue where we add a bunch of files which depend on long-running futures to provide the data. The futures hit a server which processes them roughly in order, so we get better streaming performance if we make this change.
Stream data in order it was received
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this pull request
May 30, 2021
Changes made after forking v1.1.4: v1.1.5 (2019-03-18) * Support Zip64 when compressing iterables and strings (allanlei/python-zipstream#25) v1.1.6 (2019-06-06) * Add partial flushing of ZipStreams (arjan-s/python-zipstream#1) v1.1.7 (2019-10-22) * Stream data in the order it was received (arjan-s/python-zipstream#4) v1.1.8 (2020-09-14) * New datetime parameter in write_iter (arjan-s/python-zipstream#8)
fix readme example
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Zip64 support currently doesn't work when the inputs for the generated zip file are iterables and/or strings. The reason for this is that __write() assumes a file size of 0 bytes for those inputs. Because of the file size the module doesn't enable Zip64 support. When it detects during compression that the file size is larger than ZIP64_LIMIT, it (correctly) raises RuntimeError('File size has increased during compressing').
This patch adds the optional (and thus backwards compatible) argument buffer_size to write_iter and writestr. This allows programs using the module to specify the buffer size that will result from the iterable or string, and in turn that allows __write() to enable Zip64 support when necessary.