Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

decompress applies filter after fully reading all files into memory, and documentation is misleading #56

Open
glasser opened this issue Jul 26, 2017 · 0 comments

Comments

@glasser
Copy link
Contributor

glasser commented Jul 26, 2017

The documentation of the filter option is misleading:

Filter out files before extracting

That's not really true. When you use decompress (say, with decompress-tar or decompress-unzip), it fully decompresses the entire tarball into memory before applying the filter. If you're not using the output feature, the filter option is entirely equivalent to just using Array.filter yourself on the return value.

It would make a lot more sense to me if decompress plugins ran filter as soon as they started processing a file, and dropped data from filtered-out files instead of filling RAM with garbage.

I recognize that this is a backwards-compatible change (as currently filter callbacks have access to data) so probably this means that a new "pre-filter" option should be added.

#57 is a PR to add documentation, though I'd suggest fixing the issue instead of documenting it.

glasser added a commit to glasser/decompress that referenced this issue Jul 26, 2017
Improves kevva#56, although fixing the issue would be preferable to documenting it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant