Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZLIB failing to inflate entire .gz file that contains an extra field in the header #6116

Closed
chmille4 opened this issue Apr 8, 2016 · 7 comments
Labels
zlib Issues and PRs related to the zlib subsystem.

Comments

@chmille4
Copy link

chmille4 commented Apr 8, 2016

  • 4.4.2:
  • OSX:
  • 10.10.5:

The Node ZLIB library only inflates the first 64k of a gzipped file (attached). The file has an extra field in the header, which may be causing the problem. GUNZIP on mac and linux correctly inflate it to it's full size (~33M). I've also attached a simple test script which takes the gzip file on stdin and outputs the inflated file to stdout. If you run the test script with the attached gzip file you should get 33M but you only get 64K of data.
zlib_test.js.zip

IDb.part1.gz

Note: the attached gzip file has been sliced to make it smaller so that I could upload it here. Therefore gunzip will complain and say "unexpected end of input", however this file is displaying the same behavior as the original larger gzip file.

@yorkie
Copy link
Contributor

yorkie commented Apr 8, 2016

@chmille4 Do you have tried with file stream though?

@chmille4
Copy link
Author

chmille4 commented Apr 8, 2016

@yorkie I don't quite understand. Can you give me an example of what you mean? Thanks

@yorkie
Copy link
Contributor

yorkie commented Apr 8, 2016

I mean you should select single variable when evaluating a result, you are using stdin/stdout to take inputs and to make sure what's the result, but it might not be working with Node.js, so could you debug it by the following steps:

  • read the gzip file by fs.readFile and make sure the contents length is equal to the expect size.
  • inflate the above buffer and put the result to a buffer object.
  • check that buffer's length if is also expected

If the above steps are working correctly, this issue might not be caused by zlib.

@mscdex mscdex added the zlib Issues and PRs related to the zlib subsystem. label Apr 8, 2016
@mscdex
Copy link
Contributor

mscdex commented Apr 8, 2016

Here's the script:

var zlib = require('zlib');

process.stdin.pipe(zlib.createGunzip()).pipe(process.stdout);

@mscdex
Copy link
Contributor

mscdex commented Apr 8, 2016

@chmille4 Can you try with node v5.10.1 and the master branch if that doesn't work? I know there were some zlib-related changes recently, some of which are in v5.x and some that are not yet.

@addaleax
Copy link
Member

addaleax commented Apr 8, 2016

I’m 99 % sure this is a duplicate of #4306, which has been fixed in #5120. The script works as expected with Node.js v5.9.0+.

@chmille4
Copy link
Author

chmille4 commented Apr 8, 2016

Yep, upgrading to v5.10.1. Fixed the issue. Awesome, thanks for the help!

@chmille4 chmille4 closed this as completed Apr 8, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
zlib Issues and PRs related to the zlib subsystem.
Projects
None yet
Development

No branches or pull requests

4 participants