Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multi member / bgzip compatibility #145

Closed
wants to merge 8 commits into from
Closed

multi member / bgzip compatibility #145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
15e220c
Fix inflate multistream content
Jun 3, 2018
08942f1
re-enable skipped test
rbuels Aug 31, 2018
494d03b
improve SYNC mark test and make it pass
rbuels Sep 1, 2018
3cd0bb6
add some failing tests for inflating bgzipped files
rbuels Aug 30, 2018
d3f65ab
make inflateReset discard the current inflate state
rbuels Sep 1, 2018
26d1459
improve bgzip tests
rbuels Sep 1, 2018
ee4ab1b
revert style change
rbuels Sep 1, 2018
d45ebf9
remove pre-decompressed fixture and unzip with node zlib instead to c…
rbuels Sep 1, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions lib/inflate.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,10 @@ Inflate.prototype.push = function (data, mode) {

status = zlib_inflate.inflate(strm, c.Z_NO_FLUSH); /* no bad return value */

if (status === c.Z_STREAM_END) {
zlib_inflate.inflateReset(strm);
}

if (status === c.Z_NEED_DICT && dictionary) {
// Convert data if needed
if (typeof dictionary === 'string') {
Expand All @@ -237,7 +241,7 @@ Inflate.prototype.push = function (data, mode) {
}

if (strm.next_out) {
if (strm.avail_out === 0 || status === c.Z_STREAM_END || (strm.avail_in === 0 && (_mode === c.Z_FINISH || _mode === c.Z_SYNC_FLUSH))) {
if (strm.avail_out === 0 || (status === c.Z_STREAM_END && strm.avail_in === 0) || (strm.avail_in === 0 && (_mode === c.Z_FINISH || _mode === c.Z_SYNC_FLUSH))) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, now understand. I don't remember all black magick in this conditions, and how should it work. Probably, "make cover" should generate report and show if this condition is covered or not.

Did some test failed without this?


if (this.options.to === 'string') {

Expand Down Expand Up @@ -270,7 +274,7 @@ Inflate.prototype.push = function (data, mode) {
allowBufError = true;
}

} while ((strm.avail_in > 0 || strm.avail_out === 0) && status !== c.Z_STREAM_END);
} while (strm.avail_in > 0 || strm.avail_out === 0);

if (status === c.Z_STREAM_END) {
_mode = c.Z_FINISH;
Expand Down
2 changes: 1 addition & 1 deletion lib/zlib/inflate.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ function inflateReset(strm) {
var state;

if (!strm || !strm.state) { return Z_STREAM_ERROR; }
state = strm.state;
state = new InflateState();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess, that's wrong. everything in /zlib folder is ported from upstream, and does exactly what it should. Problems on upper wrapped should not affect this code.

state.wsize = 0;
state.whave = 0;
state.wnext = 0;
Expand Down
Binary file added test/fixtures/bgzip-1.txt.gz
View file
Open in desktop
Binary file not shown.
Binary file added test/fixtures/bgzip-2.txt.gz
View file
Open in desktop
Binary file not shown.
19 changes: 19 additions & 0 deletions test/fixtures/gzip-joined
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
0
rbuels marked this conversation as resolved.
Show resolved Hide resolved
1
2
3
4
5
6
7
8
910
11
12
13
14
15
16
17
18
19
27 changes: 23 additions & 4 deletions test/gzip_specials.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
var fs = require('fs');
var path = require('path');
var assert = require('assert');
var zlib = require('zlib');

var pako_utils = require('../lib/utils/common');
var pako = require('../index');
Expand Down Expand Up @@ -62,12 +63,15 @@ describe('Gzip special cases', function () {

it('Read stream with SYNC marks', function () {
var inflator, strm, _in, len, pos = 0, i = 0;
var data = fs.readFileSync(path.join(__dirname, 'fixtures/gzip-joined.gz'));
var inputData = fs.readFileSync(path.join(__dirname, 'fixtures/gzip-joined.gz'));
var expectedData = fs.readFileSync(path.join(__dirname, 'fixtures/gzip-joined'));
var expectedDataArray = new pako_utils.Buf8(expectedData.length);
pako_utils.arraySet(expectedDataArray, expectedData, 0, expectedData.length, 0);

do {
len = data.length - pos;
len = inputData.length - pos;
_in = new pako_utils.Buf8(len);
pako_utils.arraySet(_in, data, pos, len, 0);
pako_utils.arraySet(_in, inputData, pos, len, 0);

inflator = new pako.Inflate();
strm = inflator.strm;
Expand All @@ -79,7 +83,22 @@ describe('Gzip special cases', function () {
i++;
} while (strm.avail_in);

assert(i === 2, 'invalid blobs count');
assert.equal(i, 1, 'should take it all in one blob');
assert.deepEqual(inflator.result, expectedDataArray, 'inflator produced expected data');
});

it('Read bgzipped file 1', function () {
var inputData = fs.readFileSync(path.join(__dirname, 'fixtures/bgzip-1.txt.gz'));
var expectedData = zlib.gunzipSync(inputData, { finishFlush: (zlib.constants || zlib).Z_SYNC_FLUSH });
var result = pako.inflate(inputData);

assert.deepEqual(result, expectedData, 'must get the exact right inflated result');
});

it('Read bgzipped file 2', function () {
var data = fs.readFileSync(path.join(__dirname, 'fixtures/bgzip-2.txt.gz'));

var all = pako.inflate(data);
assert.equal(all.length, 1922918, 'decompressed full data');
});
});