Skip to content
This repository has been archived by the owner on Sep 13, 2023. It is now read-only.

Don't support reading "snappy framed" format - bug or feature? #176

Open
grimen opened this issue Jul 15, 2019 · 2 comments
Open

Don't support reading "snappy framed" format - bug or feature? #176

grimen opened this issue Jul 15, 2019 · 2 comments

Comments

@grimen
Copy link

grimen commented Jul 15, 2019

Motivation: I am trying to use snappy and snappy-stream along each other they write different output because of the framed format, I would assume snappy library should handle the framed format case ideally? At minimum would make sense to detect it and raise a more developer friendly error message than "Invalid input"?

const fs = require('fs')
const snappy = require('snappy')
const snappyStream = require('snappy-stream')

// ==================================
//      SNAPPY
// ------------------------------

const inData = fs.readFileSync('data.txt')
const snappyCompressData = snappy.compressSync(inData)

console.log('snappy.compressSync DATA', snappyCompressData)


// ==================================
//      SNAPPY-STREAM
// ------------------------------

const snappyCompressStream = snappyStream.createCompressStream()

const inStream = fs.createReadStream('./data.txt')
const outStream = fs.createWriteStream('./compressed.sz')

// NOTE: "snappy" and "snappy framed" are not identical: https://github.com/google/snappy/blob/master/framing_format.txt

inStream
    .pipe(snappyCompressStream)
    .on('data', (chunk) => {
        this.buffers = this.buffers || []
        this.buffers.push(chunk)
    })
    .on('end', () => {
        const data = Buffer.concat(this.buffers)

        console.log('snappyCompressStream DATA', data, snappy.uncompressSync(data))
        // console.log('snappyCompressStream DATA', data)
    })
    .pipe(outStream)

Output:

➜  node-snappy-stream-test node compress.js
snappy.compressSync DATA <Buffer 13 48 7b 0a 20 20 20 20 22 66 6f 6f 22 3a 20 62 61 72 0a 7d 0a>
/home/jonas/Desktop/node-snappy-stream-test/node_modules/snappy/snappy.js:49
  return binding.uncompressSync(compressed, uncompressOpts(opts));
                 ^

Error: Invalid input
    at Object.exports.uncompressSync (/home/jonas/Desktop/node-snappy-stream-test/node_modules/snappy/snappy.js:49:18)
    at CompressStream.<anonymous> (/home/jonas/Desktop/node-snappy-stream-test/compress.js:36:63)
    at CompressStream.emit (events.js:208:15)
    at endReadableNT (_stream_readable.js:1154:12)
    at processTicksAndRejections (internal/process/task_queues.js:77:11)
@grimen grimen changed the title Bug: Don't support reading "snappy framed" format Don't support reading "snappy framed" format - bug or feature? Jul 15, 2019
@grimen
Copy link
Author

grimen commented Jul 15, 2019

Implementation of the framing format is optional for Snappy compressors and
decompressor; it is not part of the Snappy core specification.

From: https://github.com/google/snappy/blob/master/framing_format.txt

Well, the Google do point out that it is not part of core specification but rather an optional - but yea, would be neat if it was handled. Not sure otherwise how to ensure data integrity. 🤔

@kesla
Copy link
Owner

kesla commented Apr 23, 2020

hey,

Sorry for late response.

yeah I think you're right. I think that the best way forward here will be to support the framing format in snappy, so merge in https://github.com/kesla/node-snappy-stream.

I will then change it so that .createUncompressStream() support output from "normal" snappy and .uncompress() can handle data in the framing format.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants