How to use zlib to compress multiple files?

[ghiscoding]

I'm trying to replace code that I have working for a while that was using node-archiver

when I try fflate with the zip, the output size is larger (611Kb) than the output of archiver (591Kb), the one thing different is that archiver seems to be using zlib which is maybe why it's smaller?

const archive = archiver('zip', {
  zlib: { level: 9 } // Sets the compression level.
});

So I tried to use zlib from fflate but I don't understand how to use it. Is it supposed to produce a .zip file or another extension? Also what I'm wondering is, does archiver actually compress each file 1 by 1 with zlib and then creates a zip? or can we create the same folder structure with zlib?

So far this is what I have, it kinda works with gzip but I'd like to use zlib if possible and if I just replace the gzip with zlib then it says the output file is not a valid zip when I try to open it.

const myFiles = [
  { name: 'package.json', path: './package.json' },
  { name: 'my-lib.js', path: './dist/my-lib.js' }.
];
let fileLeft = myFiles.length;
let filesLengths = {};

const fileToU8 = (file, cb) => {
  cb(strToU8(fs.readFileSync(file.path)));
};
let processFile = (i) => {
  let file = myFiles[i];
  fileToU8(file, (buf) => {
    zlib(buf, {
      level: 6,
      // The following are optional, but fflate supports metadata if you want
      mtime: file.lastModified,
      filename: file.name

    }, (err, data) => {
      if (err) { console.log('error', err); }
      else {
        filesLengths[file.name] = [data.length, file.size];

        // If everyone else has finished processing already...
        if (!--fileLeft) {
          // write to file
          fs.writeFileSync(`my-file.zip`, data);
        }
      }
    });
  });
};

for (let i = 0; i < myFiles.length; ++i) {
  processFile(i);
}

I'd really like to use fflate in my project since it's fast and it's ESM as opposed to node-archiver which is not, but I want to have the same output file size. I don't mind compressing twice if that is what I need to do but I just can't find how to do all of that and I'm confused with the docs

below is an example of the file compression difference with node-archiver and fflate zip with level 9 for both (I'm just showing 1 file below but all files in the styles folder are also have similar differences in compression ratio)

node-archiver

fflate with zip

[101arrowz]

GZIP, Zlib, and DEFLATE are all the same compression format. Files compressed into each format should always be within 20 bytes of each other in size, all else being equal; the only differences are the headers.

The reason fflate is performing worse here is that it uses its own DEFLATE implementation, while node-archiver uses the one built into Node.js. This is important for supporting web browsers, but here it's clearly a disadvantage.

If you're sure you're using level 9 compression for both node-archiver and fflate, then there's not much more you can do. The difference in compression ratio of 1% is within the range of variance I'd expect between fflate's DEFLATE implementation and the Node.js one. You'll either have to live with the worse performance or continue using node-archiver. I will say that for other files you'll find fflate actually compresses much better than node-archiver; it all depends on the data.

[ghiscoding]

GZIP, Zlib, and DEFLATE are all the same compression format. Files compressed into each format should always be within 20 bytes of each other in size, all else being equal; the only differences are the headers.

ok so just to be clear, you're pretty much saying that the compression rate between all 3 is roughly going to be the same at the end of it? However, I'm trying to create a .zip from multiple files by using the zlib compression (which is what I'm able to do with node-archiver).

All of these things are a little confusing to me, so I'm a little confused with all of these. First question, can we create a .zip with the zlib compression and if so, then how? The only example I could find with multiple file is by using the zip, is that the only one providing this functionality (multiple files which keeps the folder structure)?

[101arrowz]

the compression rate between all 3 is roughly going to be the same

Yes, within a single implementation. Different implementations are free to compress them differently, however.

I'm trying to create a .zip from multiple files by using the zlib compression

This is technically a misnomer - "zlib" compression typically refers to the Zlib format (RFC 1950), but ZIP files specifically support the DEFLATE format (RFC 1951) and not Zlib. By "zlib" compression node-archiver probably means DEFLATE. fflate also supports DEFLATE compression in zip files; in other words, by using fflate you are automatically using what node-archiver calls "zlib compression." I understand that all this terminology can be confusing; please let me know if you want further clarification.

[ghiscoding]

By "zlib" compression node-archiver probably means DEFLATE. fflate also supports DEFLATE compression in zip files; in other words, by using fflate you are automatically using what node-archiver calls "zlib compression."

if I use zip from fflate, does that mean it's equivalent to node-archiver? or do I need to use deflate from fflate?

I'm asking all these questions because I'm just trying to see if there's any way to get to the same file size that node-archiver produces while still outputting a .zip file (a .gzip won't work for my use case, it really has to be a .zip file)

Nonetheless, thanks for all these info, even with the slightly larger file size, I still think that I'll migrate to fflate because your project is pretty much the only project supporting Tree Shaking via ESM and your fflate is the fastest too so... 🚀

Cheers

[101arrowz]

if I use zip from fflate, does that mean it's equivalent to node-archiver

Yes, pretty much. zip and zipSync are the best ways to compress zip files with fflate and use the same compression algorithm as node-archiver. I'm glad you've found the library useful!

[ghiscoding]

Great, thanks for everything. I think I have all the info I needed. :)