Zip creation of large folder on S3 using aws-s3-zipper and Node js

[nitin89]

I'm using the aws-s3-zipper module Link - https://www.npmjs.com/package/aws-s3-zipper

I'm zipping large folder(including images of size 20 GB) on S3

var S3Zipper = require('aws-s3-zipper');
config = {} // AWS config details
var zipper = new S3Zipper(config);

zipper.zipToS3FileFragments({
s3FolderName: foldername
, s3ZipFileName: foldername.zip
//, maxFileCount: 50
, maxFileSize: 1024 1024 1024 // 1 GB for each fragments
, recursive: false
}, function (err, results) {
// code on file zipped successfully
}

Problem is s3-zipper creating zip folders which includes already zipped folder.

For Example:
Suppose my bucket folder contains 5000 images of size 20GB,
Now i have provide 1GB size for each fragments in s3-zipper
So as per this i'm expecting 20 zips of 1 GB each (see above code sample)

But it creating around 40 or more zips of size around 1GB

I am new to AWS please help me, Thanks

[DanielHindi]

Have you tried to filter them out? ref: https://github.com/DanielHindi/aws-s3-zipper#filter-out-files
in which case you can filter out all previous zip files ie *.zip

[sachgit30]

Hello,

I'm using the aws-s3-zipper from Link - https://www.npmjs.com/package/aws-s3-zipper.

My requirement is to create one single zip file for hierarchy of folders and files available in S3 bucket. Is it possible to achieve this using Lambada? The file sizes could easily exceed Lambda storage limit of 512 MB.
Is this possible in Lambda?
Thanks in advance for help

[segmoti]

I had a few questions and clarifications about this script;
I was wondering if anyone had tested say creating 10GB zip files using this script?
It seems it is using listobject which lists object in utf-8 binary, so we can start listing of files from a particular location (this is for function where we are creating fragments of zip file)?
Is it deleting the temporary files immediately after zipping?
Anyone tried password protecting zip?

Thanks
Segi