Advice for Using Bull for Uploading Files to S3 with Readable Streams

[dangtony98]

For context, I'm working on an app where users can upload audio to a GraphQL (Apollo) backend to be forwarded to S3 for storage; I have a Heroku web and worker dyno and wish to off-load the upload job to the background worker. By design, the upload process gives me a readable stream accessible via file.createReadStream() that I pass through upload parameters to be stored on S3. Given that data sent via bull needs to be serializable (and ideally light-weight), what would be the recommended approach to using bull for my objective here?

A snippet of the upload process:

const uploadParams = {
    Bucket,
    Key: `ABC`,
    ACL: "public-read",
  };

uploadParams.Body = file.createReadStream();

...

try {
   await s3.upload(uploadParams).promise();

[manast]

If I client is uploading a file to your service and then the file is uploaded to S3 then I think the best is just to pipe streams from the original client request to the S3 service, no need to use a queue for this use-case, unless there are other requirements that would make the queue relevant.