How to transmit / upload files over API?

[sschuchlenz]

In my application I need the capacity to accept files via a POST request, my app uses yogiben:autoform-file for file uploads. Is there a way to achieve this?

[kahmali]

There is currently no native way to do this in Restivus, and the truth is, it's because I don't know how. I've looked into it briefly for my own applications, and it just hasn't been an immediate enough need for me to do any proper research. There are still a few basic features that I need to get in there (some fundamental REST stuff and some things for keeping code DRY), and then this will be on the top of my list after that. I'll go ahead and add it to the roadmap.

If anyone knows anything about this please feel free to chime in. Any suggestions or resources on supporting this would go a long way. If you do the research, please post anything you find here, especially if there are any other meteor packages doing this right now. I thought that the [http-methods[(https://github.com/CollectionFS/Meteor-http-methods) package supported it, but I don't see anything in a quick scan of their readme. I actually work with both of the guys that made that package (just a happy coincidence I suppose), so maybe I'll just ask them when I see them later. They know infinitely more about file management in Meteor than I do, considering they worked on all the CollectionFS stuff. Maybe they can point me in the right direction.

[dandv]

+1 for this, and also downloading files via the API - see #32.

[ricardodovalle]

+1

[DavidChouinard]

Would love this. 👍

[cuatromedios]

I have used restivus in combination with FSCollection to accept uploaded files via the API with code like

Api.addRoute('questions/:question/photo', {authRequired: true}, {
        post: function () {
            var uploadedFile = new FS.File(this.request.files.photo.path);
            var insertedFile =  Photos.insert(uploadedFile, function (err, fileObj) {
                if (err) {
                    console.log(err);
                } else {
                    console.log(fileObj.size()+" "+fileObj.name()+" "+fileObj.extension());
                }
            });

            return {
                status: "success",
                data: insertedFile._id
            }
        }
    }
);

where photo in this.request.files.photo.path is the name of the field in the body of the request

[rapito]

@cuatromedios I'm going to try your approach, you should make a gist out of it if it works with the latest version.

[satyavh]

Or, post images as base64 to your Api endpoint, then store as base64 in a mongo collection. For GET, your API can give a base64 back from the mongo collection, which you can then draw on canvas or attach to img src in the client. Mongo has native support for storing images as base64.

[sasikanth513]

I'm using base64 type image and I cannot upload images morethan 750KB size, any ideas?

[isdzulqor]

How to do this in restivus,, I wanna make a server for handling a file upload from my android native java as client. Below is the code that's implemented in node js that i get from tutorial in internet. How to do like this below in meteor js especially with this library restivus
Thank you

var fs = require('fs');

module.exports = function(app) {

app.get('/',function(req,res){
res.end("Node-File-Upload");

});
app.post('/upload', function(req, res) {
console.log(req.files.image.originalFilename);
console.log(req.files.image.path);
fs.readFile(req.files.image.path, function (err, data){
var dirname = "/home/rajamalw/Node/file-upload";
var newPath = dirname + "/uploads/" + req.files.image.originalFilename;
fs.writeFile(newPath, data, function (err) {
if(err){
res.json({'response':"Error"});
}else {
res.json({'response':"Saved"});

}
});
});
});

app.get('/uploads/:file', function (req, res){
file = req.params.file;
var dirname = "/home/rajamalw/Node/file-upload";
var img = fs.readFileSync(dirname + "/uploads/" + file);
res.writeHead(200, {'Content-Type': 'image/jpg' });
res.end(img, 'binary');

});
};

[rapito]

Meanwhile, created this gist with @cuatromedios response: https://gist.github.com/rapito/59d92e4a600c0e87ea48

[ecuanaso]

@cuatromedios @rapito
You said "where photo in this.request.files.photo.path is the name of the field in the body of the request"
what does that mean? I'm new to nodejs terminology.

Here's a gist of what I have so far.
https://gist.github.com/ecuanaso/c05665eb641f65cdf4e44d6a79fc12db

But on my console I see
TypeError: Cannot read property 'file' of undefined

[krishaamer]

+1 would love support for file uploads over the API.

[kahmali]

Hey folks! So sorry to go dark on this for so long. If I ever find the time to work on Restivus, I promise to address this (I'm currently working one job that pays the bills and also trying to simultaneously turn my own startup into something that pays some bills as well). I just don't seem to be able to find the time for Restivus nowadays. This issue has the help wanted label on it for that reason 😄 Restivus is a project I took over from some other Meteor developers, and I'd love to find someone to hold down the fort, at least for a while, to keep development cruising along. Is anyone interested in developing this feature? I would gladly accept a PR for it.

[livimonte]

How do I send the file to the API? I tried with the form and with the Meteor HTTP.post with the header 'content-type': 'multipart/form-data', but did not work. What is the correct way to send the file?

[TristanWiley]

Is there any update to this? FSCollection has been deprecated and now seems no solution.

[andrewash]

I found a solution that works for file upload using an HTTP POST endpoint. I attached the file using form-data (say in Postman, for testing purposes).

Here's the tutorial that I adapted to restivus
https://www.codetutorial.io/meteor-server-file-upload/

I am using FSCollection still, since it was already in my project as a dependency before it was deprecated.

And here's my code

API.addRoute('userrecordings',
{
    authRequired: true,
    roleRequired: ['student', 'teacher']
  },
...
post: function() {
...
      // Busboy processes the form-data file attachment
      // SRC - https://www.codetutorial.io/meteor-server-file-upload/
      const busboy = new Busboy({ headers: this.request.headers });
      const server = this;
      let files = []; // Store files in an array and then pass them to request.
      let audio = {}; // create an audio object
      this.response.write(""); // this prevents Meteor from ending the response before we've async processed the uploaded file in busboy
      busboy.on("file", function (fieldname, file, filename, encoding, mimetype) {
        audio.mimeType = mimetype;
        audio.encoding = encoding;
        audio.filename = filename;

        // buffer the read chunks
        var buffers = [];

        file.on('data', function(data) {
          buffers.push(data);
        });
        file.on('end', function() {
          // concat the chunks
          audio.data = Buffer.concat(buffers);
          // push the image object to the file array
          files.push(audio);
        });
      });

      // busboy.on("field", function(fieldname, value) {
      //   server.request.body[fieldname] = value;
      // });

      busboy.on("finish", Meteor.bindEnvironment(function () {
        // The file has been processed, so save it to the file system
        let newFile = new FS.File();

        newFile.attachData(files[0].data, {type: files[0].mimeType}, function(err){
          // we actually want the filename to have "undefined" at the end // newFile.name(files[0].filename);
          UserRecordings.insert(newFile, function (err, fileObj) {
                // Inserted new doc with ID fileObj._id, and kicked off the data upload using HTTP
                if (err) {
                  console.log(`Error while uploading user recording ${err}`);
                } else {
                  console.log(`Created user recording ${fileObj._id}`);
                }
              }
          );
          server.response.write("success");
          server.done();
        });
      }));
      // Pass request to busboy
      this.request.pipe(busboy);
    }
}

[MartinBucko]

In my case @andrewash example don't work due some weird use of server.done() function. It throw error to that say i need to user this.done() event i used it. How ever it was good start for me to solve problem with upload support.

I do some work around tho solve problem. Restivus use JsonRoutes so i use same package in meteor to inject middleware function (JsonRoutes.Middleware.use) and then i use busboy as presented in @andrewash example. After finish event i add file metedata and content to req.files[] and it is accessible in endpoint request context. Middleware is internal set to react only on PUT/POST method with correct authToken and userId request header values. In finish event i use next() to process endpoint route.

My example: https://gist.github.com/MartinBucko/f97e0649ca92cd266ca8edc2dca6548e

See https://github.com/stubailo/meteor-rest/tree/master/packages/json-routes#adding-middleware how to use JsonRoutes.Middleware

Hope this help someone. If i have more time i will like this add to restivuse in more advanced way.

[delasource]

@MartinBucko your code worked for me. But since i dont use Coffescript, i converted it. (with decaffeinate)

But i had to use Meteor.bindEnvironment(function() { ... }) to insert stuff into global Meteor Collections.

[dberringer]

I couldn't get @andrewash example to work either which is too bad because it would be a really nice way to handle this. In either case a simple solution that worked for me was to use Meteor.wrapAsync like so:

function makeLongRunningCalls(url, callback) {
  HTTP.call('GET', `http://jsonplaceholder.typicode.com/${url}`, {},
    (err, response) => callback(err, response.data)
  );
};

const makeLongRunningCallsSync = Meteor.wrapAsync(makeLongRunningCalls);

Api.addRoute('sample', {}, {
	get: function() {
		let response = makeLongRunningCallsSync();

		return response;
	}
});

[lucnat]

@cuatromedios How do you send photos from the client in order to make this work?

[ghost]

I'm using base64 type image and I cannot upload images morethan 750KB size, any ideas?

Does smb know how to increase the limit of POST body size to enable more than 750KB?
As far as I understand the limits are set in json-routes.js of simple:json-routes package

  WebApp.connectHandlers.use(connect.urlencoded({limit: '50mb'}));
  WebApp.connectHandlers.use(connect.json({limit: '50mb'}));
  WebApp.connectHandlers.use(connect.query()); 

Then I do not understand why request body becomes empty as soon as request body size exceeds ~750KB.

[diavrank]

In my case @andrewash example don't work due some weird use of server.done() function. It throw error to that say i need to user this.done() event i used it. How ever it was good start for me to solve problem with upload support.

I do some work around tho solve problem. Restivus use JsonRoutes so i use same package in meteor to inject middleware function (JsonRoutes.Middleware.use) and then i use busboy as presented in @andrewash example. After finish event i add file metedata and content to req.files[] and it is accessible in endpoint request context. Middleware is internal set to react only on PUT/POST method with correct authToken and userId request header values. In finish event i use next() to process endpoint route.

My example: https://gist.github.com/MartinBucko/f97e0649ca92cd266ca8edc2dca6548e

See https://github.com/stubailo/meteor-rest/tree/master/packages/json-routes#adding-middleware how to use JsonRoutes.Middleware

Hope this help someone. If i have more time i will like this add to restivuse in more advanced way.

Thank you @MartinBucko ,The solution in javascript language is here: (Tested on Meteor 1.10)

upload-middleware.js

import {JsonRoutes} from 'meteor/simple:json-routes';
import Busboy from 'busboy';
import {inspect} from 'util';


// images upload middlware (Restivus handle bad this.done() method, so this is workaround)
JsonRoutes.Middleware.use(function (req, res, next) {

    // upload image only on PUT to /users/:id must be presneted authToken and userId header
    if ((req.method === 'PUT' || req.method === 'POST') &&
        req.headers['content-type'].match(/^multipart\/form\-data/)) {

        // TODO: check authToken

        const busboy = new Busboy({headers: req.headers});

        // files will be avaliable in request context in endpoints
        req.files = [];
        req.data = {};

        busboy.on('file', function (fieldname, file, filename, encoding, mimetype) {

            const uploadedFile = {
                filename,
                mimetype,
                encoding,
                fieldname,
                data: null
            };

            //console.log('busboy have file...', uploadedFile);
            const buffers = [];

            file.on('data', function (data) {
                //console.log('data: ', data.length);
                buffers.push(data);
            });
            file.on('end', function () {
                //console.log('EOF');
                uploadedFile.data = Buffer.concat(buffers);
                req.files.push(uploadedFile);
            });
        });

        busboy.on('field', function (fieldname, val, fieldnameTruncated, valTruncated, encoding, mimetype) {
            //console.log('Field [' + fieldname + ']: value: ' + inspect(val));
            req.data[fieldname] = val;
        });

        busboy.on('finish', function () {
            //console.log('busboy finish');
            next();
        });

        req.pipe(busboy);
        return;
    }
    next();
});

Once the middleware was executed you can get the data in your endpoint:

Endpoints.js

import Api from "./config";
import FileOperations from "../../startup/server/FileOperations";

Api.addRoute('testUploadFile', {
    get: function () {
        let responseMessage = {
            statusCode: 400,
            body: {
                message: "Data is missing"
            },
            headers: {}
        };
        let file = null;
        let filename = this.queryParams.filename;
        if (filename) {
            console.log("Si viene el parametro");
            try {
                file = FileOperations.getFile("testFiles/" + filename);
                responseMessage.statusCode = 200;
                responseMessage.body = file.data;
                responseMessage.headers["Content-disposition"] = "filename=" + filename;
                responseMessage.headers["Content-length"] = file.data.length;
                responseMessage.headers["Content-Type"] = file.meta.mime;
            } catch (error) {
                console.error("Error during get the file: ", error);
                responseMessage.statusCode = 500;
                responseMessage.body = {
                    message: "Error during get the file"
                }
            }
        }
        return responseMessage
    },
    post: function () {
        let responseMessage = {
            statusCode: 400,
            body: {
                message: "Data is missing"
            }
        };
        console.log("User id: ", this.userId);
        console.log("Url params: ", this.urlParams);
        console.log("Query params: ", this.queryParams);
        console.log("Body params: ", this.bodyParams);
        console.log("FormData: ", this.request.data);
        console.log("FormData Files: ", this.request.files);
        if (this.request.files.length > 0) {
            let file = this.request.files[0];
            try {
                FileOperations.saveFile(file.data, file.filename, "testFiles");
                console.log("Finished file saved");
                responseMessage = {
                    statusCode: 201,
                    body: {
                        message: "File saved!"
                    }
                };
            } catch (e) {
                console.error("Error saving file: ", e);
                responseMessage = {
                    statusCode: 500,
                    body: {
                        message: "Error saving file",
                        details: e
                    }
                };
            }
        }
        return responseMessage;
    },
});

You can test the different ways to send data from postman (form-data, x-www-form-urlencoded,raw with application/json header) and it must be work. Also, I added an endpoint to how download files en the readFile method is the following:

FilesOperations.js

getFile(path) {
        const buffer = fs.readFileSync(this.path_upload_files + "/" + path);
        let syncFromFile = Meteor.wrapAsync(detect.fromFile);
        let mime = syncFromFile(this.path_upload_files + "/" + path);
        return {data:buffer, meta:mime};
    }

Note: My config file of Restivus is the following:

config.js

// Global API configuration
import uploadMiddleware from './upload-middleware';
var Api = new Restivus({
    useDefaultAuth: true,
    prettyJson: true
});
export default Api;

[hassansardarbangash]

I have used restivus in combination with FSCollection to accept uploaded files via the API with code like

Api.addRoute('questions/:question/photo', {authRequired: true}, {
        post: function () {
            var uploadedFile = new FS.File(this.request.files.photo.path);
            var insertedFile =  Photos.insert(uploadedFile, function (err, fileObj) {
                if (err) {
                    console.log(err);
                } else {
                    console.log(fileObj.size()+" "+fileObj.name()+" "+fileObj.extension());
                }
            });

            return {
                status: "success",
                data: insertedFile._id
            }
        }
    }
);

where photo in this.request.files.photo.path is the name of the field in the body of the request

I tried this solution but it says: TypeError: Cannot read property 'photo' of undefined