Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

To save stream file directly to GoogleCloudStorage via GoogleCloudFunction #513

Closed
geoseong opened this issue Oct 31, 2017 · 1 comment
Closed

Comments

@geoseong
Copy link

I want to save stream data directly to GoogleCloudStorage via GoogleCloudFunction Node.js.

I tried some ways but works nothing.
Now i'm trying at localhost server first.

so once I tried to test, this error shown :
req.file.cloudStorageError = err;
TypeError: Cannot set property 'cloudStorageError' of undefined

I want to know how this works and get some clue.. plz help me with do this.

client side code

HTML

    <form data-brackets-id='37' action="http://localhost:8989/fileupload" method="post" enctype="multipart/form-data" name="">
        <input data-brackets-id='38' type="file" name="file" id="file" value="">
        <input data-brackets-id='39' type="button" data-inline="true" value="Input" id="filesubmit">
    </form>

Script

    $('form #filesubmit').on('click', (e)=>{
        var uploadfile = $("input[name=file]")[0].files[0];
        var formData = new FormData(); 
        formData.append("myfile", $("input[name=file]")[0].files[0]); 
        
        console.log('uploadfile', uploadfile, uploadfile.type);
       
        var googleCloud = 'https://us-central1-xxxx-xxxxx.cloudfunctions.net/functionNm';
        var localurl = 'http://localhost:8989/fileupload'
        $.ajax({ 
            url: localurl,  // googleCloud
            data: formData, 
            processData: false, 
            contentType: false, 
            type: 'POST', 
            success: function(data){ 
                console.log('response data', data);
            } 
        });

    });

server side code

// start form
var Storage = require('@google-cloud/storage');
var multiparty = require('multiparty');
var fs = require('fs');
var form = new multiparty.Form();

const storage = Storage({
    projectId: [My Project ID]
});
const bucket = storage.bucket('multilang_service');

function getPublicUrl (filename) {
  return `https://storage.googleapis.com/${CLOUD_BUCKET}/${filename}`;
}

// get field name & value
form.on('field',function(name,value){
  console.log('[?] normal field / name = '+name+' , value = '+value);
});

// file upload handling
form.on('part',function(part){
    var filename;
    var size;
    if (part.filename) {
        filename = part.filename;
        size = part.byteCount;
    }else{
        part.resume();
    }   
    
    const gcsname = Date.now() + filename;
    const file = bucket.file(gcsname);
    const stream = file.createWriteStream({
        metadata: {
          contentType: part.headers['content-type']
        }
    });
    stream.on('error', (err) => {
        req.file.cloudStorageError = err;
        next(err);
    });

    stream.on('finish', () => {
        req.file.cloudStorageObject = gcsname;
        file.makePublic().then(() => {
          req.file.cloudStoragePublicUrl = getPublicUrl(gcsname);
          next();
        });
    });
    stream.end(form);
    
    part.on('data',function(chunk){
        console.log('[2] '+filename+' read '+chunk.length + 'bytes');
    });

    part.on('end',function(){
        console.log('[4] '+filename+' Part read complete');
    });
});

// track progress
form.on('progress',function(byteRead,byteExpected){
    console.log('[3] Reading total  '+byteRead+'/'+byteExpected);
});

// all uploads are completed
form.on('close',function(){
    res.send('done for test');
});

form.parse(req);
// end form
@geoseong
Copy link
Author

geoseong commented Nov 3, 2017

I think I solved problem using 'multer.js'.

and
Here's my sample.
and how to use multer between client and server.

var multer  = require('multer');
var storage = multer.memoryStorage()
var upload = multer({ storage: storage })

app.post('/fileupload', upload.single('streamfile'), function (req, res, next) {
//app.post('/fileupload', (req, res) => {
    console.log('fileupload req.file', req.file);
    sendUploadToGCS(req, res, next);
});

function sendUploadToGCS(req, res, next) {

  // start form
    var Storage = require('@google-cloud/storage');
  	var multiparty = require('multiparty');
    var fs = require('fs');
  	var form = new multiparty.Form();
    
    const CLOUD_BUCKET = '[my bucket name]';
    var filename = req.file.originalname;
    
    const storage = Storage({
      projectId: '[my gcp project id]'
    });
    const bucket = storage.bucket('[my bucket name]'');
    function getPublicUrl (filename) {
      return `https://storage.googleapis.com/${CLOUD_BUCKET}/${filename}`;
    }
    const gcsname = Date.now() + filename;
    const file = bucket.file(gcsname);
    const stream = file.createWriteStream({
        metadata: {
          contentType: req.file.mimetype
        }
    });

    console.log('gcsname', gcsname);
    console.log('file', file);
    console.log('stream', stream);

    stream.on('error', (err) => {
        req.file.cloudStorageError = err;
            console.log('req.file.cloudStorageError',err);
        next(err);
    });

    stream.on('finish', () => {
        req.file.cloudStorageObject = gcsname;
        file.makePublic().then(() => {
          req.file.cloudStoragePublicUrl = getPublicUrl(gcsname);
                console.log('req.file.cloudStoragePublicUrl', req.file.cloudStoragePublicUrl);
                console.log('req.file.buffer', req.file);
          next();
        });
    });

    stream.end(req.file.buffer);
    
};

I hope this will be helpful.
Thanks.

@geoseong geoseong closed this as completed Nov 3, 2017
ace-n pushed a commit that referenced this issue Nov 11, 2022
ace-n pushed a commit that referenced this issue Nov 11, 2022
ace-n pushed a commit that referenced this issue Nov 14, 2022
ace-n pushed a commit that referenced this issue Nov 15, 2022
ace-n pushed a commit that referenced this issue Nov 15, 2022
ace-n pushed a commit that referenced this issue Nov 17, 2022
ace-n pushed a commit that referenced this issue Nov 17, 2022
ahrarmonsur pushed a commit that referenced this issue Nov 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant