s3Adapter: Deleting objects and their attached files?

[Copyrightsworld]

Hi all.

I have now configured my Parse server to use an S3 bucket to store files.
While this is working great (on PHP. iOS) create and store an object, when i try to delete the object, object is deleted from Mongo but the file assocciated with it, is still in my S3 bucket.

What is need is, when i delete an object from Mongo, all files attached to it to get deleted too (from s3).

Any ideas how to do that?

thnx

[flovilmart]

@globyworks this is a behaviour that was originally there on parse.com. And no further effort was done to delete the files. I recall there was a button to delete orphelin files, we could as well provide an endpoint for that.

In the meantime, you can query all your objects, collect all the files, then query S3, check which ones are in extra, and delete those ones. That would be the way.

[gfosco]

If you want to delete files when deleting an object, you could use an afterDelete hook. You could either trigger the rest api delete endpoint for the file, or access the file adapter directly to delete the file from s3.

[jiawenzhang]

@gfosco can you provide sample code about how to use the S3Adapter to delete a file in cloud code?

[mnearents]

@globyworks when your files are saved to your s3 bucket, are they also saved in your MongoDB database? Is there a way to replace the file in my MongoDB database with a URL to the file in s3?

[flovilmart]

If the files are saved to S3, they should not /can't be saved in the local database

[mnearents]

Ok, I'll have to look into it more. I'm on Heroku with MongoLab and when I open my MongoLab db, I can still see an entry called [filename].m4a. But I just noticed the file size for the Class doesn't go up as much as the .m4a file size when I save. Thanks.

[Copyrightsworld]

@lususvir using my config, files are uploaded to Parse Server where they get redirected to the S3 bucket. Then the file name is saved on MongoDB and it gets attached to the path to that bucket when needed.
You can also make your clients upload directly to the S3, but that was too much of a hustle at the moment for us.

[wavo89]

for anyone in need of a straightforward explanation -
I am a not exp w the node environment, mongodb, or AWS etc, so I'm deployed via heroku and mLab.
Was confused why I kept using up more precious DB space even when I deleted files, and saw that chunks and files collections were growing in the mLab explorer despite deletions.

connected s3 with Bucketeer add on (a little simpler and less config than otherwise, I am not comfortable enough w/ AWS manipulation). also had to make sure that my parse-server dependency was updated enough to include the s3 router.
then, for deletion- installed AWS SDK on the project
got file name when deleting a the "file" on the DB , then used AWS api code from here:
http://www.tothenew.com/blog/delete-file-from-amazon-s3-using-javascript-sdk/

if you're new to terminal, fyi you can create a JS file on the main dir and then heroku run node filename.js after pushing the project per changes
from there you can test out the code and console.log object names to make sure you're doing everything correctly.

You obv have to be sure that this is the only reference to that file, or that you are deleting those other relevant references when you delete the file from the bucket.
In the end, you will have a functioning means to delete the actual file, and not just the reference.
LMK if any questions, a lot of us from Parse are green w/ this stuff by definition, so sometimes more thorough info is really helpful.

[jenlai1345]

pasting working code example here after spending a day on it.
I migrated parse server to aws + mlab, and hosted my files on s3 (figured mongoLab will probably get very expensive if my files grow to a large number).

I have a class called "classContainingFileAttr" which contains a column "pfFilecolumn" that's of PFFile type. When I delete that class object on Parse server, the file still exists in S3, so I would like to delete that orphaned file in order to save space.

in packages.json, add the following:
"aws-sdk": "latest"

in main.js, add the following:
var AWS = require('aws-sdk');

AWS.config.update({
accessKeyId : 'xxxxx',
secretAccessKey : 'xxxxx'
});
AWS.config.region = 'xxxxxx';

function deleteS3File( fn ) {
var bucketInstance = new AWS.S3();
var params = {
Bucket: 'elasticbeanstalk-xxxxxxxx',
Key: 'images/' + fn // my image files are under /images folder
};
bucketInstance.deleteObject(params, function (err, data) {
if (err) {
console.log("deleteS3File - Check if you have sufficient permissions : "+err);
}
// else {
// console.log("deleteS3File - File deleted successfully = images/" + fn + ", with error: " + err);
// }
});
}

// clean up orphaned file on S3
Parse.Cloud.afterDelete("myClass", function(request) {
deleteS3File( request.object.get("pfFilecolumn").name() );
});

[monajafi]

I've used fs-store-adapter to save files on disk. please provide sample cloud code to delete files from disk after deleting parse object too

[monajafi]

here is the fsadapter cloud code:

var fs = require('fs');
var filePath = '/parsefilespath/';

function deleteFSFile( fn ) {
    fs.unlinkSync(filePath + fn);
}

Parse.Cloud.afterDelete("Test", function(request) {
    deleteFSFile( request.object.get("image").name() );
});

[polo2244]

Hello, based on @jenlai1345 's answer and many others.
I created this class to make it easier to delete files with S3 adapter in the cloud code and implemented it in our server.
I'd love some pros and cons about this or how it could be improved.

DISCLAIMER :
I am not a professional JS developer, so the following code can be obviously edited for more efficiency and ease of use.

QUESTION:
is there any way to acquire the running app cached AWS file adapter ?
Without the need to include credentials in cloud code ?
For example the 'parse-server-mailgun' module can be imported like this :

const { AppCache } = require('parse-server/lib/cache');
const MailgunAdapter = AppCache.get('yourAppId').userController.adapter;

Code :

class FileHandler {

  constructor( request ){
    this.S3Storage = undefined;
    this.request = request;
  };


  /**
   * Call during before save, will scan for dirty PFFIle objects
   * and schedule deletions.
   * NOTE : Make sure this is called just prior to before save response.success,
   * since action cannot be reverted if beforeSave fails, files will be deleted.
   * @return {Promise}
   */
  beforeSave(){
    /* ParseObject is new, nothing to do */
    if ( !this.request.object.id ) return Parse.Promise.as();
    return this._deepStartDestroyingFiles();
  }

  /**
   * Call just after delete of PFObject.
   * All PFFiles will be scheduled for deletion
   * @return {Promise}
   */
  afterDelete(){
    this.request.original = undefined;
    return this._deepStartDestroyingFiles();
  }




  //////////INTENALS///////////



  /**
   * Will load AWS module on demand.
   */
  _loadModule(){
    if ( this.S3Storage === undefined ) {
      var AWS = require('aws-sdk');
      AWS.config.update({
         accessKeyId : '',
         secretAccessKey : ''
      });
      AWS.config.region = '';

      /* current S3Storage instance */
      this.S3Storage = new AWS.S3();
      this.bucketname = '';
    }
  }

  /**
   * Will generate an array of files to be destroyed
   * depending on which files have been modified or deleted
   * @return {Array:ParseFile}
   */
  _filesToDestroy(){
    var files = [];
    var recentObject = this.request.object;
    var originalObject = this.request.original;

    /* we are is after delete - we should delete all files */
    if ( !originalObject ) {
      var json = recentObject.toJSON();
      for (var key in json) {
        if ( recentObject.has(key) === false ) continue;
        if ( recentObject.get(key).constructor.name == 'ParseFile' ) {
          files.push( recentObject.get(key) );
        }
      }
    }else{

      /* we are in beforeSave - we should delete replaced files only */
      //use this for (var i = 0, len = arr.length; i < len; i++)  to save milliseconds
      var dirtyKeys = recentObject.dirtyKeys();
      for (var i = 0; i < dirtyKeys.length; i++) {
        var key = dirtyKeys[i];

        /* previously there was no value - no file */
        if ( originalObject.has(key) === false ) continue;

        /* previously there was a value */
        if ( originalObject.get(key).constructor.name == 'ParseFile' ) {
          /* this file was repaced, so let's delete it */
          files.push( originalObject.get(key) );
        }
      }
    }

    return files;
  }

  /**
   * Prepare files to be deleted and delete them
   * @return {Promise} [description]
   */
  _deepStartDestroyingFiles(){
    var files = this._filesToDestroy();
    // for (var i = 0; i < files.length; i++) {
    //   console.log('Destroying file ' + files[i].name());
    // }
    if ( files.length == 0 ) return Parse.Promise.as();

    return this._deepDestroyFiles( files );
  }

  /**
   * Tell S3 which files to delete
   * @param  {[type]} files [description]
   * @return {[type]}       [description]
   */
  _deepDestroyFiles( files ){
    this._loadModule();

    var promise = new Parse.Promise();
    this.S3Storage.deleteObjects( this._paramsForFiles( files ) , function(error, data) {
      if (error) {
        promise.reject( error );
      }else{
        promise.resolve( data );
      }
    });

    return promise.then(function ( data ) {
      //console.log('deleted ' + data);
      return Parse.Promise.as();
    }, function (error) {
      //console.log("Failed to delete files : " + error + ' - ' + error.stack);
      //self.scheduleFutureJobDeletion( files );
      return Parse.Promise.error( error );
    });
  }

  /**
   * Returns S3 request parameters for array of PFFiles
   * @param  {Array:ParseFile} pf_files The PFFiles array to be deleted
   * 
   * This function can be removed and directly generate parameters while we are in 
   * `_filesToDestroy()`
   * 
   * @return {object}
   */
  _paramsForFiles( pf_files ){
    var objects = [];
    //use this for (var i = 0, len = arr.length; i < len; i++)  to save milliseconds
    for (var i = 0, i < pf_files.length; i++) {
      var name = pf_files[i].name();
      objects.push( { Key : name } );
    }
    return {  Bucket: this.bucketname, Delete : { Objects: objects } };
  }

}
module.exports = FileHandler;


/* usage */

var fileHandler = require("./fileHandler.js");

Parse.Cloud.beforeSave( 'media' , function(request, response) {
  if ( something ) {
    /* other code */
    response.error('cant update object');
  }else{
    new fileHandler( request ).beforeSave();
    response.success();
  }
});

Parse.Cloud.afterDelete( 'media' , function(request, response) {
  new fileHandler( request ).afterDelete();
});

[ogtfaber]

Thanks @jenlai1345

Here is the code for Gcloud:

// Clean up orphan file after delete
Parse.Cloud.afterDelete("File", (request) => {
  const name = request.object.get('parseFile').name()
  return deleteGCFile(name)
})

function deleteGCFile(name) {
  const storageProjectID = "..."
  const storageKeyfilePath = "..."
  const storageBucket = {
    development: "...",
    production: "...",
  }[app.get('env')]

  const storage = GCStorage({
    projectId: storageProjectID,
    keyFilename: storageKeyfilePath,
  })

  return storage
    .bucket(storageBucket)
    .file(name)
    .delete()
}