Skip to content
This repository has been archived by the owner on Aug 30, 2021. It is now read-only.

Save profile images to Amazon S3 #1857

Merged
merged 4 commits into from
Sep 19, 2017
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,15 @@ $ docker run -p 27017:27017 -d --name db mongo
$ docker run -p 3000:3000 --link db:db_1 mean
```

### Amazon S3 configuration

To save the profile images to S3, simply set those environment variables:
UPLOADS_STORAGE: s3
S3_BUCKET: the name of the bucket where the images will be saved
S3_ACCESS_KEY_ID: Your S3 access key
S3_SECRET_ACCESS_KEY: Your S3 access key password


## Getting Started With MEAN.JS
You have your application running, but there is a lot of stuff to understand. We recommend you go over the [Official Documentation](http://meanjs.org/docs.html).
In the docs we'll try to explain both general concepts of MEAN components and give you some guidelines to help you improve your development process. We tried covering as many aspects as possible, and will keep it updated by your request. You can also help us develop and improve the documentation by checking out the *gh-pages* branch of this repository.
Expand Down
9 changes: 9 additions & 0 deletions config/env/default.js
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,16 @@ module.exports = {
illegalUsernames: ['meanjs', 'administrator', 'password', 'admin', 'user',
'unknown', 'anonymous', 'null', 'undefined', 'api'
],
aws: {
s3: {
accessKeyId: process.env.S3_ACCESS_KEY_ID,
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
bucket: process.env.S3_BUCKET
}
},
uploads: {
// Storage can be 'local' or 's3'
storage: process.env.UPLOADS_STORAGE || 'local',
profile: {
image: {
dest: './modules/users/client/img/profile/uploads/',
Expand Down
2 changes: 1 addition & 1 deletion modules/core/client/views/header.client.view.html
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
<ul class="nav navbar-nav navbar-right" ng-show="vm.authentication.user">
<li class="dropdown" uib-dropdown>
<a class="dropdown-toggle user-header-dropdown-toggle" uib-dropdown-toggle role="button">
<img ng-src="/{{vm.authentication.user.profileImageURL}}" alt="{{vm.authentication.user.displayName}}" class="header-profile-image" />
<img ng-src="{{vm.authentication.user.profileImageURL}}" alt="{{vm.authentication.user.displayName}}" class="header-profile-image" />
<span ng-bind="vm.authentication.user.displayName"></span> <b class="caret"></b>
</a>
<ul class="dropdown-menu" role="menu">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<form class="signin form-horizontal">
<fieldset>
<div class="form-group text-center">
<img ngf-src="vm.fileSelected ? picFile : '/' + vm.user.profileImageURL" alt="{{vm.user.displayName}}" class="img-thumbnail user-profile-picture" ngf-drop>
<img ngf-src="vm.fileSelected ? picFile : vm.user.profileImageURL" alt="{{vm.user.displayName}}" class="img-thumbnail user-profile-picture" ngf-drop>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add a config setting to the Shared config that would allow you to check whether the profile should be retrieved using local storage.

Then you wouldn't need remove the "/", thus solving path issues on the backend.

shared: {
    profileImageIsExternal: process.env.UPLOADS_STORAGE ? true : false
    ...
}

</div>
<div ng-show="vm.loading" class="form-group text-center">
<img ng-src="/modules/core/client/img/loaders/loader.gif" height="50" width="50" alt="Loading image...">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ var _ = require('lodash'),
errorHandler = require(path.resolve('./modules/core/server/controllers/errors.server.controller')),
mongoose = require('mongoose'),
multer = require('multer'),
multerS3 = require('multer-s3'),
aws = require('aws-sdk'),
amazonS3URI = require('amazon-s3-uri'),
config = require(path.resolve('./config/config')),
User = mongoose.model('User'),
validator = require('validator');
Expand Down Expand Up @@ -57,10 +60,31 @@ exports.update = function (req, res) {
exports.changeProfilePicture = function (req, res) {
var user = req.user;
var existingImageUrl;
var multerConfig;


if (config.uploads.storage === 's3' && config.aws.s3) {
aws.config.update({
accessKeyId: config.aws.s3.accessKeyId,
secretAccessKey: config.aws.s3.secretAccessKey
});

var s3 = new aws.S3();

multerConfig = {
storage: multerS3({
s3: s3,
bucket: config.aws.s3.bucket,
acl: 'public-read'
})
};
} else {
multerConfig = config.uploads.profile.image;
}

// Filtering to upload only images
var multerConfig = config.uploads.profile.image;
multerConfig.fileFilter = require(path.resolve('./config/lib/multer')).imageFileFilter;

var upload = multer(multerConfig).single('newProfilePicture');

if (user) {
Expand Down Expand Up @@ -95,7 +119,9 @@ exports.changeProfilePicture = function (req, res) {

function updateUser() {
return new Promise(function (resolve, reject) {
user.profileImageURL = config.uploads.profile.image.dest + req.file.filename;
user.profileImageURL = config.uploads.storage === 's3' && config.aws.s3 ?
req.file.location :
'/' + req.file.path;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is preventing the deleteOldFile from finding the existing file every time. I understand why this change was made. Otherwise, the front-end code wouldn't be able to find the location of the image. We might need to find a better way to handle this.

I'm on Windows, so it could be only happening in my environment. @Ghalleb Can you confirm if this is happening in your environment as well (if you're not on Windows, of course :) )

After adding a profile image, my user.profileImageUrl is /modules\users\client\img\profile\uploads\12db59fbdd2193b3b8f99867e1640175.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To fix this issue, you can update this line https://github.com/meanjs/mean/blob/master/modules/users/server/controllers/users/users.profile.server.controller.js#L112 to resolve the path for the unlink method.

fs.unlink(path.resolve('.' + existingImageUrl), function (unlinkError) {

But it seems a little hacky to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand what to do for this one.
On my environment (Ubuntu), everything is working fine.

For all the other changes, I will make them soon.

Thanks for the review.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you verify that the existing file is getting deleted, and you're not getting an error logged to the console?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, I have an error in the server logs:

Removing profile image failed because file did not exist.
info: POST /api/users/picture 200 24.550 ms - 414

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be changed back to what it was before: config.uploads.profile.image.dest + req.file.filename, if we can still use the "/" on the front-end when using local storage.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not fan of the '/' in the front end. I prefer having the final URL in the database. Otherwise if we switch from local to s3, all old URL would be bad formed. And that add more logic to the front where it shouldn't be.

user.save(function (err, theuser) {
if (err) {
reject(err);
Expand All @@ -109,24 +135,54 @@ exports.changeProfilePicture = function (req, res) {
function deleteOldImage() {
return new Promise(function (resolve, reject) {
if (existingImageUrl !== User.schema.path('profileImageURL').defaultValue) {
fs.unlink(existingImageUrl, function (unlinkError) {
if (unlinkError) {
if (config.uploads.storage === 's3' && config.aws.s3) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this check is being used in multiple places, you could move it into a local variable at the top level of this request handler.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something like: var useLocalStorage = config.uploads.storage === 'local';

Then here you can just if (!useLocalStorage)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to let place for another storage provider...
But if it will only be local or S3, I'll do it.

try {
var { region, bucket, key } = amazonS3URI(existingImageUrl);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be troublesome, if switching from "local" to "s3" storage. The first time a user updates their profile picture after switching the storage from one to another, this will create inconsistency with the data/files. This is probably more of an edge case, or migration consideration. Perhaps not a big deal?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about that as well. Normally you have to configure your app once!
An edge case for me.

} catch(err) {
console.warn(`${existingImageUrl} is not a valid S3 uri`);

// If file didn't exist, no need to reject promise
if (unlinkError.code === 'ENOENT') {
console.log('Removing profile image failed because file did not exist.');
return resolve();
}
return resolve();
}

console.error(unlinkError);
aws.config.update({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed? This is already happening up top.

accessKeyId: config.aws.s3.accessKeyId,
secretAccessKey: config.aws.s3.secretAccessKey
});

var s3 = new aws.S3();

var params = {
Bucket: config.aws.s3.bucket,
Key: key
};
s3.deleteObject(params, function (err) {
if (err) {
console.log('Error occurred while deleting old profile picture.');
console.log("Check if you have sufficient permissions : " + err);
}

reject({
message: 'Error occurred while deleting old profile picture'
});
} else {
resolve();
}
});
});
} else {
fs.unlink(path.resolve('.' + existingImageUrl), function (unlinkError) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really a fan of this solution. See my comment about adding a setting to the Shared config. This might not be needed if we can keep the "/" on the front-end when we're using local storage.

if (unlinkError) {

// If file didn't exist, no need to reject promise
if (unlinkError.code === 'ENOENT') {
console.log('Removing profile image failed because file did not exist.');
return resolve();
}

console.error(unlinkError);

reject({
message: 'Error occurred while deleting old profile picture'
});
} else {
resolve();
}
});
}
} else {
resolve();
}
Expand Down
3 changes: 3 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,9 @@
},
"dependencies": {
"acl": "~0.4.10",
"amazon-s3-uri": "0.0.3",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This library is rather new, and doesn't have many stars. However, the implementation looks fine and isn't too complicated. We'll probably have to keep an eye on this library though.

Copy link

@Isabeljason Isabeljason Sep 6, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even we follow the library updates there are few minor changes that need to be update to it. As i am a new bee to MEAN.js need support from others to get into and complicated issues like path, switching, controllers, pre & post updates need to be checked very carefully.
Update me on this if any wrong.

"async": "~2.5.0",
"aws-sdk": "^2.104.0",
"body-parser": "~1.17.1",
"bower": "~1.8.0",
"chalk": "~2.1.0",
Expand All @@ -59,6 +61,7 @@
"mongoose": "~4.11.3",
"morgan": "~1.8.1",
"multer": "~1.3.0",
"multer-s3": "^2.7.0",
"nodemailer": "~4.0.1",
"owasp-password-strength-test": "~1.3.0",
"passport": "~0.3.2",
Expand Down