Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Form request #28

Merged
merged 12 commits into from
Sep 25, 2016
193 changes: 189 additions & 4 deletions examples/direct-upload/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,22 @@
## Direct Upload Example
# Direct Upload to S3 - A Complete Guide

We are going to implement a simple solution for uploading images to an S3 bucket
via a POST request from the browser.

![upload example](https://cloud.githubusercontent.com/assets/12450298/18589369/593617dc-7c22-11e6-899d-00ffdc15ac73.png)

### Contents
- [Creating an S3 Bucket](#step-1---creating-the-bucket)
- [Creating IAM User with S3 Permissions](#step-2---creating-an-iam-user-with-s3-permissions)
- [Generate a Signed S3 Policy](#step-3---generate-a-signed-s3-policy)
- [Create a Server](#step-4---create-a-server-to-facilitate-the-credential-creation)
- [Server and S3 Requests](#step-5---write-the-client-side-code-to-send-our-requests-to-the-backend-and-then-to-s3)
- [Take it for a Spin](#take-it-for-a-spin)
- [Learning Resources](#learning-resources)

### Step 1 - Creating the bucket

+ Create an S3 bucket on [Amazon Web Services](aws.amazon.co.uk). To do so you'll need to
create an account if you haven't got one already.
+ Create an S3 bucket on [Amazon Web Services](aws.amazon.co.uk). To do so you'll need to create an account if you haven't got one already.

![sign up](https://cloud.githubusercontent.com/assets/12450298/18392395/86991fb8-76a9-11e6-83d8-f16d7751b41d.png)

Expand Down Expand Up @@ -438,6 +448,181 @@ server.start((err) => {
}
console.log(`✅ Server running at: ${server.info.uri}`)
```

#### We now have a server that can our index.html can communicate with!

### Step 5 - Write the client side code to send our requests to the backend and to S3
### Step 5 - Write the client side code to send our requests to the backend and then to S3

+ Create a `public` directory in the root of your project. Inside this new folder
create two new files `index.html` and `client.js`

Insert the following into your `index.html`:

```html
<!DOCTYPE html>
<html>
<head>
<title>S3 Upload Demo</title>
// optional stylesheet
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/flat-ui/2.2.2/css/flat-ui.min.css">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We really like the end result of Flat-UI. and for this demo it is good.
But that reminds me ... dwyl/learn-tachyons#1 😉

// include your client.js file that we are going to write
<script type="text/javascript" src="client.js"></script>
</head>
<body>
<h1>S3 Direct Upload Demo</h1>
// simple form with a file input (we'll be adding more later dynamically)
<form>
// onchange gets fired when a file is selected so we want to save the file
<input id="fileInput" type="file" name="file" onchange="uploadDemo.saveFile(this.files)"/>
</form>
// button that will submit our file
<button onclick="uploadDemo.submitFile()">Submit</button>
// container for the success message and link to our image
<div class="successMessageContainer">
<a class="imageLink"></a>
</div>
</body>
</html>
```
In your `client.js` file add the following:

```js
// wrap everything in an IIFE (immediately invoked function expression) to contain
// global variables
var uploadDemo = (function () {
// 'global' variable used to store our filename
var filename

/**
* Saves the filename to our global variable when a file has been selected
* @param {Object} file - file from our file input tag
**/
function saveFile (file) {
filename = file[0].name
}

/**
* Calls our getCredentialsFromServer function with the global filename
**/
function submitFile () {
getCredentialsFromServer(filename)
}

// function that retrieves our S3 credentials from our server
/**
* Saves the filename to our global variable when a file has been selected
* @param {string} filename - name of the file we want to upload
**/
function getCredentialsFromServer (filename) {
var xhttp = new XMLHttpRequest()
xhttp.onreadystatechange = function () {
if (this.readyState === 4 && this.status === 200) {
// after we've received a response we want to assign it to a variable
var s3Data = JSON.parse(xhttp.responseText)
// call our buildAndSubmitForm function
buildAndSubmitForm(s3Data)
// return a success message after the image has been uploaded along with
// link to image
var successMessage = document.createElement('h4')
successMessage.innerHTML = 'Image Successfully Uploaded at: '
var link = `https://<your_bucket_name>.s3.amazonaws.com/${filename}`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could the AWS_S3_BUCKET_NAME be an environment variable returned by the server in the getCredentialsFromServer call?

var imageATag = document.querySelector('a')
imageATag.setAttribute('href', link)
var imageLink = document.createElement('h4')
imageLink.innerHTML = link
var div = document.querySelector('div')
div.insertBefore(successMessage, div.firstChild)
imageATag.appendChild(imageLink)
}
}
// open the GET request to our endpoint with our filename attached
xhttp.open('GET', `/s3_credentials?filename=${filename}`, true)
// send the GET request
xhttp.send()
}

/**
* Dynamically creates and submits our form to S3
* @param {Object} s3Data - endpoint_url and params sent back from our server
**/
function buildAndSubmitForm (s3Data) {
// access the form in our index.html
var form = document.querySelector('form')
// create a new input element
var keyInput = document.createElement('input')
// set its type attribute to hidden
keyInput.setAttribute('type', 'hidden')
// set its name attribute to key
keyInput.setAttribute('name', 'key')
// set its value attribute to our filename
keyInput.setAttribute('value', `${filename}`)
// set the method of the form to POST
form.setAttribute('method', 'post')
// set the action attribute to be our endpoint_url from our server
form.setAttribute('action', s3Data.endpoint_url)
// set the encoding type to multipart/form-data
form.setAttribute('enctype', 'multipart/form-data')
// our file input **must** be the last input in the form, therefore we need
// to insert our keyInput before the first child of the form otherwise it will
// throw an error
form.insertBefore(keyInput, form.firstChild)
// set the form url to be our endpoint_url from our server
form.url = s3Data.endpoint_url
// set the form data to be our S3 params from our server
form.formData = s3Data.params
// submit the form
form.submit()
}
// return functions from our IIFE that we'll need to expose to our index.html
return {
saveFile,
submitFile
}
}())
```
#### We've now written the neccessary code needed to upload directly to S3!

### Take it for a spin!

+ In your terminal run the following command to start the server:
`$ node lib/index.js`

+ Navigate to localhost:8000. You should see the following screen. Click on **Choose File**:

![demo](https://cloud.githubusercontent.com/assets/12450298/18581819/27ac5dac-7bfa-11e6-987f-8aef76c7243c.png)

+ Choose the file you wish to upload and click **open**:

![choose file](https://cloud.githubusercontent.com/assets/12450298/18582392/afcb9598-7bfc-11e6-940d-9f1c85f44c67.png)

+ Then click the **Submit** button

![submit](https://cloud.githubusercontent.com/assets/12450298/18582412/d15503ca-7bfc-11e6-8bd8-52548bf29e2e.png)

+ You should then see the success message with the link to where the image is being hosted (*this can now be used as an img src*). Click on the link:

![success](https://cloud.githubusercontent.com/assets/12450298/18582441/fc24f646-7bfc-11e6-89e8-5dcdaa49ffef.png)

+ This should start an automatic download. Click on the download:

![image download](https://cloud.githubusercontent.com/assets/12450298/18582455/1f080edc-7bfd-11e6-8280-881ef2462e92.png)

+ You should see your image!
![one does not simply](https://cloud.githubusercontent.com/assets/12450298/18582475/4d75e28a-7bfd-11e6-83a0-2c7029cb9830.png)

+ Let's check our S3 console so that we know *for sure* that our image is there. Navigate back to your S3 buckets and click on the one you uploaded to:
![s3 buckets](https://cloud.githubusercontent.com/assets/12450298/18583007/098154ee-7c00-11e6-931e-4dec223cf95d.png)

+ You should be able to see the image that you just uploaded:
![image uploaded](https://cloud.githubusercontent.com/assets/12450298/18583075/5132dc9a-7c00-11e6-91f4-8e1103d3e49a.png)


# 🎉 We've successfully uploaded an image directly to S3!

### Learning Resources

- Amazon documentation - [Browser-Based Upload using HTTP POST (Using AWS Signature Version 4)](http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html)
- Leonid Shevtsov - [Demystifying direct uploads from the browser to Amazon S3](https://leonid.shevtsov.me/post/demystifying-s3-browser-upload/)
- Stackoverflow Q - [Amazon S3 POST api, and signing a policy with NodeJS](http://stackoverflow.com/questions/18476217/amazon-s3-post-api-and-signing-a-policy-with-nodejs)
- AWS Articles - [Browser Uploads to S3 using HTML POST Forms
](https://aws.amazon.com/articles/1434)
49 changes: 25 additions & 24 deletions examples/direct-upload/generate-credentials.js
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
var crypto = require('crypto');
var crypto = require('crypto')

function getS3Credentials(config, filename) {
function getS3Credentials (config, filename) {
return {
endpoint_url: "https://" + config.bucket + ".s3.amazonaws.com",
endpoint_url: 'https://' + config.bucket + '.s3.amazonaws.com',
params: buildS3Params(config, filename)
}
}

function buildS3Params(config, filename) {
var credential = formatAmzCredential(config);
var policy = buildS3UploadPolicy(config, filename, credential);
var policyBase64 = new Buffer(JSON.stringify(policy)).toString('base64');
function buildS3Params (config, filename) {
var credential = formatAmzCredential(config)
var policy = buildS3UploadPolicy(config, filename, credential)
var policyBase64 = new Buffer(JSON.stringify(policy)).toString('base64')
return {
key: filename,
acl: 'public-read',
Expand All @@ -23,43 +23,44 @@ function buildS3Params(config, filename) {
}
}

function formatAmzCredential(config) {
function formatAmzCredential (config) {
return [config.accessKey, dateString(), config.region, 's3/aws4_request'].join('/')
}

function buildS3UploadPolicy(config, filename, credential) {
function buildS3UploadPolicy (config, filename, credential) {
return {
expiration: new Date((new Date).getTime() + (3 * 60 * 1000)).toISOString(),
expiration: new Date((new Date()).getTime() + (3 * 60 * 1000)).toISOString(),
conditions: [
{ bucket: config.bucket },
{ key: filename },
{ acl: 'public-read' },
{ success_action_status: '201' },
{'Content-Type': 'image/*'},
['content-length-range', 0, 100000000000],
{ 'x-amz-algorithm': 'AWS4-HMAC-SHA256' },
{ 'x-amz-credential': credential },
{ 'x-amz-date': dateString() + 'T000000Z' }
],
]
}
}

function dateString() {
var date = new Date().toISOString();
return date.substr(0, 4) + date.substr(5, 2) + date.substr(8, 2);
function dateString () {
var date = new Date().toISOString()
return date.substr(0, 4) + date.substr(5, 2) + date.substr(8, 2)
}

function buildS3UploadSignature(config, policyBase64, credential) {
var dateKey = hmac('AWS4' + config.secretKey, dateString());
var dateRegionKey = hmac(dateKey, config.region);
var dateRegionServiceKey = hmac(dateRegionKey, 's3');
var signingKey = hmac(dateRegionServiceKey, 'aws4_request');
return hmac(signingKey, policyBase64).toString('hex');
function buildS3UploadSignature (config, policyBase64, credential) {
var dateKey = hmac('AWS4' + config.secretKey, dateString())
var dateRegionKey = hmac(dateKey, config.region)
var dateRegionServiceKey = hmac(dateRegionKey, 's3')
var signingKey = hmac(dateRegionServiceKey, 'aws4_request')
return hmac(signingKey, policyBase64).toString('hex')
}

function hmac(key, string) {
var hmac = require('crypto').createHmac('sha256', key);
hmac.end(string);
return hmac.read();
function hmac (key, string) {
var hmac = crypto.createHmac('sha256', key)
hmac.end(string)
return hmac.read()
}

module.exports = {
Expand Down
94 changes: 49 additions & 45 deletions examples/direct-upload/public/client.js
Original file line number Diff line number Diff line change
@@ -1,50 +1,54 @@
// Requires jQuery and blueimp's jQuery.fileUpload
var uploadDemo = (function () {
var filename

// client-side validation by fileUpload should match the policy
// restrictions so that the checks fail early
var acceptFileType = /.*/i;
var maxFileSize = 1000;
// The URL to your endpoint that maps to s3Credentials function
var credentialsUrl = '/s3_credentials';
// The URL to your endpoint to register the uploaded file
var uploadUrl = '/upload';
function saveFile (file) {
filename = file[0].name
}

window.initS3FileUpload = function($fileInput) {
$fileInput.fileupload({
// acceptFileTypes: acceptFileType,
// maxFileSize: maxFileSize,
paramName: 'file',
add: s3add,
dataType: 'xml',
done: onS3Done
});
};
function submitFile () {
getCredentialsFromServer(filename)
}

// This function retrieves s3 parameters from our server API and appends them
// to the upload form.
function s3add(e, data) {
var filename = data.files[0].name;
var params = [];
$.ajax({
url: credentialsUrl,
type: 'GET',
dataType: 'json',
data: {
filename: filename
},
success: function(s3Data) {
data.url = s3Data.endpoint_url;
data.formData = s3Data.params;
data.submit();
function getCredentialsFromServer (filename) {
var xhttp = new XMLHttpRequest()
xhttp.onreadystatechange = function () {
if (this.readyState === 4 && this.status === 200) {
var s3Data = JSON.parse(xhttp.responseText)
console.log('GET RESPONSE', s3Data)
buildAndSubmitForm(s3Data)
var successMessage = document.createElement('h4')
successMessage.innerHTML = 'Image Successfully Uploaded at: '
var link = `https://dwyl-direct-upload.s3.amazonaws.com/${filename}`
var imageATag = document.querySelector('a')
imageATag.setAttribute('href', link)
var imageLink = document.createElement('h4')
imageLink.innerHTML = link
var div = document.querySelector('div')
div.insertBefore(successMessage, div.firstChild)
imageATag.appendChild(imageLink)
}
}
});
return params;
};
xhttp.open('GET', `/s3_credentials?filename=${filename}`, true)
xhttp.send()
}

function onS3Done(e, data) {
var s3Url = $(data.jqXHR.responseXML).find('Location').text();
var s3Key = $(data.jqXHR.responseXML).find('Key').text();
// Typically, after uploading a file to S3, you want to register that file with
// your backend. Remember that we did not persist anything before the upload.
console.log($('<a/>').attr('href', s3Url).text(s3Url).appendTo($('body')));
};
function buildAndSubmitForm (s3Data) {
var form = document.querySelector('form')
var keyInput = document.createElement('input')
keyInput.setAttribute('type', 'hidden')
keyInput.setAttribute('name', 'key')
keyInput.setAttribute('value', `${filename}`)
form.setAttribute('method', 'post')
form.setAttribute('action', s3Data.endpoint_url)
form.setAttribute('enctype', 'multipart/form-data')
form.insertBefore(keyInput, form.firstChild)
form.url = s3Data.endpoint_url
form.formData = s3Data.params
form.submit()
}

return {
saveFile,
submitFile
}
}())
Loading