Skip to content

Adding Direct S3 Uploads

Jump to bottom Edit New page
Ben Koshy edited this page Apr 28, 2022 · 44 revisions

This walkthrough shows how to add asynchronous uploads to a Rails app. The flow will go like this:

  1. User selects file(s)
  2. For each file a presign request is made to fetch AWS S3 upload parameters
  3. Files are uploaded asynchronously to AWS S3
  4. Uploaded file JSON data is written to a hidden field
  5. Form is submitted instantaneously as it only has to submit the JSON data
  6. JSON data is assigned to the Shrine attachment attribute

AWS S3 setup

You'll need to create an AWS S3 bucket, which is where the uploads will be stored. See this walkthrough on how to do that.

Next you'll need to configure CORS for that bucket, so that it accepts uploads directly from the client. In the AWS S3 Console go to your bucket, click on the "Permissions" tab and then on "CORS configuration". There paste in the following:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <CORSRule>
    <AllowedOrigin>https://my-app.com</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>Authorization</AllowedHeader>
    <AllowedHeader>x-amz-date</AllowedHeader>
    <AllowedHeader>x-amz-content-sha256</AllowedHeader>
    <AllowedHeader>content-type</AllowedHeader>
    <ExposeHeader>ETag</ExposeHeader>
  </CORSRule>
  <CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
  </CORSRule>
</CORSConfiguration>

Replace https://my-app.com with the URL to your app (in development you can set this to *). Once you've hit "Save", it may take some time for the new CORS settings to be applied.

1. Installation

Add Shrine and aws-sdk-s3 to the Gemfile:

# Gemfile

gem "shrine", "~> 3.0"
gem "aws-sdk-s3", "~> 1.14"

and run bundle install.

2. Initializer

Add your S3 credentials to your application:

$ rails credentials:edit
s3:
  bucket:            "<YOUR_BUCKET>"
  region:            "<YOUR_REGION>"
  access_key_id:     "<YOUR_ACCESS_KEY_ID>"
  secret_access_key: "<YOUR_SECRET_ACCESS_KEY>"
# ...

Then create an initializer which configures your S3 storage with those credentials and loads default plugins:

# config/initializers/shrine.rb

require "shrine"
require "shrine/storage/s3"

s3_options = Rails.application.credentials.s3

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
  store: Shrine::Storage::S3.new(**s3_options),
}

Shrine.plugin :activerecord           # load Active Record integration
Shrine.plugin :cached_attachment_data # for retaining cached file on form redisplays
Shrine.plugin :restore_cached_data    # refresh metadata for cached files

3. Migration

Add the <attachment>_data text or JSON column to the table to which you want to add the attachment:

$ rails generate migration add_image_data_to_photos image_data:text # or :jsonb

This should generate the following migration:

class AddImageDataToPhotos < ActiveRecord::Migration
  def change
    add_column :photos, :image_data, :text # or :jsonb
  end
end

Run rails db:migrate to apply the migration.

4. Attachment

Create an uploader for the types of files you'll be uploading:

# app/uploaders/image_uploader.rb

class ImageUploader < Shrine
end

and add an attachment attribute to your model:

# app/models/photo.rb

class Photo < ApplicationRecord
  include ImageUploader::Attachment(:image)

  validates_presence_of :image
end

5. Form

In your form you can now add form fields for the attachment attribute, and an image tag for the preview:

<!-- app/views/photos/_form.html.erb -->

<%= form_for @photo do |f| %>
  <!-- ... -->
  <div>
    <%= f.label :image %>
    <%= f.hidden_field :image, value: @photo.cached_image_data, class: "upload-data" %>
    <%= f.file_field :image, class: "upload-file" %>
  </div>
<% end %>

<div class="upload-preview">
  <%= image_tag @photo.image_url.to_s, height: "300" %>
</div>

In your controller make sure to allow the attachment param:

# app/controllers/photos_controller.rb

class PhotosController < ApplicationController
  # ...
  def create
    @photo = Photo.new(photo_params)

    if @photo.save
      redirect_to @photo
    else
      render :new
    end
  end
  # ...

  private

  def photo_params
    params.require(:photo).permit(..., :image) # permit attachment param
  end
end

Now you should have working synchronous file uploads.

6. Direct upload

We can now add asynchronous direct uploads to the mix. We'll be using Uppy and its AwsS3 plugin, which will upload selected files directly to S3 using params fetched from Shrine's presign endpoint.

6a. Presign endpoint

We'll first configure the presign_endpoint plugin and mount it in our routes:

# config/initializers/shrine.rb
# ...
Shrine.plugin :presign_endpoint, presign_options: -> (request) {
  # Uppy will send the "filename" and "type" query parameters
  filename = request.params["filename"]
  type     = request.params["type"]

  {
    content_disposition:    ContentDisposition.inline(filename), # set download filename
    content_type:           type,                                # set content type (defaults to "application/octet-stream")
    content_length_range:   0..(10*1024*1024),                   # limit upload size to 10 MB
  }
}
# config/routes.rb

Rails.application.routes.draw do
  # ...
  mount Shrine.presign_endpoint(:cache) => "/s3/params"
end

6b. Uppy

Now we can setup Uppy to do the direct uploads. First we'll add the package to our bundle (we're assuming you're using webpacker):

$ yarn add uppy

Now we can setup direct uploads to S3, where upload params request will be fetched from Shrine's presign endpoint, and upload result will be written to the hidden attachment field:

// app/javascript/fileUpload.js

import 'uppy/dist/uppy.min.css'

import {
  Core,
  FileInput,
  Informer,
  ProgressBar,
  ThumbnailGenerator,
  AwsS3,
} from 'uppy'

function fileUpload(fileInput) {
  const hiddenInput = document.querySelector('.upload-data'),
        imagePreview = document.querySelector('.upload-preview img'),
        formGroup = fileInput.parentNode

  // remove our file input in favour of Uppy's
  formGroup.removeChild(fileInput)

  const uppy = Core({
      autoProceed: true,
    })
    .use(FileInput, {
      target: formGroup,
    })
    .use(Informer, {
      target: formGroup,
    })
    .use(ProgressBar, {
      target: imagePreview.parentNode,
    })
    .use(ThumbnailGenerator, {
      thumbnailWidth: 600,
    })
    .use(AwsS3, {
      companionUrl: '/', // will call the presign endpoint on `/s3/params`
    })

  uppy.on('thumbnail:generated', (file, preview) => {
    // show preview of the image using URL from thumbnail generator
    imagePreview.src = preview
  })

  uppy.on('upload-success', (file, response) => {
    // construct uploaded file data in the format that Shrine expects
    const uploadedFileData = {
      id: file.meta['key'].match(/^cache\/(.+)/)[1], // object key without prefix
      storage: 'cache',
      metadata: {
        size: file.size,
        filename: file.name,
        mime_type: file.type,
      }
    }

    // set hidden field value to the uploaded file data so that it's submitted
    // with the form as the attachment
    hiddenInput.value = JSON.stringify(uploadedFileData)
  })
}

export default fileUpload

You may wish to initialize on turbolinks (or turbo) loading: or alternatively, using a stimulus controller:

(A) Initialize on Turbo load

// app/javascript/packs/application.js
// ...
import fileUpload from 'fileUpload'

// if you are using turbo, listen on turbo:load instead
// listen on 'turbolinks:load' instead of 'DOMContentLoaded' if using Turbolinks
document.addEventListener('DOMContentLoaded', () => {
  document.querySelectorAll('.upload-file').forEach(fileInput => {
    fileUpload(fileInput)
  })
})

(B) Initialize using a Stimulus Controller

And add the following CSS:

/* app/assets/stylesheets/application.css */
/* ... */
.upload-preview img {
  display: block;
  max-width: 100%;
}

.upload-preview {
  margin-bottom: 10px;
  display: inline-block;
  height: 300px;
}

img[src=""] {
  visibility: hidden;
}

And that's it, now when a file is selected it will be asynchronously uploaded to your app, accompanied by an image preview and a nice progress bar.

Restricting Access to Presign Endpoint

You may want to restrict the access to your presign endpoint to protect your s3 bucket.

One way to do this (if you are using Rails) is to use constraints on your rails route (e.g., example using headers, or via plain ol' ruby):

Rails.application.routes.draw do
      # need a true statement with the constraint - in this case that a header exists using a lambda  
      mount Shrine.presign_endpoint(:cache) => "/s3/params", constraints: -> (req) {
      req.headers["HEADER_THAT_SHOULD_EXIST"]
    }

      # or you can use a ruby class
      mount Shrine.presign_endpoint(:cache), :constraints => AdminConstraint.new # only admins should get access

      # or you can route requests to a separate controller where you can return whatever rack response you design
end

## ./app/lib/admin_constraint.rb
class AdminConstraint
  def matches?(request)
    return false unless request.session[:user_id]
    user = User.find request.session[:user_id]
    user && user.site_admin? # only logged in admin users should be able to upload
  end
end

See Also

Add a custom footer
Add a custom sidebar
Clone this wiki locally