Skip to content

Adding Resumable Uploads

Jump to bottom Edit New page
Janko Marohnić edited this page Sep 18, 2019 · 33 revisions

In this walkthrough we show how to add support for resumable uploads to a Roda & Sequel app, though the instructions are equally applicable to a Rails & Active Record app. Having the ability the resume interrupted uploads is recommended when accepting large files from users (e.g. videos). The flow will go like this:

  1. User selects file(s)
  2. Files are uploaded asynchronously to a resumable upload endpoint
  3. Uploaded file JSON data is written to a hidden field
  4. Form is submitted instantaneously as it only has to submit the JSON data
  5. JSON data is assigned to the Shrine attachment attribute (instead of the raw file)

We will create a separate abstract uploader that will handle files uploaded to the resumable upload endpoint, so that we can still upload other smaller files using the regular upload flow. Uploads will be stored on AWS S3 to make it automatically work on Heroku.

NOTE: If you would like to have resumable uploads directly to S3, you can use the AwsS3Multipart Uppy plugin accompanied with the uppy-s3_multipart gem.

Installation

Add Shrine, aws-sdk-s3, tus-ruby-server and shrine-tus to the Gemfile:

# Gemfile

gem "shrine", "~> 2.9"
gem "aws-sdk-s3", "~> 1.2"
gem "tus-server", "~> 2.1"
gem "shrine-tus", "~> 1.2"

Configuration

Create an initializer that will be loaded when your app boots, where you configure your storage and load initial plugins.

# config/shrine.rb

require "shrine"
require "shrine/storage/s3"
require "shrine/storage/tus"

s3_options = {
  access_key_id:     "<YOUR_ACCESS_KEY_ID>",
  secret_access_key: "<YOUR_SECRET_ACCESS_KEY>",
  region:            "<YOUR_REGION>",
  bucket:            "<YOUR_BUCKET>",
}

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
  store: Shrine::Storage::S3.new(**s3_options),
  tus:   Shrine::Storage::Tus.new,
}

Shrine.plugin :sequel # load integration for the Sequel ORM
Shrine.plugin :cached_attachment_data # for forms

Notice the additional :tus storage, it will download files from the tus server.

Uploader

First create an abstract uploader that will handle files uploaded to the tus server.

# uploaders/tus_uploader.rb

class TusUploader < Shrine
  # use Shrine::Storage::Tus for temporary storage
  storages[:cache] = storages[:tus]
end

Next create your uploader that subclasses the abstract uploader:

# uploaders/video_uploader.rb

class VideoUploader < TusUploader
end

Now add an attachment attribute to your model:

# models/movie.rb

class Movie < Sequel::Model
  include VideoUploader::Attachment.new(:video)
end

You'll also need to add the <attachment>_data text or JSON column to that table:

Sequel.migration do
  change do
    add_column :movies, :video_data, :text # or :jsonb
  end
end

View

In your model form you can now add form fields for the attachment attribute, and an image tag for the preview:

<div class="form-group">
  <input type="hidden" name="movie[video]" value="<%= @movie.cached_video_data %>" class="upload-hidden">
  <input type="file" name="movie[video]" class="upload-file">
</div>

The file field will be used for choosing files, and the hidden field for storing uploaded file data and retaining it across form redisplays in case of validation errors.

Direct upload

We can now add asynchronous direct uploads to the mix. We'll be using a JavaScript file upload library called Uppy (which will require tus-js-client as well) to upload files to the tus server.

Tus Server

The tus server implementation we'll be using is tus-ruby-server. We'll create an initializer where we'll configure the tus server to use AWS S3 storage and to redirect download requests to the S3 objects directly (to avoid serving uploaded files through the app):

# config/tus.rb

require "tus/server"
require "tus/storage/s3"

Tus::Server.opts[:storage] = Tus::Storage::S3.new(
  bucket:            "<YOUR BUCKET>",
  access_key_id:     "<YOUR ACCESS KEY ID>",
  secret_access_key: "<YOUR SECRET ACCESS KEY>",
  region:            "<YOUR REGION>",
)

Tus::Server.opts[:redirect_download] = true # makes download requests redirect to AWS S3

We can now run the Tus::Server application on /files:

# For Roda app
route do |r|
  r.on "files" do
    r.run Tus::Server
  end
  # ...
end

# For Rails app (config/routes.rb)
Rails.application.routes.draw do
  mount Tus::Server => "/files"
  # ...
end

Uppy

Now we can setup Uppy to asynchronous uploads. First we'll pull in the necessary JavaScript and CSS files:

<!DOCTYPE html>
<html>
  <head>
    <script src="https://unpkg.com/babel-polyfill@6.26.0/dist/polyfill.min.js"></script>
    <script src="https://transloadit.edgly.net/releases/uppy/v1.0.0/uppy.min.js"></script>

    <link href="https://transloadit.edgly.net/releases/uppy/v1.0.0/uppy.min.css" rel="stylesheet" />
  </head>

  <body>
    ...
  </body>
</html>

Now we can add the following JavaScript code which will perform direct uploads to the tus server when the user selects the file, assigning the results to the hidden attachment field to be submitted:

function fileUpload(fileInput) {
  fileInput.style.display = 'none' // uppy will add its own file input

  var uppy = Uppy.Core({
      id: fileInput.id,
      autoProceed: true,
    })
    .use(Uppy.FileInput, {
      target: fileInput.parentNode,
    })
    .use(Uppy.Informer, {
      target: fileInput.parentNode,
    })
    .use(Uppy.ProgressBar, {
      target: fileInput.parentNode,
    })

  uppy.use(Uppy.Tus, {
    endpoint: '/files',
    chunkSize: 5*1024*1024, // required unless tus-ruby-server is running on Falcon
  })

  uppy.on('upload-success', function (file, response) {
    // construct uploaded file data from the tus URL
    var uploadedFileData = JSON.stringify({
      id: response.uploadURL,
      storage: "cache",
      metadata: {
        filename:  file.name,
        size:      file.size,
        mime_type: file.type,
      }
    })

    // set hidden field value to the uploaded file data so that it's submitted with the form as the attachment
    var hiddenInput = fileInput.parentNode.querySelector('.upload-hidden')
    hiddenInput.value = uploadedFileData
  })

  return uppy
}

document.querySelectorAll('.upload-file').forEach(function (fileInput) {
  fileUpload(fileInput)
})

And that's it, now when a video is selected it will be asynchronously uploaded to your tus server, and the upload will be automatically resumed in case of any interruptions. During the upload a nice progress bar will be displayed.

See Also

Add a custom footer
Add a custom sidebar
Clone this wiki locally