Skip to content

Adding Resumable Uploads

Jump to bottom Edit New page
Janko Marohnić edited this page May 23, 2018 · 33 revisions

In this walkthrough we will be adding resumable uploads to a Roda & Sequel app, which are suitable when dealing with large files (e.g. videos). We will create a separate abstract uploader that will handle files uploaded to the resumable endpoint, so that if you're uploading other smaller files they can still use the regular workflow. Uploads will be stored on AWS S3 to make it automatically work on Heroku.

Installation

Add Shrine, aws-sdk-s3, tus-ruby-server and shrine-tus to the Gemfile:

# Gemfile

gem "shrine", "~> 2.9"
gem "aws-sdk-s3", "~> 1.2"
gem "tus-server", "~> 2.0"
gem "shrine-tus", "~> 1.2"

Configuration

Create an initializer that will be loaded when your app boots, where you configure your storage and load initial plugins.

# config/shrine.rb

require "shrine"
require "shrine/storage/s3"
require "shrine/storage/tus"

s3_options = {
  access_key_id:     "<YOUR_ACCESS_KEY_ID>",
  secret_access_key: "<YOUR_SECRET_ACCESS_KEY>",
  region:            "<YOUR_REGION>",
  bucket:            "<YOUR_BUCKET>",
}

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
  store: Shrine::Storage::S3.new(**s3_options),
  tus:   Shrine::Storage::Tus.new,
}

Shrine.plugin :sequel # load integration for the Sequel ORM
Shrine.plugin :cached_attachment_data # for forms

Notice the additional :tus storage, it will download files from the tus server.

Uploader

First create an abstract uploader that will handle files uploaded to the tus server.

# uploaders/tus_uploader.rb

class TusUploader < Shrine
  # use Shrine::Storage::Tus for temporary storage
  storages[:cache] = storages[:tus]
end

Next create your uploader that subclasses the abstract uploader:

# uploaders/video_uploader.rb

class VideoUploader < TusUploader
end

Now add an attachment attribute to your model:

# models/movie.rb

class Movie < Sequel::Model
  include VideoUploader::Attachment.new(:video)
end

You'll also need to add the <attachment>_data text or JSON column to that table:

Sequel.migration do
  change do
    add_column :movies, :video_data, :text
  end
end

View

In your model form you can now add form fields for the attachment attribute, and an image tag for the preview:

<div class="form-group">
  <input type="hidden" name="movie[video]" value="<%= @movie.cached_video_data %>" class="upload-hidden">
  <input type="file" name="movie[video]" class="upload-file">
</div>

The file field will be used for choosing files, and the hidden field for storing uploaded file data and retaining it across form redisplays in case of validation errors.

Direct upload

We can now add asynchronous direct uploads to the mix. We'll be using a JavaScript file upload library called Uppy (which will require tus-js-client as well) to upload files to the tus server.

Tus Server

The tus server we'll be using is tus-ruby-server. We can run it inside our Roda app on /files:

require "tus/server"

route do |r|
  r.on "files" do
    r.run Tus::Server
  end

  # ...
end

By default the uploaded files will be stored in the data/ directory. In production we don't want to serve uploaded files through our Ruby application, as that's very inefficient. So, we want to configure our frontend server (Nginx, Apache) to allow serving files, and then use the Rack::Sendfile middleware in our app.

# config.ru (Rack)
use Rack::Sendfile, "X-Sendfile" # Apache, lighttpd
# or
use Rack::Sendfile, "X-Accel-Redirect" # Nginx

# OR

# config/environments/production.rb (Rails)
config.action_dispatch.x_sendfile_header = "X-Sendfile" # Apache and lighttpd
# or
config.action_dispatch.x_sendfile_header = "X-Accel-Redirect" # Nginx

Uppy

Now we can setup Uppy to asynchronous uploads. The easiest way to install it is to pull the JavaScript and CSS files from unpkg:

<!DOCTYPE html>
<html>
  <head>
    <!-- polyfill needed for Uppy and our custom script -->
    <script src="https://unpkg.com/babel-polyfill@6.26.0/dist/polyfill.min.js"></script>
    <!-- tus-js-client that Uppy's Tus plugin will use -->
    <script src="https://unpkg.com/tus-js-client@1.4.5/dist/tus.js"></script>
    <!-- Uppy script -->
    <script src="https://unpkg.com/uppy@0.24.4/dist/uppy.min.js"></script>
    <!-- Uppy stylesheet -->
    <link href="https://unpkg.com/uppy@0.24.4/dist/uppy.min.css" rel="stylesheet" />
  </head>

  <body>
    ...
  </body>
</html>

Now we can add the following JavaScript code which will perform direct uploads to the tus server when the user selects the file, assigning the results to the hidden attachment field to be submitted:

function fileUpload(fileInput) {
  fileInput.style.display = 'none' // uppy will add its own file input

  var uppy = Uppy.Core({
      id: fileInput.id,
    })
    .use(Uppy.FileInput, {
      target: fileInput.parentNode,
    })
    .use(Uppy.Informer, {
      target: fileInput.parentNode,
    })
    .use(Uppy.ProgressBar, {
      target: imagePreview.parentNode,
    })

  uppy.use(Uppy.Tus, {
    endpoint: '/files'
  })

  uppy.run()

  uppy.on('upload-success', function (file, data) {
    // construct uploaded file data from the tus URL
    var uploadedFileData = JSON.stringify({
      id: data.url,
      storage: "cache",
      metadata: {
        filename:  file.name,
        size:      file.size,
        mime_type: file.type,
      }
    })

    // set hidden field value to the uploaded file data so that it's submitted with the form as the attachment
    var hiddenInput = fileInput.parentNode.querySelector('.upload-hidden')
    hiddenInput.value = uploadedFileData
  })

  return uppy
}

document.querySelectorAll('input[type=file]').forEach(function (fileInput) {
  fileUpload(fileInput)
})

And that's it, now when a video is selected it will be asynchronously uploaded to your tus server, and the upload will be automatically resumed in case of any interruptions. During the upload a nice progress bar will be displayed.

See Also

Add a custom footer
Add a custom sidebar
Clone this wiki locally