Resque Heroku Scaler

This gem provides autoscaling for Resque workers on Heroku. Based on previous scaling work developed by Daniel Huckstep and Alexander Murmann.

Autoscaling behavior is provided through a separate monitor process. The scaler monitor process polls for pending jobs against the specified Resque Redis backend at a configurable interval. The scaler process runs as a worker process on Heroku.

Blog Post

For details on the motivation behind using a separate scaler process, please see this post.

Setup

Add the following environment variables to your Heroku environment:

• HEROKU_APP
• HEROKU_USERNAME
• HEROKU_PASSWORD

Include the scaler tasks in a file within lib/tasks (ex: lib/tasks/scaler.rake)

require 'resque/tasks'
require 'resque/plugins/heroku_scaler/tasks'

task "resque:setup" => :environment

In your Procfile, configure the scaler as a worker process using:

scaler: bundle exec rake resque:heroku_scaler

To run the scaler process, use the following command. Note, the scaler process is intended to run as a single instance.

heroku scale scaler=1

Require the worker extensions within the app running the workers. For example, in lib/tasks/resque.rake.

require 'resque/tasks'

task "resque:setup" => :environment do
  require 'resque-heroku-scaler'
  ENV['QUEUE'] = '*'
end

In your development environment, the scaler process can run local worker processes using the rush library. To configure, update your scaler file in lib/tasks to use the local scale manager below (ex: lib/tasks/scaler.rake).

require 'resque/tasks'
require 'resque/plugins/heroku_scaler/tasks'

task "resque:setup" => :environment do
  if Rails.env.development?
    require 'resque-heroku-scaler'
    ENV["RUSH_PATH"] ||= File.expand_path('/path/to/app', __FILE__)
    Resque::Plugins::HerokuScaler.configure do |c|
      c.scale_manager = :local
    end
  end
end