Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

Resque Heroku Autoscaler Gem

For my side project I needed some worker processes. My project is hosted on Heroku which makes it really easy to add workers. However, currently I am the only user of that project and don’t need a worker to be running all the time. Automatically scaling the workers up when there are pending Resque jobs and scaling them back down once all the work is done seemed like to be the obvious solution to this. Googling immediately brought up a good blog post by Daniel Huckstep. As good as that solution looks, I don’t like to have untested copy-and-paste code in my project. So I decided to test drive my own solution and put it into a convenient gem which of course can be found on Github as well.


In order for the scaling to work Resque Heroku Atoscaler (RHA) needs to know your Heroku app’s name, your Heroku user name and your Heroku password. Per default those are being read from the following environment variables:


If you prefer you also can configure RHA using a config block. For example you might put the following into config/initialiazers/resque_heroku_autoscaler_setup.config

require 'resque/plugins/resque_heroku_autoscaler'

Resque::Plugins::HerokuAutoscaler.config do |c|
  c.heroku_user = 'john doe'
  c.heroku_pass = ENV['HEROKU_PASSWORD']
  c.heroku_app  = "my_app_#{Rails.env}"

To use RHA in one of your jobs, just extend your job class with Resque::Plugins::HerokuAutoscaler.

For example:

require 'resque/plugins/resque_heroku_autoscaler'

class TestJob
  extend Resque::Plugins::HerokuAutoscaler

  @queue = :test

  def perform
   ...awesome stuff...

When you add the job to your Resque queue, a new worker will be started if there isn’t already one. If all jobs in the queue are processed the worker will be stopped again, keeping your costs low.

Changing Scaling Behavior

Per default RHA will only start a single worker, no matter how many jobs are pending. You can change this behavior in the config block as well:

require 'resque/plugins/resque_heroku_autoscaler'

Resque::Plugins::HerokuAutoscaler.config do |c|
  c.new_worker_count do |pending|

When calculating the new number of required workers the block given to new_worker_count will be called. Thus the example will result in starting one additional worker for every 5 pending jobs.

Please let me know if you run into any problems with the gem or can think of any features that might be helpful.

  1. This is awesome! Glad you found my post useful :)

  2. Tom Andersen says:

    Why would you limit the number of workers? As long as they are all busy actually working on something the cost will be the same if one worker or 20 are run at once. The advantage to scaling up the # of workers is that you get all the queued up jobs done faster….

    What I can’t figure out is why Heroku does not do this, at least as an option. They should also I guess charge more per hour of actual work. The current system that they try to charge for seems useless for many tasks. I have some tasks that can run an hour – and I don’t want to hold up the usual 1 minute tasks. So I had to do something like you outline above, which is a great solution, other than the limit on workers.

Post a Comment

Your Information (Name required. Email address will not be displayed with comment.)

* Copy This Password *

* Type Or Paste Password Here *