Close
Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

LABS
Parallelize Your RSpec Suite

We all have multi-core machine these days, but most rspec suites still run in one sequential stream. Let’s parallelize it!

The big hurdle here is managing multiple test databases. When multiple specs are running simultaneously, they each need to have exclusive access to the database, so that one spec’s setup doesn’t clobber the records of another spec’s setup. We could create and manage multiple test database within our RDBMS. But I’d prefer something a little more … ephemeral, that won’t hang around after we’re done, or require any manual management.

Enter SQLite’s in-memory database, which is a full SQLite instance, created entirely within the invoking process’s own memory footprint.

(Note #1: the gist for this blog is at http://gist.github.com/108780)

(Note #2: The following strategy is relatively well-known, but I thought it might be useful for Pivots-and-friends to see exactly how one Pivotal project has used this tactic for a big speed win.)

Here’s the relevant section of our config/database.yml:

test-in-memory:
  adapter: sqlite3
  database: ':memory:'

Next, we need a way to indicate to the running rails process that it should use the in-memory database. We created an initializer file, config/intializers/in-memory-test.db:

def in_memory_database?
  ENV["RAILS_ENV"] == "test" and
    ENV["IN_MEMORY_DB"] and
    Rails::Configuration.new.database_configuration['test-in-memory']['database'] == ':memory:'
end

if in_memory_database?
  puts "connecting to in-memory database ..."
  ActiveRecord::Base.establish_connection(Rails::Configuration.new.database_configuration['test-in-memory'])
  puts "building in-memory database from db/schema.rb ..."
  load "#{Rails.root}/db/schema.rb" # use db agnostic schema by default
  #  ActiveRecord::Migrator.up('db/migrate') # use migrations
end

Note that in the above, we’re initializing the in-memory database with db/schema.rb, so make sure that file is up-to-date. (Or, you could uncomment the line that runs your migrations.)

Let’s give that a whirl:

$ IN_MEMORY_DB=1 RAILS_ENV=test ./script/console
Loading test environment (Rails 2.3.2)
connecting to in-memory database ...
building in-memory database from db/schema.rb ...
-- create_table("users", {:force=>true})
   -> 0.0065s
-- add_index("users", ["deleted_at"], {:name=>"index_users_on_deleted_at"})
   -> 0.0004s
-- add_index("users", ["id", "deleted_at"], {:name=>"index_users_on_id_and_deleted_at"})
   -> 0.0003s

...

>>

Super, we can see that the database is being initialized our of our schema.rb, and we get our console prompt. We’re ready to roll!

But, running this:

IN_MEMORY_DB=yes spec spec

will still only result in a single process, albeit one running off a database that’s entirely in-memory. We want parallelization!

The final step is a script that will run your spec suite for you. You may need to edit this for your particular situation, but then again, maybe not.

#  spec/suite.rb

require "spec/spec_helper"

if ENV['IN_MEMORY_DB']
  N_PROCESSES = [ENV['IN_MEMORY_DB'].to_i, 1].max
  specs = (Dir["spec/**/*_spec.rb"]).sort.in_groups_of(N_PROCESSES)
  processes = []

  interrupt_handler = lambda do
    STDERR.puts "caught keyboard interrupt, exiting gracefully ..."
    processes.each { |process| Process.kill "KILL", process }
    exit 1
  end

  Signal.trap 'SIGINT', interrupt_handler
  1.upto(N_PROCESSES) do |j|
    processes << Process.fork {
      specs.each do |array|
        if array[j-1]
          require array[j-1]
        end
      end
    }
  end
  1.upto(N_PROCESSES) { Process.wait }

else
  (Dir["spec/**/*_spec.rb"]).each do |file|
    require file
  end
end

Then, you simply run IN_MEMORY_DB=2 spec spec/suite.rb to run two parallel processes. Increase the number on larger machines for better results!

There’s room for improvement here, notably in the naive method used to allocate the spec files to processes, but even as simple as this method is, our spec suite runs in about half the time it used to, on a dual-core machine.

Comments
  1. Not sure if I missed something, but would it be possible to use DeepTest (http://github.com/qxjit/deep-test/tree/master) to do the same thing? Just tell it the number of workers you’d like to use in your declared SpecTask and it’ll handle the parallelization for you.

  2. Mike Dalessio says:

    Ah, now I feel shame. DeepTest appears to do exactly this for RSpec.

    However, one thing I’d like to do is extend the above solution to support Cucumber. It’s not obvious from DeepTest’s docs whether it’s capable of doing this out of the box.

    What’s interesting is that DeepTest’s management of MySQL databases would support running Selenium tests, which an in-memory db cannot do (because the runner and the server are in separate processes).

    So, I may instead try to hack DeepTest to support Cucumber. Thanks for the pointer!

  3. Alex Chaffee says:

    I’d *love* it if you hack [http://deep-test.rubyforge.org/](Deep Test) to support Cucumber. I imagine that DeepTest + Cucumber + WebRat + Selenium => smoking CPU (which in this case is a good thing, since you’re not wasting cycles waiting for the browser’s I/O).

    It looks like someone forked it onto [GitHub](http://github.com/qxjit/deep-test) so if you do your work there, let me know and I’ll get in touch with Dan to synchronize the [http://deep-test.rubyforge.org/](Rubyforge version).

  4. Alex Chaffee says:

    I’d *love* it if you hack [Deep Test](http://deep-test.rubyforge.org/) to support Cucumber. I imagine that DeepTest + Cucumber + WebRat + Selenium => smoking CPU (which in this case is a good thing, since you’re not wasting cycles waiting for the browser’s I/O).

    It looks like someone forked it onto [GitHub](http://github.com/qxjit/deep-test) so if you do your work there, let me know and I’ll get in touch with Dan to synchronize the [Rubyforge version](http://deep-test.rubyforge.org/).

  5. Hey guys,

    You should check out my project Testjour (http://github.com/brynary/testjour/tree/master). It parallelizes Cucumber runs over SSH and handles MySQL database management. Check out Testjour’s own Cucumber features for example usage.

    We use it to run our giant (11k steps) Cucumber build across mac minis in the office. I’m planning on extending it with RSpec support soon, and I’ve got a long term goal of integrating it with EC2.

    Cheers,

    -Bryan

  6. Just a heads up, all – the repository on Github is the official one. It’s maintained by David Vollbracht, one of the main contributors to and maintainers of DeepTest, and (in cooperation with Dan Manges) it’s considered official now. The commits made there are propogated over to Rubyforge; use it, and contribute to it (yay!), in preference to that one. One of these days we’ll coordinate it with the ThoughtWorks GitHub account; apologize for the confusion, and stay tuned.

  7. grosser says:

    I made a plugin of those scripts, and changed some things that did not work out for me (like loading spec/spec_helper first) http://github.com/grosser/parallel_specs hope you like it or can contribute :)

  8. Dan says:

    We also do this across EC2, we deal with the DBs (Mysql, Postgres, Sqlite) and currently support Test:Unit and Rspec. We are seriously considering Cucumber, but are still evaluating the demand.

    Anyways we would love to get feedback from additional users so if you have any projects you want to try out just let me know and I would be happy to hook you up with an account.

  9. […] execution to several machines in your lab. Here are a couple of articles related to this topic: – Pivotal Labs on how to parallelize RSpec tests – Java.net blog post on how to parallelize JUnit tests – How to distribute Selenium tests on […]

Post a Comment

Your Information (Name required. Email address will not be displayed with comment.)

* Copy This Password *

* Type Or Paste Password Here *