Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

Geek glossary: re-entrant and idempotent

Whilst writing some Chef recipes for our project’s Continuous Integration server the other day, my pair and I came across a commit message to some third party code that claimed to make a routine re-entrant. We both realised that we didn’t clearly understand the difference between re-entrancy and idempotency and decided to look the terms up.

Read more »

A Stately Resque

Helps ActiveRecord save race condition in Resque

We have a Rails app that is saving a new ActiveRecord object, and then immediately afterwards, enqueueing a Resque job that looks up that record by its ID. Sometimes, the lookup inside the resque job fails – it claims no such row with that ID exists.

Read more »

Accessing the Packages that underlie Apple's App Store

Apple uses its App Store as a mechanism to distribute software, and it works quite well when a human operator is available to interact with it.

Unfortunately, many configuration management tools (e.g. chef, puppet) can't interact with the App Store, but they can interact with MAC OS X installer packages (.pkg, .mpkg files). We'll show you how to extract the underlying installer package file from the App Store.

This was tested under OS X 10.8.1 installing the OS X server package. It may not work for other packages.

Chef cookbook linting with foodcritic

There's a growing movement for testing chef cookbooks, which is great to see. I haven't gotten to play with them all, but some that I've come across are the minitest chef handler, chefspec, and cucumber-chef.

The lowest hanging fruit however is linting, which is where foodcritic comes in. Foodcritic parses your cookbook, and warns you about many common errors.

Food critic will yell at you about all sorts of things - if you are accessing node attributes inconsistently, if you're passing valid ruby as a not_if string, if you're using /tmp instead of file_cache_path, and many more. For a full list, see the github pages

Automatically running foodcritic on your public cookbooks is easy with Travis CI. Just add a .travis.yml to your repo that looks like this:

script: "gem install foodcritic -v 1.1.0 && foodcritic -f any ."
  - 1.9.3

And follow the travis getting started guide. You'll then get an email if you commit something that isn't quite up to par.

You can see a pivotal_workstation run on travis. We've still got a little ways to go to get to green, but everything it's telling us are things we'd like to do. (pull requests welcome!)

Apple Orchard – Turning Chef Recipes into OSX Images

Last week Brian Cunnie posted a great writeup of what we've been working on for building OSX Lion workstations. Today, I'd like to introduce Apple Orchard - how we transform those chef recipes into ready to use OSX Lion images.

The story starts a few months ago, one morning after Standup while putting our dishes from breakfast away. Over the past few days we'd been discussing how our ops group would take chef recipes (generated for the most part by developers) and turn them into machine images that they could deploy on a moments notice. I approached Sean Beckett, our Director of Operations, and told him of my vision:

no manual steps

He looked at me like I was crazy, and he was obviously trying to figure out how to talk me down off the ledge. I told him how Jenkins could run a job after every checkin (a fact he was well aware of) and how all it had to do was...... He backed away slowly.

A few weeks later, Brian Cunnie had gotten past the Minimum Viable Image release marker in Tracker, and I told him of my vision:

no manual steps

He also looked at me like I was crazy, which he often does. The next week I had the afternoon to pair with him, and we got to work. We already had Jenkins building our recipes on a mac mini with Deep Freeze (software which allows you to reboot to a clean state), so we copied that Jenkins job at got to work. We got an iMac, and partitioned the disk into two.

Step 1

We boot the image to the persistent side, and image the dynamic side with a "mostly pristine" image that has X Code preinstalled, and has SSH turned on. We then set the machine to boot from that partition, and reboot.

Step 2

When it comes up, we SSH in, upload an SSH key, clone our public and private cookbooks (our private cookbook is used for our site licenses) and run soloist. If it succeeds, we move on to step 3.

Step 3

Reboot the machine to the persistent partition, and wait for it to come up. This isn't part of Step 2 because we want to leave the machine in the dirty state for troubleshooting if the chef run failed, and only trigger this if it succeeds.

Step 4

Put a script in place that will automatically rename the machine when it first boots, take an image of the partition using diskutils, scan it for restore, and move it over to our Deploy Studio server, and create a symlink so the 'lion HEAD' build points to the newly generated image.

That's it. We occasionally promote a 'lion HEAD' build to 'lion STABLE', so that we've always got an image on hand that we're confident in. But the overhead of cutting a new image is now simply changing a symlink.

There are a lot of moving parts, and sometimes it breaks. With time, it's become more reliable, but still has a lot of external dependencies. We've recently been trying out a strategy of pre-populating the chef and homebrew caches, which seem to be helping. Another caveat we've run into is that so far with Lion we've been unable to produce a universal image that will boot both MacBook Airs and iMacs, but we hope this may have changed with the latest 10.7.2 update.

Many thanks to Brian Cunnie - while I was the reason for this madness, he's done most of the heavy lifting with my occasional help.

It's all open source on github, at Values for our infrastructure are hard coded, but if you'd like to generalize it and use it, fork and make a pull request.

Got Lion? Now Get Everything Else.

So you have a nice new Apple machine running Lion, but you don't want to spend the next few hours installing software.

What to do?

The short answer: use chef/soloist in conjunction with a slew of recipes developed at Pivotal Labs to help install the most common set of components.

Here's what to do:

  1. make sure you have Xcode installed; if not, you can install it through the App Store. Don't worry--it's a free application.
  2. Do the following:

    sudo gem install soloist
    mkdir -p ~/workspace
    cd ~/workspace
    git clone
    cat > ~/soloistrc <<EOF
    - $HOME/workspace
    - pivotal_workstation::meta_osx_base
    - pivotal_workstation::meta_osx_development
    - pivotal_workstation::meta_ruby_development
  3. It typically takes an hour for the chef run to complete. You can do other things while it's running (but if you reboot or logout you'll need to restart the chef run).

[Chef is a framework written by OpsCode to help configure in maintain one or more machines using 'recipes' (ruby scripts, more or less).]

[Soloist is an application which makes running chef-solo (the version of chef which runs without any centralized server) easier. It was written by my co-worker Matthew Kocher.]

What Is Installed by Default??

At the end of the chef run [as of this writing, and if all goes well] the following software will be installed:

  • java
  • Skype
  • SizeUp
  • Firefox
  • DropBox
  • Chrome
  • Mouse Locator
  • Homebrew
  • Pivotal Git scripts
  • Gitx
  • Vim
  • CCMenu
  • TextMate
  • KeyCastr
  • rvm
  • MySQL
  • postgres
  • RubyMine
  • ImageMagick
  • node.js

The following services will be enabled:

  • sshd
  • screen sharing
  • locate

The following preferences will be set:

  • better bash (history, PS1, inputrc)
  • faster key repeat
  • git (settings, a few scripts)
  • TextMate preferences (e.g. soft tabs)
  • RubyMine preferences (e.g. keymaps)

What if I don't want all that software? I just want TextMate and node.js!

Want to change the software that is installed? It's simple: just change your ~/soloistrc file. Here's a soloistrc that will only install TextMate & node.js:

- ./workspace
- pivotal_workstation::textmate
- pivotal_workstation::node_js

If you're interested in seeing all the recipes available (and there are quite a few), just browse the recipes in the pivotal_workstation repo.

Why Did You Choose That Set of Software?

Early in June, several pivots (Sean Beckett, Matthew Kocher, and David Goudreau, and I) met to decide on the bare minimum set of software and features that our developers would need to function on a new Lion Machine.

This set of chef recipes is the result of that meeting. There have been some changes (we have had great difficulty writing recipes to install firefox addons, so we iceboxed the story; some of our developers contributed recipes for things they wanted, so we added those).

Why Did You Choose Chef?

We chose chef/soloist partly because felt that our previous process had reached the end of its usefulness and were familiar with chef from our work automating server configuration.

Here's how our previous process worked:

  1. For minor releases (e.g. 10.6.7 → 10.6.8), we would take the previous golden image (a golden image is a snapshot of the disk drive of a machine with the applications, preferences, and settings that we wanted), install it on a workstation, upgrade the OS and possibly upgrade some of the applications. We would then use DeployStudio to take an image of the workstation, and that image would become the new golden image.
  2. For major releases (e.g. 10.5 → 10.6), we would re-create the golden image by hand, manually installing & configuring the individual software. For the Leopard/Snow Leopard transition, my co-worker Kevin Fitzpatrick spent a week painstakingly configuring the new machine. We then took an image using DeployStudio, and that image became the golden image.

This approach had several shortcomings:

  • It was monolithic: if you were a developer, there was no choice: there was only one image. This wasn't so bad when we were strictly a ruby shop, but when we expanded into android and iphone development, the monolithic approach began to show some shortcomings.
  • There was cruft in the image: the golden image had been built up over years.
  • It wasn't clear that the Golden Image would make the jump to Lion: Lion introduced some big changes (e.g. no PPC executables).
  • We were hesitant to approach the developers to ask them what they would like to see on the new image; we worried of re-igniting a Holy War.

We looked for alternatives. We wanted the following features:

  • We wanted to be able to install all (or almost all) of the features automatically (with minimal user intervention)
  • We wanted to be able to pick-and-choose which features were installed; the needs of an Android developer were different than those of an iPhone developer.
  • We want our developers to be happy; sure, they could install the features that they wanted or fix a problem with their workstation, but we want to go a step further: we want to provide the resources they need to write a recipe to minimize the effort the next developer has to go through. [This has been a success: several of our recipes were written in conjunction with developers.]


Integration tests for the cookbook took several days to set up. We use Faronic's Deep Freeze on a fairly pristine mac mini to ensure that we have a clean machine when we run our chef scripts. Continuous integration has proven invaluable for collaboration, for we quickly learn if a commit has unintended consequences.

In the more complex chef recipes, we attempt to write tests to test that they [the recipes] have succeeded; sshd_on.rb is a good example of testing that a service (sshd) was correctly started.


The chef runs, especially the initial one, are flaky. Our current chef run must download software from over 40 different servers, any one of which being down or having changed the download location can cause a failure. For example, Little CMS, a dependency of ImageMagick, resided on, which was down for a few days. Our integration tests failed during that period.

If you encounter a server being down or a file that has moved, please send us a pull request with an updated download location, or just comment-out the broken recipe.

Target Audience

Our target audience is developers, which is great: they understand errors, and often contribute code fixes. Our goal is for Pivotal Ops to provide a framework for Pivotal Engineers to write the recipes that build the workstations they want.


I am grateful to Matthew Kocher, who more than anyone helped me write the bulk of the ruby scripts. Also to Sean Beckett, without whose support this would never have happened. And to the many pivots who offered suggestions & help.

The simplest deployment that could possibly work

I've been working on a basic chef solo based rails deployment. I started with Building an AMI, then bootstrapping to get the instance running chef with capistrano. Since part two left us with chef running on a box with recipes, it's time to get a rails app running and serve some requests.

For the moment, I've created an application.rb recipe, which is the only thing in the soloistrc. It declares dependencies:

include_recipe "joy_of_cooking::daemontools"
include_recipe "joy_of_cooking::mysql"

and then gets on with dealing with the app server.

'What is daemontools'? I hear you cry out. it's the best way I've found to ensure processes are running as you expect. If you aren't using daemontools (and you probably aren't) it's worth looking at seriously. Daemontools takes the pids out of process management. There are no pid files to get out of sync with processes, and there's no polling to see if something is up. Daemontools know if something exists because it is a child process, and when it exits it returns control to the supervise process. Deamontools is worth wrapping your head around, but for now, all you need to know is that you give it a script and it runs the script in an infinite loop. I stole the install procedure from Michael Sofaer's Hellspawn, so you'll have to excuse the fact that it looks like ruby instead of chef.

ruby_block "install daemontools" do
  block do
    directory = "/package/admin"
    repo = "git://"
    dir_name = "daemontools-0.76"
    FileUtils.mkdir_p directory
    system("cd #{directory} && git clone #{repo} #{dir_name}")
    system("cd #{File.join(directory, dir_name)} && ./package/install")
  not_if "ls /command/svscanboot"

Once daemontools is installed, we move on to installing mysql. A database isn't necessary for an app, but it's necessary for doing anything interesting. I made a philisophical decision to compile mysql from source. I tend to view compiling from source as a continuum - you usually would compile your own app - you usually wouldn't compile your own kernel. The DB is closely related enough to your app that it's nice to have exact documentation about what you're running. It'll also lower the barrier to entry to trying out Percona or Drizzle.

The mysql recipe is a little too long to inline here, but you can and should check it out on github. The recipe starts by installing some dependencies (by all means, leverage your package management system here) create a mysql user and download/cmake/make install. Add a deamontools run script, and up it comes. Set up the users and we're all set. The most interesting part of the recipe to me is the block which waits for mysql to start up:

ruby_block "wait for mysql to come up" do
  block do
    Timeout::timeout(60) do
      until system("ls /tmp/mysql.sock")
        sleep 1

Usually I'd just throw in a sleep, but I was ashamed to share that with the world. This is a technique I'll carry forward, and would love to see it make its way into chef so those less prone to dropping into ruby could make use of it. (Also, if there's a better way or something that's already in Chef that I haven't come across, I'm all ears)

Once mysql is up and running, we do the git clone/bundle/db:create/db:migrate steps rails developers have come to know and love, then write out a deamontools run script and use svc to make sure the unicorn restarts on every deploy. I'm not a fan of the cap style cached copy/magical rollback that the chef deploy resource reimplements, so for now I'm rolling my own.

Once the app is ready to be started, write out a daemontools run script:

file "/service/unicorn/run" do
  content %{#!/bin/bash
cd /var/staging/foo/src
export RAILS_ENV=staging
source /home/mkocher/.rvm/scripts/rvm
rvm use ruby-1.8.7-p299@captest
exec /command/setuidgid mkocher rackup -p 3000
  mode "0755"

And daemontools starts up the app. The run script is pretty easy to follow - set up RVM, chose our ruby, and exec unicorn.

Hitting the app on 3000 reveals a rails app in all its scaffolded glory:

app screenshot

There's more work still do to. There's a repetition that needs to be refactored out. Capistrano, Chef and Rails all need to share some knowlege about the world. Cap and chef need to know the directory we're deploying the thing to, chef and rails need to know what the database configuration is, and so on. Cap has settings, Chef has nodes and Rails has Configure blocks. I'm leaning towards reinventing the weel again very simply, but I'm open to suggestions. Other todo items are putting nginx in front of the app server, and moving to a dedicated database server.

I'm not sure which direction this will go next, but I hope by now you're convinced that chef recipes aren't magic - they're just a thin ruby wrapping around the things you do to configure a box - execute some commands and edit some files.

All the recipes mentioned are available on github and MIT licensed, please steal them and make them better.


  • You'll need to add whatever ports you'd like open to your EC2 security group.
  • Having mysql on a AMI backed instance with the data on the local disk is the opposite of production ready.
  • Sometimes things in the world changes - either mirror files yourself, or expect the occasional mirror to disappear failing a chef run. Wayne Seguin will also occasionally change the install script location for RVM. Chef recipes are living things which must be tended to occasionally.

Chef Solo – Tugging on the bootstraps

There's certainly no wrong way to run chef, however, a reasonable set of defaults are usually helpful as a starting place. I'm building an example of a simple deploy with chef, with the goals of ending up with something that's easy to understand, copy and modify. Eventually, I hope this allows for discussion of and iteration on what a solid ruby app stack should include.

The first stumbling block with chef for most people is not usually building am ami. It's usually getting chef and their recipes runningon a box. At Pivotal we're firm believers in doing the simplest thing that could possibly work, and so what follows is the just least I could do to cleanly run chef. Essentially, you want to install the ruby prerequisites, install a known Ruby (let Wayne figure out how), install chef, get some cookbooks on the box and run chef. This boils down to a script written in Bash to get you to the point you can run Ruby, and some capistrano tasks to install gems and run chef.

I've written Soloist, a gem I use for invoking chef solo easily. The goal is for it to be a thin layer over chef-solo to make it more friendly to use. I'll use it here, but all it's doing is generating a solo.rb and node.json files - it's not magic.

Here are the files I ended up with:

├── Capfile                   <- How we tell cap what to do
├── Gemfile                   <- The three gems we need
├── Gemfile.lock
├──          <- The bash script to bootstrap an instance
├── chef
│   └── cookbooks         <- Chef cookbooks go here
│       └── joy_of_cooking
│           └── recipes
│               └── default.rb
├── config
│   └── deploy
│       └── staging.rb        <- What servers are in your environment
└── soloistrc             <- How you'll tell chef what to run

First off we'l start with a bash script. You can see the full thing on github, but the psuedocode is:

as root:
    Create an app user account
    allow password authentication       # adding an Authorized key would be fine too
    add epel as an RPM repository       # for an easy git instal
    install git
    install the rvm prereqs             # readline, zlib, openssl and more
    enable passwordless sudo            # it's the only way to live
as user:
    install rvm
    add rvm to bashrc
    set rvm to trust all rvmrcs, install rubys on use and install gemsets on use

That's it. My example is 41 lines of bash (including 11 blank and 6 comment lines).

Capistrano is a reasonable multi server SSH tool. Parts of it are fundamentally flawed, but the "ssh out to some servers and run some commands" works just fine. So, we'll use it to upload and run the bootstrap script:

desc "bootstrap"
task :bootstrap do
  set :user, "root"
  set :default_shell, "/bin/bash"
  upload "", "/root/"
  run "chmod a+x /root/"
  run "/root/"

The boostrap task sets the user to root because everywhere else the user is set to the app user. We set the shell to /bin/bash to override the use of rvm-shell. We then upload the bootstrap script and run it.

The deploy task consists of:

desc "Deploys"
task :deploy do

To see the details of each task, see the Capfile. Be aware the first time you deploy, rvm will compile the requested ruby - this will take a few minutes.

This leaves us at 4 lines in the terminal to spin up as many ec2 instances as amazon will let you and run chef on them:

ec2-run-instances ami-e67e8d8f -k mine2 -t m1.large
ec2-describe-instances  # you'll need to add the IP(s) of your
                        # new instance(s) to the staging.rb file
cap staging bootstrap
cap staging deploy

Currently, the chef run consists of a recipe which touches a file in the root directory. For my next trick, I hope to actually run a web app. Stay tuned.

All the source code is available on github.

Lessons learned:

  • Rubyforge will be down when you'd like to to be up.
  • Learning the ec2 shell commands is well worth the time over using the web interface.

Open questions/issues:

EC2 AMI Building Adeventures

As rails developers we're often presented with the task of finding the right ops solution for clients. While Heroku has turned out to be a great answer for many of our clients, there are others who don't fit in the (very well made) fixed size box that Heroku has built.

For these projects, the best answer we've found is going the devops route using chef solo. The first problem encountered is "how do I bootstrap my servers", which by its very nature is a hard problem that requires a fairly through understanding of both your projects needs, operations and chef. The best answer has been to ignore bootstrapping at the beginning, and use chef to document/automate what's really different about your app.

Getting from servers being a bunch-o-magic-bits to a 15 minute bootstrapping followed by a chef run is where most of the value of automated configuration is.

However, this answer doesn't satisfy the primal urge to automate everything. To that end, I'm embarking on a side project of bootstrapping a rails server from scratch with chef solo. My hope is that it will be useful as a starting point for rails projects looking to automate their infrastructure. I turned over some turtles, and found the stack ended when I had to chose which AMI to boot up on EC2.

I chose Centos in hopes of leveraging the knowledge of operations experts, and went looking for a basic AMI. There are thousands out there, but what I really wanted was something fully documented in code. Amazon provides "standard" images which are centos based, but they're too much of a collection of "magic bits" for my tastes. Rightscale is nice enough to give out their scripts, and I've adapted them (by way of Nicky Peeters ) to build the barest of bare centos AMIs.

The biggest hurdle was getting a 32 bit AMI that was as close as possible to a 64 bit AMI. I had the urge to standardize on the 64bit only, but the price difference is fairly substantial.

I figured this was a one afternoon project, but it turned into a full two weekend project as I learned many things about EC2, Centos and Linux.

Lessons learned or relearned:

  • EC2 requires different /etc/fstab's for different instance types.

  • EC2 does not use the AMI's installed kernel by default, and which one you pick is important. If your centos install hangs at "Creating /dev", this is probably the reason. PV-Grub looks interesting, but I didn't want to add another moving part yet.

  • A special ldconfig is necessary on 32bit servers, but seems to break 64bit servers. If you're seeing linker errors durring the boot up process, this might be the problem.

  • The version of yum used to create the image is important - using the yum provided with an earlier 5.x point release did not yield a bootable 5.5 Centos image.

  • The version of amazon's AMI building tools is important. The --kernel option is not available in older versions of the bundle command, which didn't turn out to be necessary but did cause problems. There may have been another reason which currently escapes my memory. The script now installs them at the beginning of the run.

  • /dev/ptmx needs to be created by hand on 64bit servers, but is unnecessary or already created on 32bit servers. /dev/ptmx provides psudo terminals for SSH - you can't get into your server 64bit server without creating one.

Further questions:

  • Is everything above really true? Is there anything that can be left out or simplified?

  • How do I make a matching vagrant box from the same script?

  • EBS Backed AMIs seem to be better in many ways. Can the same script be pointed at an EBS volume?

As this is my first pass, I'm not sharing prebuilt AMI's yet as they're likely to change fairly rapidly.

The script is available on github.

Standup, 6/7/2010


Our custom Chef recipe stopped working on EY cloud

We have some custom Chef recipes that we have been running on EY Cloud for some time. The code for these custom recipes was using the chef-deploy gem, which was apparently always available in the environment in which the recipe ran.

However, when we ran our recipe last week, it blew up trying to require the gem. We fixed this by removing the require and folding some of the code from the gem into our recipe.

There is a corresponding issue in the EY community forums: