Close
Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

LABS
EC2 AMI Building Adeventures

As rails developers we’re often presented with the task of finding the right ops solution for clients. While Heroku has turned out to be a great answer for many of our clients, there are others who don’t fit in the (very well made) fixed size box that Heroku has built.

For these projects, the best answer we’ve found is going the devops route using chef solo. The first problem encountered is “how do I bootstrap my servers”, which by its very nature is a hard problem that requires a fairly through understanding of both your projects needs, operations and chef. The best answer has been to ignore bootstrapping at the beginning, and use chef to document/automate what’s really different about your app.

Getting from servers being a bunch-o-magic-bits to a 15 minute bootstrapping followed by a chef run is where most of the value of automated configuration is.

However, this answer doesn’t satisfy the primal urge to automate everything. To that end, I’m embarking on a side project of bootstrapping a rails server from scratch with chef solo. My hope is that it will be useful as a starting point for rails projects looking to automate their infrastructure. I turned over some turtles, and found the stack ended when I had to chose which AMI to boot up on EC2.

I chose Centos in hopes of leveraging the knowledge of operations experts, and went looking for a basic AMI. There are thousands out there, but what I really wanted was something fully documented in code. Amazon provides “standard” images which are centos based, but they’re too much of a collection of “magic bits” for my tastes. Rightscale is nice enough to give out their scripts, and I’ve adapted them (by way of Nicky Peeters ) to build the barest of bare centos AMIs.

The biggest hurdle was getting a 32 bit AMI that was as close as possible to a 64 bit AMI. I had the urge to standardize on the 64bit only, but the price difference is fairly substantial.

I figured this was a one afternoon project, but it turned into a full two weekend project as I learned many things about EC2, Centos and Linux.

Lessons learned or relearned:

  • EC2 requires different /etc/fstab’s for different instance types.

  • EC2 does not use the AMI’s installed kernel by default, and which one you pick is important. If your centos install hangs at “Creating /dev”, this is probably the reason. PV-Grub looks interesting, but I didn’t want to add another moving part yet.

  • A special ldconfig is necessary on 32bit servers, but seems to break 64bit servers. If you’re seeing linker errors durring the boot up process, this might be the problem.

  • The version of yum used to create the image is important – using the yum provided with an earlier 5.x point release did not yield a bootable 5.5 Centos image.

  • The version of amazon’s AMI building tools is important. The –kernel option is not available in older versions of the bundle command, which didn’t turn out to be necessary but did cause problems. There may have been another reason which currently escapes my memory. The script now installs them at the beginning of the run.

  • /dev/ptmx needs to be created by hand on 64bit servers, but is unnecessary or already created on 32bit servers. /dev/ptmx provides psudo terminals for SSH – you can’t get into your server 64bit server without creating one.

Further questions:

  • Is everything above really true? Is there anything that can be left out or simplified?

  • How do I make a matching vagrant box from the same script?

  • EBS Backed AMIs seem to be better in many ways. Can the same script be pointed at an EBS volume?

As this is my first pass, I’m not sharing prebuilt AMI’s yet as they’re likely to change fairly rapidly.

The script is available on github.

Comments
  1. Toby Crawley says:

    Have you taken a look at [BoxGrinder](http://boxgrinder.org/)? It takes a yaml config file as input, and can spit out preconfigured images for ec2/virtualbox/vmware in 32 and 64 bit. The images are as close to identical as they can be, taking in to account platform variances.

  2. Matthew Kocher says:

    I hadn’t seen it, looks very interesting. I don’t mind reinventing the wheel, but it’s great to find someone maintaining a similar wheel.

  3. – The script looks pretty tight to me, I’ve done scripts to build AMIs a number of times and what you have looks pretty similar. You clean up more than I usually do but that isn’t a bad thing. I would say it is pretty boiled down if you are set on using a script.
    – It might be better to create a script to generate a Vagrant box first and then transfer that to EC2. I started to put together a script that would make it easier but after writing about [converting an install from Virtualbox to EC2](http://www.ioncannon.net/system-administration/1246/converting-from-virtualbox-or-vmware-to-ec2-now-easier-than-ever/) I never got back to it.
    – I don’t see anything that should keep you from using the script on an EBS backed instance. You could make it configurable by adding a volume id at the top and then using that instead of the loopback if it is set. You may want to take a look at my script that will [build CentOS 5.5 for EC2 using the stock kernel](http://www.ioncannon.net/system-administration/1205/installing-cent-os-5-5-on-ec2-with-the-cent-os-5-5-kernel/) Be ware that the kernel has changed and I haven’t updated the script, it should be a decent starting point if you want to use EBS and also to see how to boot with pv-grub.

    If I had time to go back and fiddle with it again I would probably try going the Vagrant route. I liked the idea of being able to build something locally and just export it.

  4. Matthew Kocher says:

    I also like the idea of a local VM being transfered, but I think it’s probably better discipline to develop chef recipes on a local image and run them on a parallel cloud instance.

    I’d come across your post on pv-grub and had it bookmarked. I’m not sure I’ll takle on pv-grub, but if I do it’ll be the first place I look. I’m a little hesitant about kernel upgrades breaking something with a pv-grub install and not having console access, but that probably stems from my lack of understanding about how it all works and superstition I’ve always had around kernel upgrades.

Post a Comment

Your Information (Name required. Email address will not be displayed with comment.)

* Copy This Password *

* Type Or Paste Password Here *