Close
Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

LABS
CI dot Pivotal Labs dot com

At Pivotal Labs we take Continuous Integration (CI) seriously. Every project has a dedicated machine that serves as a CI environment. Each checkin on the project causes a build to be kicked off. A “build” means checking out the code from scratch and running of all the project’s tests, which, for a Rails project, means unit and functional tests, JavaScript tests and Selenium tests. For the JavaScript and Selenium tests, we run multiple browsers on multiple OS’s (e.g. IE 7 on Windows XP, FF 3 on OS X, etc).

We consider it critically important to keep each project’s build green (i.e. successful) at all times. A build is the heartbeat of the project: if it’s green, everything is healthy; if it turns red (i.e. fails), the team is encouraged to jump on the problem and get it back to green right away. We want red builds to go away quickly; the longer a build stays red, the longer it takes to track down the problem and the more likely it is that additional tests will be broken (the “broken windows effect”).

In order to facilitate this level of discipline, we’ve learned over the years that making the status of our CI environments obvious and visible to the team is critical. If a team isn’t acutely aware of the status of its build, it’s unlikely that a red build will get noticed and fixed quickly. You can have the CI server email the team, but that doesn’t work very well when the whole team is pairing all day: it might be a few hours before someone notices the email. You can install plugins in your browser or system tray that show build status, which helps, but still, they’re not always obvious enough. The best way we’ve found to keep the team informed is to display the status of the build high on a wall near the team as a big red or green indicator. That way, even when you’re busy coding, it’s easy to notice the build going red. These days we use 2 wide screen TVs, positioned in the office so that one is easily seen from any developer station.

When there’s only a single project going on, we’ve found that a screen that’s simply all red or all green is effective. At Pivotal Labs, though, we have many projects going on at once. Rather than putting numerous TVs up on the wall, we’ve created an application that aggregates each project’s CI status into one page. It’s only visible internally, of course. It displays the build status of each of our projects – all the client projects, internal projects, and open source projects that Pivotal Labs is involved with.

Recently we decided to bring up an external instance of the aggregator that shows the status of our open source projects. We’ve also pulled in the status of some of the open source projects that Pivotal depends on (e.g. Rails, CCrb). It’s available at ci.pivotallabs.com. The idea is to provide the same level of visibility into build status for open source developers, or teams that rely on their products, as we have internally at Pivotal Labs. Feel free to display the page on a monitor/TV/projector in your office! It refreshes itself every 30 seconds.
If you have an open source project and you’d like us to run your build and display its status, or if you already have a build and you’d like us to add it to the page, there’s no charge – just let us know (email contact@pivotallabs.com).

Comments
  1. Edward Hieatt says:

    Hah, yes, those are cool. We’ve had some success in the past also with the [Ambient Orb](http://www.ambientdevices.com/cat/orb/orborder.html), and I’ve seen people using lava lamps, Christmas lights, etc. Whatever is super-visible, impossible to ignore, and updates as soon as the build fails is great.

  2. Josh Knowles says:

    I’m curious as to if you have you experimented with distributing this at all? We’re seeing build times in the 45 minute range (unit, integration, selenium, screw-unit) which is obviously too long of a feedback loop. Do you feel a similar pain?

  3. Joe Moore says:

    I want to get a [Chumby](http://www.chumby.com/) and have it cycle through the builds.

  4. Bruno Michel says:

    Great app. I’d like to have same for our projects. Do you have any plan to open-soure it ?

  5. Sam says:

    Maybe I missed it but what CI tool do you use at Pivotal for your Rails projects? CruiseControl?

  6. Chad Woolley says:

    @Sam – Yes, we use cruisecontrol.rb (you can see by clicking through the “visit” link for each project hosted on our server.

    I’ve looked at Integrity some, but the main thing it doesn’t offer is polling. This is an important feature, both to monitor repositories we don’t own and can’t set up commit hooks, and to ensure that builds still happen even if the CI server happens to be down when the commit hook fires.

  7. Edward Hieatt says:

    @Bruno Michel – We don’t have plans to open-source it right now. The current goal is to put it out there to help the community get visibility into the status of open-source projects’ builds. If you have an open-source project that you’d like us to add, do get in touch.

  8. Chad Woolley says:

    @Josh

    About the distributed builds – I’ve thought about that a lot. I think
    we don’t have issues at Pivotal mainly because:

    * We have shorter-term projects with fast builds
    * We work to keep our builds fast
    * We have small teams, so slow feedback isn’t too bad, we just deal
    with any redness and communicate the workarounds. On larger teams,
    with more integration (and communication) issues, redness is more of a
    problem.

Post a Comment

Your Information (Name required. Email address will not be displayed with comment.)

* Copy This Password *

* Type Or Paste Password Here *