Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

How You Can Learn to Stop Worrying and Love Continous Integration

I just had a discussion with a co-Pivot about the resentment that many teams develop about Continuous Integration – especially when the release process requires a green tag from CI, and a broken build is standing in the way.

Slim Pickens, Dr. Strangelove

As anyone who has worked with me will attest, I’m hardcore on CI and consider any team which leaves a build red for longer than a workday to be sorely lacking in discipline.

OK, OK, there are always extenuating circumstances, but I still believe that most resentment of CI stems from underlying antipatterns and smells, rather than problems with CI itself. For example:

  • “The Customer Has To See This Feature RIGHT NOW”: Frequent releases are a great thing, but if you cannot wait for a green build to deploy, you have some deeper problems. Often, this is because a team doesn’t manage customer expectations well. The customer should understand that CI is a critical part of the Agile process which ensures that only reliable, quality releases get pushed to staging or production. Any problem which is preventing a green build should be fixed before the release is deployed. If the customer is not willing to allow you that time and flexibility, perhaps they are too addicted to new features, and the entire team needs to have a heart-to-heart about Code Debt in the next retrospective.
  • “It Works For Me, But Fails On CI: The important question is which environment is more like production – your development environment or CI? If you are developing on Windows or Mac, and your production box is some other flavor of Xnix, then your CI box should be as close a possible to production. Ideally, you should be able to log on to the CI instance and debug the failing test there. Usually, your CI box is not configured correctly. If it is hard to keep your CI environment in sync with production, then perhaps you should look into automation (because you KNOW you or your sysadmin will probably forget to do the same thing when you push to production, right?). If the problem is that your development environment is not the same as production, and it is a legitimate problem, then CI just saved you some stress on the next deploy.
  • “Intermittent” Failures: Same deal as the prior point. CI runs your tests much more than you do. For web apps, it hopefully runs them in more browsers than you do. In my experience, many “intermittent” bugs are real bugs which are just very hard to isolate. It could be an AJAX bug that only happens when the site is run remotely, not via localhost. It could be a performance problem which only shows up on a slower system, not your fastest-on-the-market dev box. It could be a dependency on an external resource that happens to be unavailable sometimes, such as a web service, remote storage, etc. Again, just being aware of these issues puts you ahead of the game. For browser bugs, dig in and find out WHY it is failing intermittently. It may be a real bug. For intermittent outages of external resources, you may just have to live with it, but you don’t have to live with the intermittent failures in CI. Mock out the resource or disable the tests in the CI environment. Yes, this is OK, especially if you leave them enabled in your development environment. Another option is to automatically repeat these tests a few times with a delay, and only fail the entire build if they fail repeatedly. Big services like Amazon or Google might drop a request occasionally, but still respond to a subsequent request.
  • Slow Test Suites: This is an insidious problem, because once your suite is slow, it is often a monumental effort to make it fast again. It is much better to be proactive, and monitor any slow-running tests like a hawk, relentlessly mocking out slow resources or replacing broad functional tests with faster, more targeted unit tests. You can also always split your tests into different suites, running your fastest tests continuously, and the entire slow deploy-test suite only nightly or periodically. As long as your customer isn’t addicted to immediate features, it should be fine to only deploy from nightly builds.
  • The Failing Test That “Doesn’t Matter”: This is my pet peeve. Whenever I break CI, I fix it ASAP. If I ignore a “minor” broken test, the next thing I check in may be a major FUBAR which gets past my local tests for some reason (see prior points). Some who know me might even say it is LIKELY to be a major FUBAR. The point is, I don’t trust myself or my local box, I trust CI. Now, if ANOTHER developer breaks the build, and tries to tells me they are not going to fix it because it is a “minor” problem, that really chaps my hide. They are ripping huge holes in my nice safety net, forcing me to expend much more time and attention on the tests that I run on my local environment, and causing me more stress and work in general. Stop making excuses, and fix the damn build NOW, or comment out the failing test.

Now, I’m sure that all of the above points can be debated or shown to be inapplicable in a specific situation. Plus, if you are dealing with imperfect CI and development tools (which is always the case), you will have some degree of pain which is directly attributable to CI. It would be great to hear about some of these situations in the comments.

Bottom Line: Integration is always one of the most painful parts of software development. Doing integration with high quality and low risk is even harder. Most developers who have been on a non-Agile project of any significant size have experienced days-long integration hell and ulcer-inducing all-night production deployments. Continuous Integration doesn’t make that pain and stress go away, but it does break it down into small, bite-sized pieces that can be easily handled on a daily basis. All for the low, low cost of being proactive and disciplined, which makes you a better developer anyway.

  1. Don says:

    Anybody who commits failing tests should be fitted with a shock collar. Any time that test fails for another developer, the collar is activated. It could be done, we have the technology.

  2. Chad Woolley says:

    @don: Shock collars. Interesting concept. I wonder if there is a CI plugin for that.

    But seriously, on a small team which has very good communication, I will often rely on CI to run the full suite for me. In fact, if I am the only developer or pair working on the project, I’ll often work this way – especially if the test suite takes a long time and I’m not working on a project on which anybody else has a dependency. I’ll do a few spot checks on specific unit or functional tests that I know I might have broken, and if things seem OK, I’ll check in and keep an eye on CI (like I always do).

    In fact, on a small, close team, this can even be a positive thing. It shows that you all understand the code well, and are confident enough do frequent commits, move on to the next task, let CI run your tests, and fix things quickly or roll them back if they break.

    Yeah, I know there is autotest and other stuff, but I never got into this. It seems to get in the way of a normal TDD flow, especially on a large project, or if there is the possibility of test environment (db) conflicts from autotest and manual TDD.

  3. Al Sargent says:

    Great post! Couple of questions:

    – What CI program do you recommend? We’re trying to pick one…

    – Would you recommend integrating functional tests into a CI build? We’re trying to figure out if this is a good idea.



  4. Chad Woolley says:


    Thanks. There are a lot of CI tools, and more every day (ThoughtWorks just made a big announcement of their new enterprise-targeted product).

    Personally, I am currently partial to [CruiseControl.rb]( (CCRB), because it is simple, easy, open source, written in ruby, hackable, and I know the guys that work on it. Unless you are on huge team and enterprise project, CCRB should give you most of what you need out of CI, which really isn’t too much.

    As for functional tests, yes, I’d definitely recommend integrating them. Automating time-consuming and boring functional tests is a great use of CI. For a simple example on how you can do this with Selenium, see my tutorial: [](

    Good luck, and let me know how it goes!

    — Chad

  5. Don (different to the other don) says:

    I’m having a good time with ruby now, but i was wondering how to setup those long running tests to run every night?
    I followed this railscast

    And i have a task that generates about 100,000 primary records with 1,000,000+ detail records. This is great for testing queries against large datasets.
    But it takes a good 10-20 minutes to generate (the build server is kinda old) plus if one the specs fail, they usually fail by timing out. I also want to expand this to the production size of around 1,000,000 primary records, but this will take too long for during the day!

    I have created an different environment called large_test, with its own database but I would like to know how to
    – set up a separate set of specs that will not run with “rake spec” is run
    – set up a nightly cruise control build that will appear on the CC dashboard in the morning with the (hopefully) good news.

  6. Chad Woolley says:


    There’s a few steps to do what you want:

    1. Make a different spec suite which only runs the specs you want. If you look at how your current spec helper is written, you will probably see somewhere it globs up all *_spec.rb files in your spec folders. To have some that DON’T run as part of that, either put them in a different place, or name them differently. You could have a whole separate slow_specs folder, or name them *_spec_slow.rb. Whatever works. Then, you can make a separate suite and rake task with a different glob which only runs these specs (and possibly your fast ones as well).

    2. Once you have this “slow_specs” rake task, you can make a separate cruise project which only runs this task, with a special name like ‘myproject_slowsuite’. You can do this with a switch in your cruise_config.rb to execute the different task instead of your default task or ‘cruise’ task. This switch can be based off the project name – for example, if a regexp match of the cruise project name matches “slowsuite”, then run your slow task.

    3. To run this slow suite project only nightly, I know there is something in cruise you can override to have a different time-based build trigger instead of the normal commit-based trigger. I’ve done this, but don’t have the code handy.

    So, that’s it. Sorry I’m short on details now – give it a shot and ask for help with help on the ccrb users list if you get stuck – I follow it.

    — Chad

Post a Comment

Your Information (Name required. Email address will not be displayed with comment.)

* Copy This Password *

* Type Or Paste Password Here *