The RunCodeRun Blog

It's Okay to Break the Build; It's Not Okay to Forget About It

When should your Continuous Integration service notify you about the results of a build?

When we originally implemented email notifications for RunCodeRun, we thought to ourselves, “We get too much email as it is. Let’s not send an email after every build; let’s just send notifications when the build status changes. You know, when the build breaks, we’ll obviously want to know. And when when it goes back to green, we’ll want to know that as well.”

That seemed like a reasonable approach at the time. Then along came reality. You see, we do get too much email as it is. And as a result, sometimes when the build goes red, that status change notification might get lost in the noise. A few days go by – or even worse, a few weeks go by – and all the while, the dev team is committing away in WorksOnMyBox Land, none the wiser. Maybe the team introduced a dependency that happens to reside on the development machines, but they forgot to declare that dependency in the project. Maybe they unknowingly implemented a feature that only works on their development machine’s operating system or architecture.

If your deployment environment looks even the slightest bit different than your development environment (and who’s doesn’t?), these are things that you want to know about. And if you make another change to the code, and these problems still exist, you want to know.

With the changes we pushed out today, RunCodeRun will now remind you of your ill-fated builds after each build failure. And then when you fix the build, RunCodeRun will notify that you’re back to green.

If you’re not getting any build notifications at all, you can enable them in your account preferences. (You may also want to double check your spam folders, and be sure that you have bot@runcoderun.com on your email whitelist.)

Remember: it’s okay to break the build, but it’s not okay to forget about it.

Building Java Projects with RunCodeRun.com (or JRuby, or Scala, or Clojure...)

RunCodeRun is known for free open source builds, but did you know it builds Java as well? RunCodeRun still assumes that you are using Rake instead of Ant, but thats easily remedied with a Rakefile that forks an Ant build (note that there are many resources for building Java with Rake or other JRuby based tools as well). Since Ant isn’t available on RunCodeRun by default, you’ll need to include a couple of Ant jars in your source code.

Setup Ant

First off, add the following two jars to a “lib” or “vendor” directory: ant.jar and ant-launcher.jar. (I’m assuming you’re using a recent version of Ant.)

Then in your build.xml file, define (or find) the target that will build your code and run your tests. We’ll need this target name in the next step.

Setup Rake

Now, create a file called Rakefile in the root directory of your project. Put the following in your file:

# Ruby Rake file to allow automatic build testing on RunCodeRun.com
task :default => [:test]
task :test do
  classpath = [File.join(".", "lib", "ant.jar"), File.join(".", "lib",
"ant-launcher.jar")].join(File::PATH_SEPARATOR)
  exec "java -cp #{classpath} org.apache.tools.ant.Main -emacs all"
end

Note this snippet assumes that you’ve put the jar files in a lib directory and that you’re using an Ant target called all. The -emacs option to Ant doesn’t really have anything to do with Emacs, the editor. It just tells Ant not to print all the “[PROCESS NAME]” line prefixes it normally outputs. If you like those prefixes, just remove the option.

Lastly this assumes that Java is on the command line, and that your project runs on Java 6. Since a lot of our projects happen to use bits of JRuby or Clojure, we tend to stick with the latest updates of Java 6. Your mileage may vary.

Testing from the Command Line

Now you’re ready to test your setup! I recommend creating a failing test in your project and calling ant all (or whatever your default build-and-test target was) to see the failure. Then immediately type the following from your command line: echo $?

This will output the return value of your ant program. It should be non-zero. If it’s zero, something’s wrong with your build, and Ant is not reporting failure correctly. (This would make all of your tests always green, even if they’re broken. Sounds like fun, but it isn’t.) Dig into your Ant build to find out why it’s returning the wrong value. (Hint: if you’re calling a <java fork="true"> tag anywhere, make sure you also have a failonerror=”true” attribute.)

If you have a non-zero value, though, then you’re ready for the next test. Try typing rake to have Rake call your Ant build. Again, you should see a failure and calling echo $? should give you a non-zero value. You’re almost ready!

Now go and fix your failing test and call running rake again. Type echo $?, and if you get zero, then you’re ready to go!

Set up your project on RunCodeRun, push to GitHub, and check your build status. You’re building and testing your Java project on RunCodeRun! For a running example, check out env-js - a JavaScript browser environment that uses Rhino & Java for its tests on RunCodeRun.

Testing against multiple Rails versions

It should be obvious from the fact that we built RunCodeRun that we at Relevance believe strongly in continuous integration. CI is more than just testing—it gives you something qualitatively different, over and above what good testing discipline gives you.

One example of that is what Chris Eppstein (one of the Haml and Sass team) did last week. He’s using RunCodeRun to test the Haml gem and plugin against multiple versions of Rails (including the 2.3 release candidate). Chris’ latest Haml build status page shows the full test suite being run four times before reporting success.

When we run your project’s tests under RunCodeRun, we set the RUN_CODE_RUN environment variable, so your tests can know where they’re running and do something special, if desired. Chris makes use of that in the first part of his Rakefile to set a different default task for RunCodeRun builds:

if ENV["RUN_CODE_RUN"] == "true"
  task :default => :"test:rails_compatibility"
else
  task :default => :test
end

The test:rails_compatibility task runs the entire test suite against several Rails versions, failing if the tests fail against any of those releases:

namespace :test do
  desc "Test all supported versions of rails. This takes a while."
  task :rails_compatibility do
    `rm -rf test/rails`
    puts "Checking out rails. Please wait."
    `git clone git://github.com/rails/rails.git test/rails` rescue nil
    begin
      rails_versions.each do |version|
        Dir.chdir "test/rails" do
          `git checkout #{version}`
        end
        puts "Testing Rails #{version}"
        Rake::Task['test'].reenable
        Rake::Task['test'].execute
      end
    ensure
      `rm -rf test/rails`
    end
  end
end

We’re absolutely delighted; RunCodeRun is perfect for this. As Chris’ comment points out, it’s a time-consuming process, so you don’t want to run all of these tests before every checkin; that would slow your development down. But you do want to learn relatively quickly if you’ve done something that will fail on older Rails versions. It’s an ideal workflow: run the ordinary test task before each checkin, and then have RunCodeRun test against all of the supported Rails versions after each checkin.

We hope to make this even easier in the future … we’ve already got a few ideas. For now, though, Chris’ approach works just fine.

We hope other plugin developers will follow Chris’ example. This makes it easier to support older Rails versions. By testing against the upcoming release candidate, you can ensure that you’ll be able to support the new version right out of the box. By testing against edge, you can get early warning of things that might break in the future.

Email outage - last Thursday until Saturday

Late last week, starting on Thursday, February 5th, we experienced an email outage due to some internal email notifications going haywire. This issue was fixed on Saturday. The outage meant that you may have missed build notifications during this period, so it’s possible your build was fixed or broken without your knowledge during this time. We apologize for any inconvenience this caused!

We’ve removed the offending notification and made some changes to our email system to prevent this from happening again. If you haven’t received notifications after Saturday that you think you should have, please let us know.

It's Okay to Break the Build

“My name is Stuart Halloway, and I believe it is OK to break the build.”

Several developers have recently asked me “How should developers who break the build be punished?” With only that information, my answer is easy: They shouldn’t.

Of course breaking the build is not a good thing, but let me make a couple of hypothetical examples to demonstrate the point.

Example 1. Jane’s Birdhouses (JB)

JB is an ecommerce + social networking site for a brick-and-mortar birdhouse store. Built with Test-Driven Development (TDD), JB has three test suites:

• a few hundred unit tests that execute in about 20 seconds.
• a few dozen integration tests that execute in about 90 seconds.
• a few dozen acceptance tests in Cucumber and Selenium that must be run multiple times against different browser versions. As a result, these “long tests” take about 30 minutes to execute.

Now, let’s consider how the JB build might break due to developer negligence:

• A developer could commit code without bothering to run the unit tests or integration tests locally.
• A developer could commit code knowing that the tests were broken.

Both of these things very bad behavior, and should be condemned by any agile team.

On rare occasions, extenuating circumstances might justify such “negligent” commits, but documenting such corner cases is not my purpose here. More interesting to me are the legitimate, respectable commits that might nevertheless break the build.

For example, what if a developer commits code to JB that passes the unit and integration tests, but breaks the long-running tests? That’s not good, but it may be better than the alternative. If you insist that developers run the “long tests” locally before committing to the build box, then you cannot call yourself agile. Rapid feedback is critical to agility, and taking several 30-minute breaks a day will never meet that standard.

So you should not insist on perfect builds. What should you do if a commit breaks the long tests? Stop what you are doing, figure out why, and fix it. The problem will fall into one of three categories:

1. The problem could have been detected by a fast test, and that fast test was an obvious thing to write. Solution: write the fast test, and beat up the negligent dev.
2. The problem could have been detected by a fast test, but the needed long test was not obvious until the long test found it. No negligence here. Solution: write the fast test.
3. The problem could not have been detected by a fast test, or the cost of writing the fast test is extremely high. This case sucks—but it does justify running the long tests. Fix the problem, and know that similar problems in the future will also break the build.

Another possibly legitimate build-breaker is uncovering a hitherto-unknown dependency in an application’s setup. Similar rules apply:

• If the problem should have been obvious, then shame on the developer. Example: adding a necessary Rubygem on a local box without bothering to add it to the build (ie config.gems in a Rails project)
• If the problem was more subtle, then just fix, learn, and move on. Example: the latest version of support library Foo introduces a new dependency on Bar. The authors of Foo did not document the dependency clearly, and the dev didn’t notice because he already had Bar installed.

Example 2. Jim’s Enterprise App (JEA)

JEA is a much bigger, more serious application than JB. It accesses five separate databases, corresponding to five separate business units. The tests are correspondingly more complex, and the test suite takes four hours to run.

The JEA team is committed to quality. The original application didn’t have unit tests, but the team has worked heroically to add them, and code coverage now stands at 80.1%. Developers are docked on their performance evaluations if they commit code that

• breaks a test
• drops coverage below 80%

How should we interpret this? Despite possible good intentions, the JEA team fails agile at Square 1. First, the developers have to run a four-hour test suite locally before committing. Otherwise, they risk automatic censure for breaking the build.

Second, the coverage ain’t going to go much above 80%, ever. Think about it: Bill writes a new module. When he runs the tests, coverage goes up to 80.2%, but a test somewhere else in the app breaks. If Bill simply deletes the other test, he meets both his objective performance criteria: no broken tests, and coverage stays above 80%. Talk about perverse incentives!

In my experience auditing code, I have seen far more codebases like JEA than like JB. My guess is that most projects never partition their tests into slow and fast, even when the test suite has grown so slow that agile is a dimly-remembered dream. Moreover, should JEA even be a single project? Probably not. Statistically, large projects suck, so if you have any reasonable way to break your project into little ones, you probably should.

Wrapping Up

Is breaking a build bad? Yes, sometimes. Usually, even. But context matters, and teams that ignore context will never be agile.

Disabled projects

You can now disable projects in RunCodeRun. “Disabling” basically means that the project and its history will be maintained, but future builds will not run. Any future pushes to that project will be ignored silently.

We will be adding some more smarts to the project listing soon, and disabled projects will not show up in the default view when those changes are in place. So if you are getting sick of dead projects cluttering up the view, disable them now and they will be filtered out soon.

Post-Receive Changes from the GitHubs

GitHub recently rolled out a change to allow multiple post-receive urls per project. This allows you to integrate with as many services as you like from GitHub. Our instructions for getting started with Github integration have been updated to reflect this change.

More importantly, GitHub seems to have lost many of the pre-existing post-receive urls in the transition. If you’ve noticed that your project is no longer building on RunCodeRun, please go to your github project’s Admin Services tab and add “https://runcoderun.com/github” to your post-receive settings. Then fire off a test hook to get make sure things are working again. We had to reconfigure every project that Relevance has in RunCodeRun, so don’t be surprised if your project has not been pushing to RunCodeRun for the last couple days.

We came, we saw, we Rumbled

RunCodeRun sponsored Rails Rumble this past weekend and offered continuous integration to participating teams. We knew going in that this would be a great way to give a useful service for test-driven teams and learn a lot about how RunCodeRun stands up to load.

Things started well on Friday night and we got some great feedback from the initial teams who signed up. As development heated up on Saturday, we realized that keeping up with the rapid commits was going to be a challenge. Before the Rumble, we’d never had more than 10 builds queue up at any one time. By Saturday night we saw several teams committing every few minutes, and the queue backed up to thirty or forty builds.

We fired up additional builders to run more builds in parallel, and we started pruning old builds that had been queued too long to be useful. On Sunday we automated a job to prune old builds and mark them as “cancelled”, which kept things moving at a reasonable pace until the teams raced to the end on Sunday night.

By the end of the Rumble, about 30 teams had their projects building on RunCodeRun, with around 1,000 successful builds and 800 failing builds in all. The average build time toward the end of the competition was about five minutes.

This was a great experience for us, and we think the service was useful for participating teams despite the backlog. We now have a much better idea of more steps to take to scale and where the pain points are.

The RunCodeRun team is all about releasing early and often and gathering feedback as early as possible. Our thanks go out to the Rails Rumble organizers and to the teams who took RunCodeRun for a spin. We look forward to improving the service based on the lessons we learned at the Rumble. Don’t hesitate to contact us if you have specific feedback, or want to learn more.

The Rails: they be Rumbling with RunCodeRun

The starter pistol went off (metaphorically) and the Rails Rumble is off and running! For the next forty-eight hours developers from all over the country (maybe even the world?) will be vying for that most precious of geek cred — bragging rights from a well-publicized competition. Women will swoon (er maybe not), code will be written, and gods will be created.

If you’re a Rails Rumbler, follow these directions to get your project setup with RunCodeRun. If you start out with a healthy, green build, its that much easier to keep it that way. If you get stuck, grab some help on IRC from the #runcoderun channel on Freenode.

If you’re just along for some voyeuristic pleasure, you can check out all the competitors and their build status on our Rails Rumble project list. Also, be sure to follow runcoderun on Twitter for updates during the competition.

RunCodeRun sponsors Rails Rumble 2008

Rails Rumble is this weekend, and RunCodeRun is going to help out by offering continuous integration to the first 100 teams that are interested. Rails Rumble is my favorite type of competition: the kind where people build real apps and can only win if they actually ship. Each team of up to four developers has one weekend to build a Ruby on Rails app from scratch. Last year over 90 teams participated, and this year well over 200 are expected.

RunCodeRun will have a team of people in the Rails Rumble room at Raleigh Rubycamp to help anyone on-site. You can also find us in #runcoderun on freenode in IRC, and we’ll be closely monitoring tickets at Lighthouse. Watch the FAQ for additional Rails Rumble specific issues as we update it throughout the week.

Hack > Rumble > Ship > Runcoderun > Win