Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

One of my roles in my team is the build person. I am responsible for maintaining/updating our build scripts and making sure we are building 'smoothly' on the continuous integration server. I usually do not mind this job, though often it feels like I am constantly babysitting the CI server.

This job can be annoying at times because if the build breaks I have to drop the story I am working on and investigate the build failure. Build failures happen daily on our team. Sometimes developers simply do not build locally before committing so tests fail on the CI server. In this situation I like to quickly get to the person who had the 'bad commit' so the build does not stay broken too long. Sometimes (a lot less frequently) a strange condition exists on the CI server that needs to be debugged.

I know that many mature teams use Continuous Integration but there is not a lot of material out there about good practices.

Do my problems point out that our continuous integration is not very mature or is this simply part of the job?

What are some good practices to follow? What are the characteristics of mature continuous integration?


Instead of answering some comments I am going to make an update instead. We have a single, simple command that does exactly what the build server will do when building the app. It will compile, run all unit/integration and some quick UI based tests.

Reading everyone's answers, it feels we might have two major problems.

  1. CI Server not complaining loud enough when a build fails.
  2. Developers do not feel like its everyone's responsibility to make sure their commit goes through successfully.

What makes things harder in my team is that we have a large team (10+ developers) AND we have a couple of off-shore team members committing when we are not even at work. Because the team is large and we established that frequent, small commits are preferred, we sometimes have really a lot of activity in a day.

share|improve this question
Wow, I've heard of developers not testing code on their local machine before committing, but not building it? That's practically criminal. –  Aaronaught Jun 9 '11 at 1:30
@Aaronaught I'd guess @c_maker actually meant "not running the full suite of tests before committing code" –  Alison Jun 9 '11 at 9:22
@Alison: Maybe that's so, although "breaking the build" to me means committing code that doesn't build. A failed test is a much less critical issue; it's usually not going to prevent other developers from getting their work done. –  Aaronaught Jun 9 '11 at 13:55
@Aaronaught: I'd be curious about the build speed. If a full build takes 20 minutes or more, I can definitely see people testing that their piece of of it compiled and passed tests but not doing a full build. You'll usually get away with that shortcut, but sometimes it will bite you. I've seen this exact mistake made at Google by very, very competent developers. –  btilly Jun 9 '11 at 14:28
@Aaronaught personally I'd see it as the opposite - code that compiles might still fail automated testing and therefore "break the build". –  Alison Jun 10 '11 at 10:07

9 Answers 9

up vote 26 down vote accepted

First and foremost: each person is responsible for the build process. It sounds like members in your team are not mature... No one gets away with writing code and fobbing it off to the CI server hoping that it works. Before committing code, it should be tested on their local machine. You should be sure that the code you're checking in isn't going to break the build. Of course, there are cases when the build break unintentionally (e.g. if a config file has been changed or sloppy commit was inadvertently made).

Most CI servers (I've only used Hudson) will send an automated email detailing the commits made that caused the build to break. The only part of your role is to stand behind them looking tough until the suspect fixes whatever they broke.

share|improve this answer
What happens if its late in the day and they leave work? Should there be a rule that you cannot commit unless you can make sure it resulted a successful build? –  c_maker Jun 9 '11 at 0:52
I agree with @dietbuddha. Don't strain yourself, just roll back the commit and send the offender a stern warning saying that if they ever do that again they will be in Big Trouble. –  Aaronaught Jun 9 '11 at 1:32
I am afraid that threatening will encourage large overarching commits instead of small focused ones which are preferred. –  c_maker Jun 9 '11 at 1:41
Anyone who breaks the build is to get coffee / cake / whatever your team's preferred candy for everybody. Or get coffee for everybody all day long, or... There are many measures you can come up with that make breaking the build unwelcome, while not being so threatening as to get people to avoid submitting. The latter can also be somewhat addressed by requiring everybody to at least submit their changes once a week. (Once a day is too often when working on something more substantial.) –  Marjan Venema Jun 9 '11 at 6:12
Who cares about who breaks the build, as long as they FIX it? –  user1249 Jun 9 '11 at 14:00

Your team got one thing wrong:

Being responsible for the build server is not the same as being responsible for the build.

It is the responsibility of the person checking in code to make it "work" (for some value of work). The sole reason for having a build server is to catch oversights in this process. Any build server worth its salt can notify the person(s) who have checked in code since the last build (as well as anyone else who is interested) with the following information:

  • Build broke!
  • What went wrong when building!
  • What has changed since the last build!

This very frequently happen by email. It is important that this happens quite quickly after each check-in.

The person can then see what went wrong, and fix it, and the build server then notify anyone interested that the build is back to normal. This should happen all by itself without any involvement of others but the check-in culprits.

So to answer your question: A mature CI environment do NOT require the involvement of a janitor in its normal operation.

Also, if "strange conditions exists" happen very frequently then find out what cause it, and make the system more robust.

share|improve this answer

Change the process. If a commit breaks the build, automatically rollback that commit and notify the developer who broke it. It's silly to allow an error by one team member to slow down the rest of the team. Or instead of having integration builds done automatically, have the developers check out the integration machine, and if the build succeeds, then they can commit. Continuous integration does not mean "check in whatever junk you like and someone will fix it for you".

The "golden branch" strategy doesn't work unless there is a gatekeeper for the golden branch.

A DVCS like Git might help; instead of committing, the developer can just submit a changeset for integration to the CI server, and the server can then attempt to integrate the changeset. If the integration succeeds, the changeset is merged, and if not it is rejected.

share|improve this answer

First off, developers should not be breaking builds regularly - they should be building and running the tests locally before they commit to the CI branch. It should be a mark of shame to break the build, and it's important to enforce that. I've done it through posting statistics, and I've seen other teams have a "build bag" where you put in a dollar every time you break the build. At the end of the project that money goes toward beer.

If shame/personal pride doesn't work, you may have to move to heavier stuff (e.g. threatening termination). Breaking the build before leaving for the day should be a major offense. And every developer should have a build status notification on their desktop. The best part of all this is that it encourages smaller commits, which are preferred anyway.

That said, the build will sometimes break (CI configuration reasons, for instance). And sometimes people will screw up and leave for the day with the build broken. That's why you should aim for a process which allows for quick and easy rollbacks to known good versions. If you can always roll back to the last good build (and have the rolled-back version get deployed to all the necessary places), then in the worst-case scenario of a broken build where the culprit has left for the evening you can roll back to the last good version and yell at him/her in the morning.

I can't recommend the Continuous Delivery book enough. If you're looking for a guide on how to mature your CI process, give it a try.

share|improve this answer
Agreed about breaking the build before leaving. You commit a change, you wait for the build to finish so you know it worked. It didn't work? Rollback your change or fix it, before you leave. Don't want to do that? Don't commit changes last thing in the day. –  Carson63000 Jun 9 '11 at 6:32
"should" is nice but any human factor is a non-zero chance of happening. If the build is crucial then have one or more staging build servers. –  user1249 Jun 30 '11 at 12:14

One problem I often see is that developers cannot perform a local build that has exactly the same steps as a CI build. That is, the CI server is configured to include extra steps such as unit/integration tests, coverage, etc that cannot be performed locally. Inevitably, developers will be bitten by one of the steps that they cannot perform locally and will begin to question why they're bothering to build a release locally at all prior to check-in.

I keep my entire build self-contained and have the CI server simply kick off a release build with no extraneous configuration/steps defined. Developers can run a release build locally prior to check-in which includes all the steps that will be performed by the CI build and be far more confident that nothing will break when they do check-in.

Added benefits to this approach include:

  • it's easy to switch between CI servers because you've not invested a bunch of time configuring extraneous steps
  • all build-time tooling is under source control, which means all developers are using exactly the same tool set to build your system
  • in addition to the above point, it is simple to alter build tooling in a controlled manner

PS. the whole build babysitter concept is ridiculous, but others have covered that.

share|improve this answer
amen. just because something can be done, doesn't mean it always should be done. –  gbjbaanb Jun 9 '11 at 12:00

As Jonathan Khoo said you should all be responsible for the build server and the build script. There are three reasons:

  1. At the moment you have a case of a "Bus of 1" which means that if you got run over by a bus then the whole knowledge of the build server and build scripts is lost.
  2. The scripts that have been written (rightly or wrongly) by you have only had your input. Just like any code the more people that are involved the broader the knowledge base that can be applied to it.
  3. Finally when something goes wrong only you are feeling the pain. Pain is a good thing but not when it is isolated. At the moment you are dealing with the pain but if everyone else was having the pain then you would find they would eventually be more stringent in testing code before committing.

I am very involved with CI myself and fall into the trap of being the one that maintains the scripts but here are a couple of things you can do to mitigate that.

  1. Build scripts should not just run on the CI servers but on local development machines as well. They should produce the same outputs, run the same tests and fail for the same reasons. This allows developers to run the script before committing their code.
  2. If any one does break the build make it public via the use of task tray popups, emails, flashing lights, noises etc. It serves to not only embarrass the team members but it provides an opportunity for everyone else in the team to jump up and assist.
  3. For a while avoid fixing the build. Get someone else to do it. If no one else jumps up wait for bad things to happen and use it as a learning point for the whole team to understand why the CI server is important.
  4. Try and keep your build server and development machines as devoid of installed third party components as much as possible, especially keep the GAC clean. Rely on third party components being in the projects library folder. This helps identify missing components more quickly.
share|improve this answer
I strongly disagree with embarrasing anybody. Would you like your compiler to flash an alarm when you have a syntax error? –  user1249 Jun 30 '11 at 12:17
@Thorbjørn This is a CI server not your local development box. The point is the team should do everything in their power to prevent checking in code that breaks the build. Hopefully people work in a fun friendly environment and the embarrassment I am talking about is not mean spirited but it makes people think next time before they commit. But we do have a funny sound that plays when the build server breaks. –  Bronumski Jun 30 '11 at 18:32
I still disagree. A building server is just a building server and everybody can make mistakes. Just notify the culprit and let him fix it. If he doesn't fix it, then we can begin considering whether anybody else should know. –  user1249 Jun 30 '11 at 19:28
@Thorbjørn Perfectly entitled to disagree and disagreement to certain extent is good as it allows us to discuss different ideas. Looking forward to disagreeing with you again, now I must get back to belittling junior developers :) –  Bronumski Jul 1 '11 at 10:28

I heard about a thing Microsoft (maybe?) do, which is to have the role of build babysitter move around the team. The way they do this is that when someone breaks the build (which should probably include checking in something which fails its tests), they assume the role. This makes people responsible for the consequences of their actions in a very direct way. And given that it's a somewhat annoying job to do, it encourages them not to break the build again.

The person who is currently responsible for the build could have a special hat. There could be a ceremony for handing it over.

Note that as Thorbjørn says, being responsible for the build is not the same as being responsible for the build server. Responsibility for the server could rest permanently with one or more more infrastructurally inclined members of the team while responsibility for the build moves around.

Now, details of process aside, i am going to join the chorus of people expressing dismay at developers checking in without having run a build and test. Unacceptable!

If you have some members of the team who are more likely to break the build (and based on my own experience, i'm mostly thinking of the members in another country), and if you're using some nice modern source control like Mercurial or Git, you could have them check in to a different branch to the rest of the team, run a separate CI process on that, and automatically merge changes from that branch to the trunk after a successful build (note that you'd have to run a second build and test after the merge before checking the merge in!). Since automatic merging is not always successful, this would eventually leave the branch requiring manual attention, though, which could be a real pain. Might be less painful than having broken code checked in for the rest of the team, though.

share|improve this answer

One strategy is to use lots of small branches for lots of small projects. Then when someone breaks the build, they are just breaking the build for themselves. So they get the annoyed email from the build server and it is up to them to worry about it.

Another is to improve people's responsibility level. For example if you use something like Rietveld then people can't commit without passing peer review. (The process in practice is much lighter weight than you think it is. But it does force people to "pipeline" and work on multiple things at once.) Preserving the build is the responsibility of both the committer and reviewers. If anyone is either regularly breaking the build or approving things that break the build, then don't let them give final approvals for commits. Combine with a process where anyone can easily roll back any change, and the build won't break as often, and won't stay broken once the change is made.

share|improve this answer
The 'lots of small branches' does not work well when you have one large app that everyone contributes to. –  c_maker Jun 9 '11 at 11:11
no, it does work well. However you shift the pain from development time to merge time. If you merge small work packages regularly, then this doesn't hurt much at all. –  gbjbaanb Jun 9 '11 at 12:14
@gbjbaanb: Right, that's why I specified small branches and small projects. As a further bonus, if the main build is broken for an hour, the odds are that other people are likely to be able to continue working because their builds haven't been broken. –  btilly Jun 9 '11 at 14:21
@c_maker: Every strategy comes with a set of trade-offs, and none is right for all situations. However both that I gave you are being used right now with considerable success in multiple organizations. –  btilly Jun 9 '11 at 14:22

I am going to break with the chorus here, and say that actually, breaking the build occasionally is not such a bad thing, as it shows that actual work is being done. Yes, developers should build and test before they commit, but the main burden of testing should be borne by the development automation tools, including the continuous integration server. The tools are there to be used, and unless the build breaks now and again, then it is not clear that you are pushing as hard as you could. I do, however, think that the build should never ever remain broken for any significant length of time, and would even favour automatic rollbacks or a multi stage commit process if that helps support the twin objectives of quick feedback from centralised, automated test facilities, plus a "green" trunk.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.