up vote 82 down vote favorite
4
share [g+] share [fb]

I'm sitting here frustrated that a team member broke the build and then goes on a long-weekend vacation. What happens on other teams when someone breaks the build? What would you consider a fair punishment?

link|improve this question
3  
How does it always happen that people break the build right before they leave the office? I've never seen anyone break it in the middle of the day when you can actually walk over there right away to talk to them about it. – Jesse Taber Mar 31 '09 at 21:27
4  
Whoah - victimization isn't going to help that programmers' morale. A quiet word from the team lead is more appropriate. – JBRWilkinson Feb 18 '10 at 14:53
2  
One place I worked, our lead PhD was going on a three week vacation. His last check-in comment on the Friday night was, "Does this work for anyone else?". Short answer: no. – Eddie Parker Dec 9 '10 at 19:12
show 4 more comments
feedback

migrated from stackoverflow.com Jul 6 '11 at 23:38

This question came from our site for professional and enthusiast programmers.

closed as not constructive by Mark Trapp Jul 7 '11 at 0:05

This question is not a good fit to our Q&A format. We expect answers to generally involve facts, references, or specific expertise; this question will likely solicit opinion, debate, arguments, polling, or extended discussion. See the FAQ.

40 Answers

1 2

My experience with build breaks has been that the guys checking in the most breaks are often the ones doing the most work.

Yes, there are sloppy developers checking in crap. There are also some geniuses that break on one platform or another. When the average developer breaks the build in 1% of their check-ins, and the geniuses break the build in 0.1% of their checkins, the geniuses still break the build more.

Sometimes, you need to know why. Is there a cultural or process reason? I know that in our case, those "sweeping changes" alluded to above were part of the problem. If you need to touch 120 files once in a while (yes, I've had a delivery in clearcase within the last year with ~120 elements in it, and, yes, it almost broke the build, but I managed to catch the files I missed before the next build - this time), you need to either expect build breaks or find a procedural improvement that can catch them faster/easier (continuous integration, for example, probably would have been sufficient). Ostracising your star developers who are the only ones gutsy enough to make sweeping changes probably isn't the right answer.

link|improve this answer
8  
I love this answer! – Wayne Koorts Aug 13 '09 at 1:56
1  
I agree with this. When I started my development career, I assumed that breaking the build was a cardinal sin. But everybody does it now and then, me included. It's not the end of the world. In many cases you can see how to fix it yourself when you discover that someone else broke it. If not, go talk to that person and find out how to fix it. There's no point in getting adversarial over something that happens every so often. I'm working in a place where one guy is a real stickler for this, and all it does is create workplace tension for no good reason. – Kyralessa Mar 5 '10 at 15:02
feedback

We have a bee outfit, consisting of springy antenna, wings, and a wand.

This is an upgrade from our previous rabbit outfit :)

You have to wear it until the end of the day (including when going out to get lunch, if you break the build in the morning).

link|improve this answer
20  
I don't think embarrassing people is a good strategy. – David G Oct 17 '08 at 13:57
12  
One problem with trying to embarrass people is that not everyone would be embarrassed by that. The last thing you want is someone breaking the build so he can pick up pizza dressed as a bee. – Dave DuPlantis Oct 17 '08 at 14:12
2  
It's not really that embarrassing in our environment (it might be in others but we're very informal and casual). It's just intended to be a bit of fun, which is how everyone takes it. – Greg Beech Oct 19 '08 at 15:35
5  
To protect the innocent? But he's guilty! – Germán Nov 14 '08 at 2:19
2  
I see no picture or link? – Oorang Nov 19 '09 at 4:44
show 5 more comments
feedback

Take No Prisoners

I maintain a nerf arsenal.

Revert the check-in and aim for the face.

link|improve this answer
8  
Hate check-in reverters :) Give me a moment to fix it, will you! It's a different thing if I'm away, but if you just need to get something to build while I'm fixing it then for pete's sake just leave your change in your workspace. – romkyns May 10 '09 at 14:41
2  
+1 for "aim for the face" – iandisme Jan 19 '10 at 18:27
show 2 more comments
feedback

$1 in a bucket that is saved for release celebration, beers, etc...

link|improve this answer
2  
I like it, contribute to group morale :) – JettGeek Oct 24 '08 at 12:44
show 1 more comment
feedback

You Broke the Build

from YBTB

link|improve this answer
1  
I am laughing so hard right now ;-) – matt_dev Apr 29 '09 at 23:46
show 1 more comment
feedback

None. Encourage programmers not to be scared. It is when they break but not fix the build, then may {insert choice higher being} have mercy on their soul.

link|improve this answer
2  
I worked on an enormous project that took nearly 24 hours to build, so breaking it meant a lot of time lost. 2000 people worked on that project, and many were waiting for a good build to come out, so that made it all the more expensive. Don't break the build. – Jay Bazuzi Oct 27 '09 at 0:35
14  
Programmers scared of breaking builds will be scared of making changes, and therefore scared of working on the code. Not productive in the least. Revisit program without fear. Sounds like the project was a little too monolithic and poorly architected. 2000 people shouldn't be responsible for or dependent on a single build. Individual components with stable version would have been a better choice. Alternatively implement some measures to detect build break changes before they get checked into source control. – vfilby Oct 29 '09 at 18:28
1  
imho, continuous integration that notifies everyone of build-breaking makes breaking the build no big deal at all because you'll fix it. You lose like 2 seconds of time. So CI makes punishment obsolete in my experience. – apollodude217 Aug 6 '10 at 21:45
feedback

USB foam dart gun.

alt text
It is the geek way!

link|improve this answer
5  
could could probably code something watch for broken builds and aim and fire it at the person who broke it automatically. – Scott Cowan Oct 17 '08 at 9:34
1  
Yes; scott, that's exactly what I wanted to implement here. We didn't get around to it though. Definitely the most awesome solution. – Noon Silk Aug 13 '09 at 1:47
show 4 more comments
feedback

A guy I worked with, in a past job, worked in a place where if you broke the build, you got to have, all to yourself, in your very own cubicle:

A giant cardboard cutout of Jar-Jar Binks.

link|improve this answer
show 1 more comment
feedback

We have a CI system that will send an email and let people know that the build has broken (tests are failing or whatever). It saves us a lot of time. But seriously...

It's just not a big deal.

Why? We use git, so one of two things happens:

  1. The rest of us continue to work without those newest changes until the guy who pushed bad code fixes his code.
  2. One of us throws away his changes and resets the main branch and lets him rework it and try it again.

The biggest problem I've run into is having people who don't check in changes. I'd rather have pushed changes and build an after-the-fact branch to stash stuff away while it gets fixed than to have people who sit on changes for months to hide stuff.

link|improve this answer
feedback

We sometimes use fingerpointing:

alt text

(from codesmack)

link|improve this answer
feedback

Is death a bit harsh? :)

link|improve this answer
27  
depends on the proximity to the launch date. – Rob Allen Oct 17 '08 at 12:59
show 2 more comments
feedback

Yes, breaking the build and going on vacation (or even home for the night) is bad. I'm talking about the occasional break that gets fixed right away, which is what most of the replies in this thread seem to be referring to.

On our team we have no such "punishment policy" no matter how unwritten or informal. I see no need, because we all know that breaking the build keeps the others on the team from updating or checking in (and that's all it does). That's plenty of motivation for us to be careful.

So where is the need for punishment? To me it smells of 8 year-olds on a playground pointing at each other singing: "nyaa, nyaa, nyaa-nyaa, nyaa... you broke the bui--ild."

link|improve this answer
2  
I agree. unprofessional. – Tim Dec 17 '08 at 23:10
2  
I once worked on a team where one developer always broke the build. EVERY check in. And he was a senior developer. Because there was no policy, there was no discipline. And discipline was most certainly needed. – salt.racer Feb 6 '09 at 19:01
show 1 more comment
feedback

Revert the offending commit.

Having to merge changes when they return is usually punishment enough.

link|improve this answer
show 1 more comment
feedback

We have a rubber chicken that hangs outside the cubicle of the person who most recently broke the build.

link|improve this answer
feedback

It's my understanding that even "fun" broke-the-build markers are considered to be less than optimal because they discourage frequent check-ins, which prevents problems from being caught earlier while they're smaller, not to mention increasing the difficulties of merging.

On the other hand, I did like the dollar into the beer fund suggestion.

link|improve this answer
feedback

The person breaking the build becomes the build administrator

link|improve this answer
2  
What? You're not using automated builds? – Kelly French Jul 21 '09 at 19:54
feedback

I "suggest" donations to charity organisations. Works great for people coming late for meetings, too.

link|improve this answer
1  
He makes donation offers they can't refuse ;-) – ypnos Nov 30 '08 at 9:29
show 5 more comments
feedback

Have the build server automatically start playing country western music over a loud speaker which cannot be shut off until the build is fixed...

link|improve this answer
show 1 more comment
feedback

We have just started throwing shoes at the offender.

link|improve this answer
show 2 more comments
feedback

About five years ago things got so bad that I got very "librarian" about it, and started a spreadsheet with WALL OF SHAME in large letters at the top and entered every breakage into it. Each row had the name of the person responsible, the reason why it occurred, and an explanation of the steps taken to ensure it didn't happen again. It was surprisingly effective, but excruciating to have to administer. And people complained about it. So I told them "This is an optional scheme - if you don't want to take part, just never break the build."

Now we have continuous integration, the full nightly build only breaks extremely rarely - if there's a powercut or a disk goes pop. Though the old timers still speak in hushed tones of the dark days when we had THE WALL.

link|improve this answer
feedback

What about punishment ?

alt text

(from codesmack)

link|improve this answer
4  
for (int i = 0; i < 1000; i++) Console.WriteLine("I will not break the build."); – Martin Aug 13 '09 at 1:59
feedback

The person who broke the build is now responsible for testing the build.

(with the benefit that that programmer will be incentivized to catch the next offender)

link|improve this answer
feedback

You seem to practice agile, you better check it out and fix it, NOW!!! :)

Update: Punishment for the guy who broke the build, fix the next broken build ;)

link|improve this answer
feedback

I know one team that made the developer who broke the build wear a dress for a day.

link|improve this answer
1  
I know developers that wear dresses on regular days... do you have any women at your company? – emddudley Oct 5 '10 at 14:54
show 1 more comment
feedback

The best way how to deal with this is to create automated checkin policy, which prevents committing (Checkin in) broken builds, or at least reports the broken build very soon after checked in.

One possible implementation for this can be:

  • have a dedicated build server
  • have a set of automated functionality tests (executed on the build server) which check the build can successfully built, can be run and performs basic functionality OK
link|improve this answer
feedback

I'm strongly disagree with any punishment. A simple friendly and private note from the others is enough.

Punishing others could damage morale. Maybe it sound fun, but could hurt someone feeling and to avoid mistakes in the future it minimize his development efforts (loosing creativity).

For me CI means to realize mistakes as soon as possible and not, developing without mistakes. Every human makes mistake. Any kind of solution, which based on: "that this is so simple, no one ever going to mess up" is going to fail.

As always if the build can be broken than it is architectural mistake. Saying don't break the build is no different as the note: Please use large capitals in this text box or we cannot parse your order and can seriously damage our servers. Or with other words. How do you going to develop user friendly stable applications, if the tools and processes what you use and develop for your self aren't satisfying these criteria.

If you use any kind of script or build tool you can create a process, where the check-in procedure involves code update, build, QA and check-in.

However this could only lower the number of broken builds. As for example special file extensions wasn't checked in into the last step could affect the build.

The solution is double QA. When you fix someone bike, you first test it then the customer test it as well before accepting it.

If you use transactional SCM, than you can maintain two branch. One is where everyone checks in the code. On that branch a build tool runs, which builds and QA against every transaction (check ins). If it succeed it pass the transaction to the main branch. If not it throws away that transaction. (Also notify the checker :-) ).

With this the developers will have a branch, where they always have stable code. And a branch where they can check-in without risk.

But have in mind the effort and benefits. If you are working locally with 2 - 5 person, spending to much time creating this environment maybe not worth the effort. A simple please fix it could be enough. On larger teams working in different location could be handy.

Also this won't answer code QA issues on some level, recognize integration issues on early level and point out invalid business logic.

link|improve this answer
show 2 more comments
feedback

"He whose release doth cause offence shall bake a cake in recompense"

Serves us very well, and we get some great cakes out of it! Helps people take responsibility but without feeling beat up about it.

link|improve this answer
3  
Problem is of course whether the person's cake-baking skills is on pair with their build-breaking skills... – Andreas Magnusson Oct 17 '08 at 11:00
1  
"Four large eggs. One cup semi-sweet chocolate chips. 3/4 cups butter or margarine. Fish shaped crackers. Fish shaped solid waste. Alpha resins. Unsaturated polyester resin. 2/3 cups granulated rhubarb. One tablespoon all-purpose rhubarb. One cross borehole electro-magnetic imaging rhubarb." -GLaDOS – Erik Forbes Oct 20 '08 at 23:15
show 2 more comments
feedback

I guess someone could have to write "I will not break the build" 1000 times, but then again, programmers would just do:

for (int i=0; i<1000; i++) System.Console.WriteLine("I will not break the build");

but in the projects I've use continuous integration on, we used CCTray, and it popped up with a message to all developers on the project within a minute of checking in broken code. All the programmers would just convene at the desk of the culprit shaking their fists and suggesting that it be rolled back or fixed code be submitted. It usually wasn't an issue.

link|improve this answer
feedback

A small, stuffed Cthulhu in a coffee cup that sits on the culprit's desk, hight "Works". The developers always say "Works on my machine", so we made it so!

link|improve this answer
feedback

One placed I worked, we had a broom. Its use came from checking in "sweeping changes" which usually broke something or other. So we appropriated a broom and it eventually changed into an I-broke-the-build indicator.

link|improve this answer
feedback
1 2

Not the answer you're looking for? Browse other questions tagged or ask your own question.