Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Sometimes, people really have a bad day...
I think it's interesting to learn why that day was bad and learn from our mistakes made on that day.

So... What was your worst software development day like?

share|improve this question

closed as primarily opinion-based by Robert Harvey, Thomas Owens Aug 9 '13 at 20:41

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

7  
Having to listen to 3 hrs of managerial BS after having little sleep due to splitting up with a girlfriend. –  Job Jan 10 '11 at 16:15
show 3 more comments

26 Answers

up vote 36 down vote accepted

It was the day I wrote a SQL delete statement to run against a large table and forgot the where clause. Thankfully we had last night's backups handy.

share|improve this answer
2  
Reminds me of the time one of the admins at AQ Worlds renamed his character and forgot the WHERE clause. Suddenly everyone in the DB was named Cap'n Rhubarb! Luckily they were still in the very early beta stages at the time, so they just threw out the DB and restarted the beta. –  Mason Wheeler Sep 10 '10 at 21:16
4  
I've heard of the same thing happening so now I always use transactions when doing this. But now I get caught by forgetting to commit or rollback instead –  Joe Philllips Sep 11 '10 at 16:46
16  
One of the many reasons why I hate SQL. Whoever thought it was a good idea to come up with a syntax where you're only one accidental keystroke away from trashing your data when writing any UPDATE or DELETE statement? –  richeym Sep 14 '10 at 8:12
1  
Ahh - I've done this except with an UPDATE using dummy data on a row I was testing. Staring at several thousand 'Mr Dummy' records doesn't fill you with confidence. –  Jonathon Jan 10 '11 at 16:25
1  
Related from today: serverfault.com/questions/227338/mysql-disaster-recovery –  Mark Henderson Jan 27 '11 at 4:38
show 1 more comment

Not a day: ten of them. I just spent almost two months fixing an algorithm in a library that was causing paradoxical behavior in the output of a simulation. After all that hard work, I re-ran the simulation and was still getting the same wrong results, so for a week and a half I examined every bloody line in my simulation, every day getting no change. So by now I have about nine weeks of disappointed looks from my advisor, who has no advice to offer.

Then I figure out that I'm still using the old library.

share|improve this answer
show 2 more comments

For me those were not days of technical stalls, but the days where things in the air changed.

1st worst day: Having just joined the company, I was shown the code in .NET 1.1 and told we were going to stick with it for a while. At the interview we were talking about 2.0/3.5 and I was supposed to work with that. There's nothing that can ruin a new working relationship right at the start like a good lie...

2nd worst day: We weren't allowed to write stored procedures for the only reason the boss didn't like them.

3rd worst day: After having finally migrated to .NET 3.5 the boss told us to stay clear of that new stuff like LINQ and lambda functions for the only reason it was something new and not time-proven.

It was then when I understood I didn't belong with them.

share|improve this answer
3  
+1 for all of them. Been there before. :-/ –  Ryan Hayes Jan 10 '11 at 16:35
1  
I've run into the "doesn't like stored procedures" before, so annoying, yet so common. –  Bmw Jan 27 '11 at 15:02
1  
And of course, none of the database stuff is in source control, so when some snarky DBA comes along and fucks up the SP you can't roll back to the working one. Grrr. –  Chris Kaminski Feb 7 '11 at 21:51
show 2 more comments

After spending a 100 hour work-week feverishly integrating software in an overcrowded under-ventilated makeshift office above the shop floor where they were assembling our simulator, catching a late flight to D.C., and then working until 6AM stomping out the last bugs, and trying unsuccessfully to figure out why the motion base was pitching backwards instead of forwards whenever you stomped on the brakes.

Luckily, none of the generals we gave demos to on that day seemed to notice the glitch.

2 days later, the misplaced negative sign was obvious.

Lesson Learned: Get Some Sleep.

share|improve this answer
6  
Me and a friend were working on a late assignment for our networking course (which was the third heavy programming assignment we still had 80% to do during finals week), and after being up for about 2 days straight, my professor told us to go home and get some sleep, or he will not accept the assignment. –  Jeremy Heiler Jan 10 '11 at 20:02
show 3 more comments

Tired or not feeling too well, so it takes a lot longer to concentrate enough to get something done, combined with a whole pile of interruptions. End result: nothing done at all.

share|improve this answer
add comment

2 in the morning, about to go live with a new automation program, and we hit the start button and the dispenser head crashes into the guarding causing about $1000 damage. This was due to a line of code from our template logic that clearly said it would disable motion if the guard wasn't out of the way, but upon further investigation, the underlying code had been bypassed.

That was my worst day. Then recently I participated in the flattening of a much more expensive robot arm in a 600 ton press due to a bug in the fault handling, so now that's my worst day.

Wiping out data is one thing (you should always have backups), but causing physical damage is quite another.

share|improve this answer
4  
Bugs that transcend user space into meat space are never fun. :-) –  Andrew Flanagan Jan 10 '11 at 18:49
7  
@Andrew Flanagan: my colleagues who have done automation in slaughter-houses would find some disturbing humor in your choice of the phrase "meat space" :) –  Scott Whitlock Jan 10 '11 at 19:42
1  
Reminds me of writing s/w that controlled hydraulic stuff with 5000 PSI pressure. One wrong move and you have a jet of oil from a vent port that will cut your arm off. I never did that, but I did manage to dump 20 gallons of oil on the floor from a tank that didn't pump out. You use a LOT of rags mopping up 20 gallons of oil. –  quickly_now Jan 27 '11 at 3:56
1  
@quickly_now - had a similar situation with a power steering unit tester, but that was only about 1000 PSI. I was on the e-stop really fast, but it's amazing how much power steering fluid came out in half a second. :) –  Scott Whitlock Jan 27 '11 at 12:46
show 1 more comment

The worst day for me starts off when I have a TODO list of 5 items. I get latest code and the build is broken. Go off to find the person responsible.

Go get some coffee while I wait, but the pot is empty. Come back to my desk to an email about a surprise meeting that starts in 10min.

Get back from the meeting with some new items on my TODO list, that have to wait b/c someone has come and asked for some help/advice.

Finally sit down to code and I get a urgent bug from the QA team, only it's not really defined. Have to go find the QA team and find out what's wrong.

Get back to my desk with an updated ticket, only to see that it's time for a status meeting. Status meeting actually morphs into a 2 hour design meeting.

By the time the end of the day rolls around, nothing has been done and my TODO list has now doubled in size.

share|improve this answer
3  
Thats the kind of "normal day" I lived through for 5 solid years. –  quickly_now Jan 27 '11 at 3:59
2  
That sounds all too familiar. –  sevenseacat Jan 27 '11 at 7:19
3  
I'm confused, you started with a TODO list of only 5 items?! –  CaffGeek Feb 7 '11 at 22:03
show 1 more comment

The worst that's not disaster-related is when I pulled a 25-hour shift and when I asked my boss if I could go home at 2:30am and remote in if necessary replied with "Well, you know I'd love to let you, but the moment you were gone, something would happen that would need you on-site." Somehow, I think if something went wrong, having my burned-out self working on it would not have done it any good.

share|improve this answer
8  
I once had a boss tell me I should be able to work three days in row without sleep. Needless to stay I moved on very shortly after that conversation. –  HLGEM Jan 10 '11 at 19:09
add comment

I think there were several candidates for "worst."

  1. Most embarrassing. The day I restored the development database backup to the production server (oops, wrong way). It set off the alerts, and about the time I realized I just screwed up, every phone in the office, and every pager, including "the football" (an Iridium pager, because the principles lived "in the mountains" and were frequently in areas where there was no cell phone coverage) went off. Within minutes, they were calling the clients apologizing for why their "four nines reliability" website went down.

  2. Most tense. There was a really nasty meeting with several department heads, and the CEO, CTO and CIO. It was one of those meetings where you know someone is going to get fired (namely me because the department I ran was "losing" data and could no longer be trusted). Essentially, one of the trading partners was losing EDI transactions (these were prescriptions and they could not get lost or folks might die) and the fickle finger of you're-fired was waving in the air. One of the steps involved one of the staff to fax certain things to a pharmacy that had such old computers that while some transactions went EDI, others had to go fax (they were still using Excel 2 when Excel 97 was current - and we couldn't make spreadsheets in a format that Excel 2 could open). So I mentioned the fax bit and we went to the binder where this idiot kept the faxes. Opening to a random page, I found several doctors crossed off. "Why is this doctor crossed off?" I asked. The CEO is checking the name with what the trading partner had in their system and that doctor didn't appear. "Oh they weren't ready, so we crossed them off until they were ready" said the idiot. All of a sudden, the laser beams of death stopped pointing at me - and were focused clearly on that idiot. If she wasn't a close personal friend of the CEO, she'd have been escorted off the premises right then and there.

  3. Most crunch time. The day we had a fugly deployment of a website. Everyone had to stay, and my boss didn't want anyone to go home because of emergencies and regulatory deadlines involved. I was in his office at about 2am begging to go home, and he wouldn't say yes. Next thing I know they're shaking me awake and the sun's coming up. It appears I passed out in his office and no one needed me at all, so they just let me sleep. Apparently I snore real loud and one co-worker recorded it for a ring tone.

share|improve this answer
4  
+1 for the ring tone :) –  l0b0 Jan 27 '11 at 9:14
2  
+1 for categorizing "worst" into several candidates –  Spoike Jan 27 '11 at 9:22
add comment

As a student, I worked part-time on a fancy audio algorithm. It had been originally written by another student based on some obscure mathematical paper. For the life of me, I could not figure out what that code was doing. There were some 3k lines of code in the project and I did many an improvement on many parts of it, but any time I modified the central algorithm, I messed it up.
This went on for seven months. Every week I would spend my twelve hours looking at that code but its workings would still be mystifying to me. Eventually, I gave up. After seven months, I went to my professor and told him I could not do it. Thankfully, I did not lose the job.

A few months later I re-read that mathematical paper and suddenly it all made sense. I went to my professor and told him that I would like to take another stab at that algorithm. In one crazy afternoon I solved the problem that had stymied me for seven months.

But really, staring at 3k lines of mystifying code for seven months is not something I would want to inflict on anyone.

(Bastis law: You can solve any programming problem if you stare at it long enough.)

share|improve this answer
6  
IMHO, taking a break from the problem probably did you a lot of good. "Staring at a problem" suffers from very rapidly diminishing returns I think. Stare at it a it, go away, come back to it tomorrow/next week/next month/next year, leave it again, repeat until done. Just 'cos you're not actively thinking about it doesn't mean your subconcious isn't. –  timday Jan 10 '11 at 19:40
show 2 more comments

Took a newly created position in a start up as head of technical support. I was given the office and computer of one of the programmers who did support and development, which still had the full development environment installed (He got a new computer and office.). During a client support call, I ran the client's issue in debug mode and found the programmer's error and not only reported the bug, but indicated the line of code and recommended the correction.

What should have put me on the path to being another developer on the team, lead to having the IDE uninstalled from my machine. I got to suffer working with the program just like all the other users. That didn't last long, because they rewrote a whole new and improved version of the application - the first release failed to install - go figure. The need for technical support doubled.

In the end, I developed a whole library of things , NOT to do. I realize the experience wasn't all bad - it was the last time I had my own office.

share|improve this answer
1  
@Mark C - yes, apparently I wasn't suppose to be looking at the code or embarassing the programmers. I had just started, so I didn't ask many questions. –  JeffO Jan 27 '11 at 14:56
1  
@Chris Kaminski - things have opened up a lot more in this profession over the years (My situation happened in '98) and cooperation is reinforced. I hope the trend continues. –  JeffO Feb 8 '11 at 12:31
show 5 more comments

50,000+ Errors:

I had to dump a project and start a-fresh in all three times (the solutions would work in the dev environment, but in production all sorts of intermittent crashes would occur). On the final time round, everything worked, or almost everything.

Unfortunately one bug had crept in that caused an infinite loop, the system had an error reporting module that would log any errors, generate a report file and email it to the development team.

The sheer agony, email after email was sent. Even after killing the application, the emails kept coming for the next 24 Hours! ouch.

But it does have a good ending: The system has been running 24/7 for nearly three years to date, with zero glitch and continues to hum away...

share|improve this answer
2  
Ahhh, the old infinite loop email alert problem. Been there, done that. The IT team turned off SMTP services for a long time in the development environment after that. They could not do that in prod, but they made us pay in dev. –  Mark Freedman Jan 10 '11 at 14:04
add comment

My worst days are related to bad practices, i.e. working on a tight schedule without proper project management in place, going live with basically untested software, and a desperate frenzy to to save what can be saved, only to make things worse by making more errors of haste.

share|improve this answer
show 2 more comments

Being manager of a development group, and having a serial string of disasters (for all sorts of reasons) in shipping code, shipped products, and a few salespeople who were very pissed off about this.

This included installations in several major buildings as part of the building management systems - so things that did not work right were sheeted home to us even though, as it later turned out, it was something else (somebody elses equipment at fault).

Being hauled into daily and weekly meetings with the CEO to be told "fix this immediately" when in fact we'd been battling the trouble for some months and were no closer to a resolution.

Being blamed for somebody elses bad equipment, for months and months, being pressured to fix it (when we didn't know the cause). Being screwed over by sales people. Being threatened with lawsuits.

No fun.

We did in fact fix it. We found something else in the building causing the trouble by putting people in there (at our companies expense) and gathering irrefutable evidence.

Oh! The joy I got from writing the letter for the CEO to sign...

share|improve this answer
add comment

What happens when

A big product is about to release, all folks in office know that and a party arrangement is going on. QA has given thumbs up and in excitement he told about it to client, you are about to build the final version, written the mail and only attachment of that build is remaining.

And all of a sudden everything crashes. You can't figure out why a 1 mb build is taking a time worth 10mb build. The workstation restarts and says no bootable media found..... You call your support guy, who comes and says it will take a whole day either to backup or to restore the data.

It was the same place where you are having a party and now you, the lead and the project manager was trying to convince the client about delay of the release. After two hours of arguing he agrees to delay the launch by two days. You have to agree on finishing the builds by the next day. At the end of the day you have to cancel the ticket for the bus to my home to find out that there is no ticket available for next four days, your entire leave(4 days) is going to be wasted.

Embarrassment, disappointments, hot arguments.... And the bottom line is that this day is supposed to be a victory day...

However I canceled my leaves. The application launched as expected and we have got some positive feedback in first two days. And finally I had got an extra paid leave and a happy endings.

share|improve this answer
1  
we didn't have version controls at that time.... Policy was to have version controls only for web projects, not for samples and mobile related products..... This was the case which forced management to introduce version control everywhere.. –  Prasham Feb 9 '12 at 7:37
show 1 more comment

A decade ago when I was working remotely and had a week of work wiped out by a third party defragment program. Since then I haven't had defrag on autopilot. I won't let them run until I check in my changes or run the backup.

share|improve this answer
1  
Also defrags are less useful these days with all the caching in harddisks and operating systems. Don't let ANYTHING mess with the filesystem unless you have backup. –  user1249 Nov 21 '10 at 13:13
show 1 more comment

I was working on a large change, and when I went to merge my code back, I somehow took the wrong changes in the diff tool, at various places, and ended up with a completely unworkable changeset. :| (Fortunately I was able to rollback/reapply, but it took a bit of wrangling and I wasn't too happy)

Then one time at my first 'real' IT job I was writing a script to migrate a ton of UNIX-based websites to a new server, and quickly learned the value of input validation... The trailing space at the end of my username list caused "undesirable" results. Luckily my mentor at the time agreed to fix my mistake for a bottle of rum ;)

share|improve this answer
show 1 more comment

Worst two days of debugging: Had a bug in converting km/h into m/s. It took 48 hours to find out that the bug could be in such a simple place.

share|improve this answer
add comment

The day I found myself going back to the office at 4AM after two hours sleep having just figured out what the problem was that caused a bug I had run into at midnight when the product was supposed to go off for CD mastering at midday that day.

I wish I had known a lot more about good software development practice than I did at the time, because I could have saved myself a tonne of grief on that project with more and better unit testing and a less tightly coupled and messy design. One of the times when having a senior developer around to point me in the right direction ( instead of being the senior developer ) would have been a big improvement...

share|improve this answer
add comment

My personal worst was coming in Monday morning and finding out the update I pushed out Friday night wasn't tested very well, and Microsoft's InternetDecode API borked on string values longer than 1400 characters, losing 90% of the data my customers entered over the weekend. I learned a damn good lesson that day about unit tests.

share|improve this answer
show 2 more comments

The discovery of the side effects of a static in .Net that didn't quite work like I had thought. My initial thinking was that this was a way to keep a value alive on the server for allowing multiple windows to be spawned in an application. Only when there was the bug of someone seeing some data that didn't seem right did I realize what a big boo-boo I had made. That would probably be the worst day I had as once I knew what it had done, I had to find a fix and fast as this was discovered in production. I have since become very cautious of ever having static classes.

share|improve this answer
add comment

My worst day, well that might be the day a very complex import process did not do one of the many tasks in it and all the people at one particular organizational level did not get into the table that defined their user rights. After a whole day of tracking through the process (including getting one of my coworkers to track it through with me as I could not find a problem), rerunning the offending piece of code with no changes put the users in correctly. I still have no idea why they didn't go in the first time, they were in the import file correctly, every step that modified them did it's job correctly and the final piece of code worked correctly! arrrrgggggghhh. This is my most frustrating because I couldn't find something to fix.

Anopther really bad day happened when we did an import at 5 pm on a Friday and immediately had user login problems. Turned out to be very bad data given to us by the client, but we spent most of that evening fixing the problem. Lesson learned, never run a process on Friday just before quitting time! Other lesson learned, add checks to ensure the data is correct (although the particular data problem we had was not one anyone would have anticipated beforehand and we did have checks for the things that normally could go wrong.). Third lesson, whatever checks you have, you will miss something big someday.

Then we had a bad day when we had to develop and sendout emails to every sales rep of all our clients because someone else had accidentally forgotten to put clientId in the where clause of a proc that sent out a nightly email and sent all the information about one client to every other client we had. Wasn't our mistake, but fixing it under the immediate prosesure of how soon can you have this done was nerve-wracking. Especially since we certainly couldn't afford to have a bug in the way the apology email was sent out! No lessons learned from this other than to make sure someone code reviews stuff before it sends emails to three-gazillion people.

share|improve this answer
add comment

That happenned in my propagation: It's when the tester discover a very serious bug in my part of work at the end of the work-day. It's just several hours before the demo, and I'm too intense to work it out.

Fortunately, a senior developer point out that's a bug generated by incorrect server configuration.

share|improve this answer
add comment

It was the final day to submit the product to the clients but product was not 100% completed to meet the customer standard. Also there were new bugs which were not there before so we had to neutralize them too.

But after all our team were able to complete the product before the next day since one of our best programmer didn't even slept at that day to finish it. We were able to deliver the product to the customer too.

That is the worst software development day I ever had.

share|improve this answer
add comment

I wrote a database update to the "current" specification document for our manufacturing production system. This is on a proprietary database system that you can't easily jump in and look at like today's SQL Server, etc. Unfortunately, the specification document left off the fact that another developer had appended some important accounting info at the end a few months before. So, when I ran my update, it stripped all the accounting info off, while leaving all the production data in place. Was not detected for some hours (since accounting wasn't billing at the time), so the backup was pretty old. A lot of stuff got out of sync by silently failing too. End result, we (as in everybody in the company) ended up rolling back to the latest backup and rekeying a whole day's worth of work!

share|improve this answer
add comment

On day ONE, I was given a task to enhance an existing C# project. But I was not able to compile the project since several pieces of the code was not checked in by a developer who left the company in an unhappy manner. To my horror, his development machine was not available at all.

Later, I heard from others that he made a build few days before leaving the firm and that was already in production. I immediately requested for those binary dlls and .NET Reflector came to my rescue as a result of which, I was able to recover the missing source code.

share|improve this answer
show 1 more comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.