What was your worst software development day like?

Question

Sometimes, people really have a bad day...
I think it's interesting to learn why that day was bad and learn from our mistakes made on that day.

So... What was your worst software development day like?

Answer 1

It was the day I wrote a SQL delete statement to run against a large table and forgot the where clause. Thankfully we had last night's backups handy.

Answer 2

Not a day: ten of them. I just spent almost two months fixing an algorithm in a library that was causing paradoxical behavior in the output of a simulation. After all that hard work, I re-ran the simulation and was still getting the same wrong results, so for a week and a half I examined every bloody line in my simulation, every day getting no change. So by now I have about nine weeks of disappointed looks from my advisor, who has no advice to offer.

Then I figure out that I'm still using the old library.

Answer 3

For me those were not days of technical stalls, but the days where things in the air changed.

1st worst day: Having just joined the company, I was shown the code in .NET 1.1 and told we were going to stick with it for a while. At the interview we were talking about 2.0/3.5 and I was supposed to work with that. There's nothing that can ruin a new working relationship right at the start like a good lie...

2nd worst day: We weren't allowed to write stored procedures for the only reason the boss didn't like them.

3rd worst day: After having finally migrated to .NET 3.5 the boss told us to stay clear of that new stuff like LINQ and lambda functions for the only reason it was something new and not time-proven.

It was then when I understood I didn't belong with them.

Answer 4

After spending a 100 hour work-week feverishly integrating software in an overcrowded under-ventilated makeshift office above the shop floor where they were assembling our simulator, catching a late flight to D.C., and then working until 6AM stomping out the last bugs, and trying unsuccessfully to figure out why the motion base was pitching backwards instead of forwards whenever you stomped on the brakes.

Luckily, none of the generals we gave demos to on that day seemed to notice the glitch.

2 days later, the misplaced negative sign was obvious.

Lesson Learned: Get Some Sleep.

Answer 5

Tired or not feeling too well, so it takes a lot longer to concentrate enough to get something done, combined with a whole pile of interruptions. End result: nothing done at all.

Answer 6

2 in the morning, about to go live with a new automation program, and we hit the start button and the dispenser head crashes into the guarding causing about $1000 damage. This was due to a line of code from our template logic that clearly said it would disable motion if the guard wasn't out of the way, but upon further investigation, the underlying code had been bypassed.

That was my worst day. Then recently I participated in the flattening of a much more expensive robot arm in a 600 ton press due to a bug in the fault handling, so now that's my worst day.

Wiping out data is one thing (you should always have backups), but causing physical damage is quite another.

Answer 7

The worst day for me starts off when I have a TODO list of 5 items. I get latest code and the build is broken. Go off to find the person responsible.

Go get some coffee while I wait, but the pot is empty. Come back to my desk to an email about a surprise meeting that starts in 10min.

Get back from the meeting with some new items on my TODO list, that have to wait b/c someone has come and asked for some help/advice.

Finally sit down to code and I get a urgent bug from the QA team, only it's not really defined. Have to go find the QA team and find out what's wrong.

Get back to my desk with an updated ticket, only to see that it's time for a status meeting. Status meeting actually morphs into a 2 hour design meeting.

By the time the end of the day rolls around, nothing has been done and my TODO list has now doubled in size.

Answer 8

The worst that's not disaster-related is when I pulled a 25-hour shift and when I asked my boss if I could go home at 2:30am and remote in if necessary replied with "Well, you know I'd love to let you, but the moment you were gone, something would happen that would need you on-site." Somehow, I think if something went wrong, having my burned-out self working on it would not have done it any good.

Answer 9

As a student, I worked part-time on a fancy audio algorithm. It had been originally written by another student based on some obscure mathematical paper. For the life of me, I could not figure out what that code was doing. There were some 3k lines of code in the project and I did many an improvement on many parts of it, but any time I modified the central algorithm, I messed it up.
This went on for seven months. Every week I would spend my twelve hours looking at that code but its workings would still be mystifying to me. Eventually, I gave up. After seven months, I went to my professor and told him I could not do it. Thankfully, I did not lose the job.

A few months later I re-read that mathematical paper and suddenly it all made sense. I went to my professor and told him that I would like to take another stab at that algorithm. In one crazy afternoon I solved the problem that had stymied me for seven months.

But really, staring at 3k lines of mystifying code for seven months is not something I would want to inflict on anyone.

(Bastis law: You can solve any programming problem if you stare at it long enough.)

Answer 10

Took a newly created position in a start up as head of technical support. I was given the office and computer of one of the programmers who did support and development, which still had the full development environment installed (He got a new computer and office.). During a client support call, I ran the client's issue in debug mode and found the programmer's error and not only reported the bug, but indicated the line of code and recommended the correction.

What should have put me on the path to being another developer on the team, lead to having the IDE uninstalled from my machine. I got to suffer working with the program just like all the other users. That didn't last long, because they rewrote a whole new and improved version of the application - the first release failed to install - go figure. The need for technical support doubled.

In the end, I developed a whole library of things , NOT to do. I realize the experience wasn't all bad - it was the last time I had my own office.

Answer 11

I think there were several candidates for "worst."

1. Most embarrassing. The day I restored the development database backup to the production server (oops, wrong way). It set off the alerts, and about the time I realized I just screwed up, every phone in the office, and every pager, including "the football" (an Iridium pager, because the principles lived "in the mountains" and were frequently in areas where there was no cell phone coverage) went off. Within minutes, they were calling the clients apologizing for why their "four nines reliability" website went down.

2. Most tense. There was a really nasty meeting with several department heads, and the CEO, CTO and CIO. It was one of those meetings where you know someone is going to get fired (namely me because the department I ran was "losing" data and could no longer be trusted). Essentially, one of the trading partners was losing EDI transactions (these were prescriptions and they could not get lost or folks might die) and the fickle finger of you're-fired was waving in the air. One of the steps involved one of the staff to fax certain things to a pharmacy that had such old computers that while some transactions went EDI, others had to go fax (they were still using Excel 2 when Excel 97 was current - and we couldn't make spreadsheets in a format that Excel 2 could open). So I mentioned the fax bit and we went to the binder where this idiot kept the faxes. Opening to a random page, I found several doctors crossed off. "Why is this doctor crossed off?" I asked. The CEO is checking the name with what the trading partner had in their system and that doctor didn't appear. "Oh they weren't ready, so we crossed them off until they were ready" said the idiot. All of a sudden, the laser beams of death stopped pointing at me - and were focused clearly on that idiot. If she wasn't a close personal friend of the CEO, she'd have been escorted off the premises right then and there.

3. Most crunch time. The day we had a fugly deployment of a website. Everyone had to stay, and my boss didn't want anyone to go home because of emergencies and regulatory deadlines involved. I was in his office at about 2am begging to go home, and he wouldn't say yes. Next thing I know they're shaking me awake and the sun's coming up. It appears I passed out in his office and no one needed me at all, so they just let me sleep. Apparently I snore real loud and one co-worker recorded it for a ring tone.

Answer 12

Being manager of a development group, and having a serial string of disasters (for all sorts of reasons) in shipping code, shipped products, and a few salespeople who were very pissed off about this.

This included installations in several major buildings as part of the building management systems - so things that did not work right were sheeted home to us even though, as it later turned out, it was something else (somebody elses equipment at fault).

Being hauled into daily and weekly meetings with the CEO to be told "fix this immediately" when in fact we'd been battling the trouble for some months and were no closer to a resolution.

Being blamed for somebody elses bad equipment, for months and months, being pressured to fix it (when we didn't know the cause). Being screwed over by sales people. Being threatened with lawsuits.

No fun.

We did in fact fix it. We found something else in the building causing the trouble by putting people in there (at our companies expense) and gathering irrefutable evidence.

Oh! The joy I got from writing the letter for the CEO to sign...

Answer 13

A decade ago when I was working remotely and had a week of work wiped out by a third party defragment program. Since then I haven't had defrag on autopilot. I won't let them run until I check in my changes or run the backup.

Answer 14

I was working on a large change, and when I went to merge my code back, I somehow took the wrong changes in the diff tool, at various places, and ended up with a completely unworkable changeset. :| (Fortunately I was able to rollback/reapply, but it took a bit of wrangling and I wasn't too happy)

Then one time at my first 'real' IT job I was writing a script to migrate a ton of UNIX-based websites to a new server, and quickly learned the value of input validation... The trailing space at the end of my username list caused "undesirable" results. Luckily my mentor at the time agreed to fix my mistake for a bottle of rum ;)

Answer 15

50,000+ Errors:

I had to dump a project and start a-fresh in all three times (the solutions would work in the dev environment, but in production all sorts of intermittent crashes would occur). On the final time round, everything worked, or almost everything.

Unfortunately one bug had crept in that caused an infinite loop, the system had an error reporting module that would log any errors, generate a report file and email it to the development team.

The sheer agony, email after email was sent. Even after killing the application, the emails kept coming for the next 24 Hours! ouch.

But it does have a good ending: The system has been running 24/7 for nearly three years to date, with zero glitch and continues to hum away...

Answer 16

My worst days are related to bad practices, i.e. working on a tight schedule without proper project management in place, going live with basically untested software, and a desperate frenzy to to save what can be saved, only to make things worse by making more errors of haste.

Answer 17

Worst two days of debugging: Had a bug in converting km/h into m/s. It took 48 hours to find out that the bug could be in such a simple place.

Answer 18

What happens when

A big product is about to release, all folks in office know that and a party arrangement is going on. QA has given thumbs up and in excitement he told about it to client, you are about to build the final version, written the mail and only attachment of that build is remaining.

And all of a sudden everything crashes. You can't figure out why a 1 mb build is taking a time worth 10mb build. The workstation restarts and says no bootable media found..... You call your support guy, who comes and says it will take a whole day either to backup or to restore the data.

It was the same place where you are having a party and now you, the lead and the project manager was trying to convince the client about delay of the release. After two hours of arguing he agrees to delay the launch by two days. You have to agree on finishing the builds by the next day. At the end of the day you have to cancel the ticket for the bus to my home to find out that there is no ticket available for next four days, your entire leave(4 days) is going to be wasted.

Embarrassment, disappointments, hot arguments.... And the bottom line is that this day is supposed to be a victory day...

However I canceled my leaves. The application launched as expected and we have got some positive feedback in first two days. And finally I had got an extra paid leave and a happy endings.

Answer 19

The day I found myself going back to the office at 4AM after two hours sleep having just figured out what the problem was that caused a bug I had run into at midnight when the product was supposed to go off for CD mastering at midday that day.

I wish I had known a lot more about good software development practice than I did at the time, because I could have saved myself a tonne of grief on that project with more and better unit testing and a less tightly coupled and messy design. One of the times when having a senior developer around to point me in the right direction ( instead of being the senior developer ) would have been a big improvement...

Answer 20

My personal worst was coming in Monday morning and finding out the update I pushed out Friday night wasn't tested very well, and Microsoft's InternetDecode API borked on string values longer than 1400 characters, losing 90% of the data my customers entered over the weekend. I learned a damn good lesson that day about unit tests.

Answer 21

The discovery of the side effects of a static in .Net that didn't quite work like I had thought. My initial thinking was that this was a way to keep a value alive on the server for allowing multiple windows to be spawned in an application. Only when there was the bug of someone seeing some data that didn't seem right did I realize what a big boo-boo I had made. That would probably be the worst day I had as once I knew what it had done, I had to find a fix and fast as this was discovered in production. I have since become very cautious of ever having static classes.

Answer 22

My worst day, well that might be the day a very complex import process did not do one of the many tasks in it and all the people at one particular organizational level did not get into the table that defined their user rights. After a whole day of tracking through the process (including getting one of my coworkers to track it through with me as I could not find a problem), rerunning the offending piece of code with no changes put the users in correctly. I still have no idea why they didn't go in the first time, they were in the import file correctly, every step that modified them did it's job correctly and the final piece of code worked correctly! arrrrgggggghhh. This is my most frustrating because I couldn't find something to fix.

Anopther really bad day happened when we did an import at 5 pm on a Friday and immediately had user login problems. Turned out to be very bad data given to us by the client, but we spent most of that evening fixing the problem. Lesson learned, never run a process on Friday just before quitting time! Other lesson learned, add checks to ensure the data is correct (although the particular data problem we had was not one anyone would have anticipated beforehand and we did have checks for the things that normally could go wrong.). Third lesson, whatever checks you have, you will miss something big someday.

Then we had a bad day when we had to develop and sendout emails to every sales rep of all our clients because someone else had accidentally forgotten to put clientId in the where clause of a proc that sent out a nightly email and sent all the information about one client to every other client we had. Wasn't our mistake, but fixing it under the immediate prosesure of how soon can you have this done was nerve-wracking. Especially since we certainly couldn't afford to have a bug in the way the apology email was sent out! No lessons learned from this other than to make sure someone code reviews stuff before it sends emails to three-gazillion people.

Answer 23

That happenned in my propagation: It's when the tester discover a very serious bug in my part of work at the end of the work-day. It's just several hours before the demo, and I'm too intense to work it out.

Fortunately, a senior developer point out that's a bug generated by incorrect server configuration.

Answer 24

It was the final day to submit the product to the clients but product was not 100% completed to meet the customer standard. Also there were new bugs which were not there before so we had to neutralize them too.

But after all our team were able to complete the product before the next day since one of our best programmer didn't even slept at that day to finish it. We were able to deliver the product to the customer too.

That is the worst software development day I ever had.

Answer 25

I wrote a database update to the "current" specification document for our manufacturing production system. This is on a proprietary database system that you can't easily jump in and look at like today's SQL Server, etc. Unfortunately, the specification document left off the fact that another developer had appended some important accounting info at the end a few months before. So, when I ran my update, it stripped all the accounting info off, while leaving all the production data in place. Was not detected for some hours (since accounting wasn't billing at the time), so the backup was pretty old. A lot of stuff got out of sync by silently failing too. End result, we (as in everybody in the company) ended up rolling back to the latest backup and rekeying a whole day's worth of work!