33

Similar to the question I read on Server Fault, what is the biggest mistake you've ever made in an IT related position. Some examples from friends:

I needed to do some work on a production site so I decided to copy over the live database to the beta site. Pretty standard, but when I went to the beta site it was still pulling out-of-date info. OOPS! I had copied the beta database over to the live site! Thank god for backups.

And for me, I created a form for an event that was to be held during a specific time range. Participants would fill out the form for a chance to win, and we would send the event organizers a CSV from the database. I went into the database, and found ONLY 1 ENTRY, MINE. Upon investigating, it appears as though I forgot an auto increment key, and because of the server setup there was no way to recover the lost data.

I am aware this question is similar to ones on Stack Overflow but the ones I found seemed to receive generic answers instead of actual stories :)

What is the biggest coding error/mistake ever…

3
  • 6
    guinness questions! Oct 25, 2010 at 0:03
  • Why o why was this question closed? I feel the need to add my own big mistake: I took down a LIVE website for three hours (ie. 500 HTTP status) after deploying into Production after being told specifically NOT TO DEPLOY TO PRODUCTION. Worst of all, I tried to hide the mistake, but the client noticed and didn't seem very happy. Mar 24, 2015 at 5:51
  • Easy one for me. Paid out investment withdrawals but converted to cents twice. Paid out 3 mil instead of 30k. Runner up - bad SQL where clause. Notified 4k people that their $1 - $10 investments grew by $1800 over night. Obviously some of these clients went and spent the money. Recovered fairly quickly financially. Broken confidence and guilt never really went away. How to recover: take responsibility. Admit your mistake. Don't blame anything else even if there were contributing factors. This was met with understanding and team effort to get it fixed. Client appreciated the honesty.
    – Reasurria
    Dec 20, 2017 at 5:32

37 Answers 37

54

Issuing a SQL UPDATE with a 'bad' WHERE Clause that matched everything.

Lesson Learned: Always issue a SELECT first to see what will be changed.

13
  • 15
    Or set autocommit to false so you can see how many rows would be affected. Oct 25, 2010 at 5:09
  • 2
    Oh, I've done that. Very nasty...
    – glenatron
    Oct 25, 2010 at 11:15
  • 4
    Haha I think everyone with some SQL under their belt has done this at some time, me included
    – billy.bob
    Oct 25, 2010 at 11:55
  • 3
    Which is why updates should be done on dev first! No hot fixes directly to prod. And be very grateful for audit tables if you have them when you do this.
    – HLGEM
    Oct 25, 2010 at 15:05
  • 13
    I did that once to reset someone's password. After I set everyone's password to the same thing, I simply told my boss that tech support would be getting a lot of phone calls and that we should tell the caller we had to change their password for security reasons. Oct 25, 2010 at 16:35
36

Typo's

Spent three solid days on this chunk:

if($func == "remove")
{
    $p->comments[$index]->removed = true;
    $p->save();
}
else if($func == "approve");
{
    $p->comments[$index]->approved = true;
    $p->comments[$index]->removed = false;
    $p->save();
}

See the error? It's the semicolon at the end of the else if. I couldn't figure out why my removed comments weren't removing. Dug into the database, the AJAX request I was using, POST variables, had error_log's and alerts everywhere. Ended up re-writing the method and it worked. Then I did a diff with the original version and noticed the semicolon.

Typo's are the hardest to track down on a language that isn't compiled or pre-checked by something. Even errors like this would be difficult on a compiled language. Change a == to = and suddenly you have assignment inside an if.

28
  • 8
    Here is where tools like Resharper can really pay back for themselves - it would have highlighted this as a warning and prompted you to remove it. Oct 25, 2010 at 7:57
  • 24
    Is the title an (intended) pun? :o
    – Agos
    Oct 25, 2010 at 9:32
  • 3
    @Rogue Coder: the difference between single and double quotes is vanishingly small. It's a misuse of time to spend brain cycles worrying about it. See the test at the bottom of phpbench.com Oct 25, 2010 at 9:54
  • 2
    @back2dos: Errr... right. So you recommend dropping all those mentioned languages in favor of haXe? Something that first appeared in 2005? Go right ahead with that.
    – Josh K
    Oct 25, 2010 at 15:06
  • 10
    Thank you for the argument in favor of the One True Brace Style. +1
    – eswald
    Oct 25, 2010 at 17:33
28

Thinking that programming is mostly about building cool new stuff from scratch.

2
  • 1
    When I first started I too had this thought. Then I learnt there is far more maintenance of other peoples than anything. That's why I try now to write the best code (with comments) that I can.
    – user2358
    Feb 11, 2011 at 4:50
  • 2
    There's still plenty of new code to write. A lot of open source projects require coding from scratch that implements the same functionality as a closed system.
    – PP.
    Feb 11, 2011 at 8:31
23

My biggest mistake was to think that programming is easy money...

4
  • 27
    If you got into programming for the money, you should get a new career. Not that the money isn't there, it is, but you need the obsessive love for programming that true developers have, or you will burn out very quickly
    – johnc
    Oct 26, 2010 at 0:02
  • 1
    You made a good point. I really did not get into programming for the money, but I really thought that it would be a lot more easy to live from it. Programming is my passion and I would do that even if I have to pay for. Oct 26, 2010 at 16:42
  • 4
    @johnc same would go for most professions. I honestly don't understand people who go to university to "learn programming". I started when I was 14 with K&R. I went to university to "learn electronics" - but guess what, I ended up as a programmer anyway. The very best musicians taught themselves. The very best programmers taught themselves. That's life.
    – PP.
    Feb 11, 2011 at 8:30
  • amen brother, amen
    – Misters
    Dec 24, 2013 at 5:02
23

Accidentally deleted the database that stored all our customer info, order history and invoices going back to the start of the company (several years worth).

To be fair, my employer has to share some of the blame though. They had the only copy of that database stored on the Mac SE (yes this was a long time ago) that they gave me (a brand new employee, first job out of college) as my workstation and had never even considered making a backup.

So, thinking it was a copy of the DB I dragged it to the trash can. Because of the size of the file it deleted it immediately. We eventually got it back after paying an ungodly amount to a data restoration service, but for about 5 days we couldn't fulfill or bill any orders, and had no way to access information about any customer. It pretty much brought the (3 person company) to a standstill.

9
  • 15
    Ouch. Note to self: Always take a backup: even if there is already supposed to be a backup.
    – Kramii
    Oct 25, 2010 at 10:58
  • 1
    @Karmii: Best advice, hands down.
    – Chris
    Oct 25, 2010 at 14:18
  • @Kramii - Glad I learned that lesson early in my career.
    – JohnFx
    Oct 25, 2010 at 14:20
  • Didn't you get fired? Jan 13, 2011 at 19:54
  • 3
    There was enough blame to go around. I was just a kid back then and probably more naive than I would be today about assuming that a real company would do anything so silly as putting the only copy of the DB on my desktop. Experience has shown me that it is foolish to overestimate my employer's preparedness, even at a big company.
    – JohnFx
    Jan 14, 2011 at 23:34
15

when phpmyadmin asked:

"You are about to DESTROY a complete database! Do you really want to DROP DATABASE xxx?"

I hit Enter.

4
  • 26
    Great example of when defensive programming would have helped prevent major issues. If you're doing something where the repercussions could be grave, like perhaps dropping tables/databases, default the user response to "No" so that they must ACTIVELY agree to the action.
    – Hugo
    Oct 25, 2010 at 3:22
  • 6
    @Hugo: +1. I had a professor at uni (ex Unix/database sysadmin) who used to say "whenever you are doing anything serious, always take your hands off the keyboard and sit on them for about ten seconds and think about what's about to happen." Oct 25, 2010 at 5:00
  • 1
    @Hugo: I go farther--"You are about to DESTROY a complete database! If you really want to DROP DATABASE xxx, type DELETE:" Jan 14, 2011 at 5:31
  • 3
    @Hugo @Loren Pechtel - "You are about to DESTROY a complete database! A web search of your login has turned up multiple occurrences of 'lol', and therefore this action is being denied. Please see your administrator or learn to communicate."
    – detly
    Jan 14, 2011 at 6:18
15

Not sure it was my "biggest mistake", but definitely a memorable one. In my first week or two at a new job, I was assigned to do a small "feature enhancement" that involved modifying the sort order of some items on one of the most popular pages on the site. I wasn't too familiar with the code base, but quickly found the relevant Comparator and added calls to a couple harmless-looking methods inside the compareTo method without thinking too much about it. The code tested locally and in QA w/o issues and went live.

I had a 1 on 1 with my boss that same day and with a big smile on his face, he congratulated me on getting my first "feature" live so soon after joining. We both looked on as he visited the corresponding page to see the feature in action. The page took 47 seconds to load. My heart sank. He refreshed the page: 53 seconds. His smile vanished. I went back to my desk and spent a long evening debugging and pushing critical patches to the live site.

It turns out one of the harmless looking methods I added was a remote service call that resulted in at least one DB hit. In each compareTo call. So on a page where 2,000+ items were being sorted, I was making ~6000 (nlogn) DB hits. Ouch.

3
  • Was the DB hit obvious in your code, or did it occur in some method/property you were calling?
    – dbkk
    Oct 25, 2010 at 19:25
  • The DB hit was actually hidden behind a method that looked like a simple getter: I added a myObj.getFoo() call in the compareTo method. The JavaDoc on the relevant subclass did note the possibility of a DB hit, but the JavaDoc on the myObj interface (which is what I looked at when writing the code) mentioned nothing of the sort. In either case, I think it's a clear demonstration of (a) inadequate testing on my behalf and (b) what can happen if you don't follow good naming conventions. Oct 25, 2010 at 19:39
  • more likely it was the (very common) case of the test environment not being representative of the production environment. You tested it, it performed decently, so you concluded it was good, never knowing (and you couldn't have, then) that the test database contained only a fraction of the data contained in the production database.
    – jwenting
    Apr 20, 2011 at 6:23
15

I've done this:

rm -rf /bin

(Actually, I didn't do exactly that. That would be dumb and inexcusable. I did it in a more subtle, roundabout way that resulted in essentially that command being executed.)

Needless to say, the Unix system was unusable after that point and had to be re-installed. I was a beginning sysadmin at the time and the senior guy overseeing me was understanding.

Some good came of the experience. I learned how to list directories without having any of the /bin commands to use.

echo *
1
  • 1
    +1 learned tricks like simulating ls and cat using only Bourne shell constructs from the single-user mode on a DEC-BSD system
    – Arcege
    Feb 11, 2011 at 5:05
11

The customer wants a mailout to their entire userbase that will basically send a personalised email about an event and when they click on a link in the email will automatically log them in to a form with half their data filled in.

I write the code, test the code, it works fine, the links work, it's pretty smooth. Having checked and double checked everything, I switch over to the live list and away we go.

The live list is significantly bigger than my sample data and the mail server falls over. I have to stop my console app, then realise I'll have to restart it from where it stopped. Fortunately this feature is not too hard to add and I have logged what users the mailout has already gone to, so having got the mailserver up and running again, and reconfigured the code to be able to restart from the point it failed previously and to pause to let the mailserver catch up, away we go.

Unfortunately I had somehow managed to not get all my code updating right. I don't remember the details of what I did, but the query to get the user data was getting one set of data, the query to generate the hash was getting another, so if a user clicked on the link they would get to a form that contained someone else's personal data from their account details. And this in a niche industry where there were lots of competetive small businesses on the list.

It didn't take long for the phone to start ringing at the customers' office...

That was the only mistake so far that has lead to me offering my resignation.

2
  • +1 for real data significantly larger than sample data :-( Jan 14, 2011 at 3:41
  • IMO this is one of the worse ones >_< Feb 11, 2011 at 4:44
7

I once used a cross join without a filter. Worked just fine on the test database with a fraction of the data. When it was deployed, it ended up with ~60M rows in an average query.

1
  • 14
    Actually your biggest mistake was in not developiong against a database of the appropriate size. Never use a small test datbase to develop for a large production database. You can't write performant code unless you have the right sized dabase.
    – HLGEM
    Oct 25, 2010 at 15:08
7

I crashed a 100 000$ laser scanner.

The controller was storing positions as integer so I divided everything by 10000 to get the actual position in inches.

When the last digit was a 0, it was omitted so the Z axis was 10 times to far.

Hillarity ensued

6

"We need to do a rewrite."

On the otherhand, it was a five year old VB6 app.

5

Allowed my manager to brow beat me into leaving a job I loved. I should have let him mess things up, get fired and then I should have stepped up and clean up the pieces. Instead I quit and have regretted it ever since.

2
  • 17
    Managers rarely get fired over such things. To get to manager level you have to be very good at covering your rear; you probably became the scapegoat the minute you left the company. Maybe you could have handled it differently, but don't beat yourself up over it. Oct 26, 2010 at 4:41
  • had he messed up, you'd have been blamed and fired (or at least have a miserable time until you quit in disgrace).
    – jwenting
    Apr 20, 2011 at 6:24
4

Seven years ago my boss & company owner, whom I'd not yet met I was so new to the job, was in another state doing a demo of our research web site to a couple of potential clients. Memberships fees ran in the low to mid five figures, so these demos were a big deal to our fledgling company.

In the middle of his demo, as I was doing some database work, I made a change to the live database and updated every survey question in every survey to the same text, something like "This is a test question" because of a flaw in the WHERE. My co-worker and I scrambled for the backup and cringed, hoping he wasn't demoing that section of the site right then but waiting for the email in all caps that gratefully never appeared.

The good side was the boss finally sprung for a development box.

3

I was running Xenu's link sleuth on our intranet to try and clear up some of the many broken links that have built up over the years (most, unsurprisingly enough were links from the wiki to the shared network drive).

After about 30 minutes I started noticing some of the new items had some odd pictures, it looked like people had used some of the grainy old stock photos that were on the system but I ignored that thinking that maybe there just wasn't anything newer that had what they wanted.

Another 10 minutes later I noticed the featured items were changing randomly for some reason. At this point it dawned on me what was happening. The intranet uses windows authentication and some of the functions (such as selecting news images and featured items) were coded to response to HTTP GET requests. The link checker had been using my authentication and had crawled to pages on the admin side, loyally doing its job and following everylink it found including ones like

/admin/displayItems/icon_update.asp?image=eastereggs.jpg&itemID=3174

3
  • 7
    This seems to be how a lot of people learn when it's appropriate to use GET and when it's appropriate to use POST
    – Oli
    Oct 25, 2010 at 12:08
  • Agreed, I was about to start work on redeveloping the intranet and this firmly placed proper use of GET and POST at the top of the requirements. ASP MVC also helps makes this easy and intuitive to do.
    – Chao
    Oct 25, 2010 at 12:35
  • 1
    not really your mistake, but a good story nonetheless :-) Oct 26, 2010 at 5:30
3

So far my only stomach-twisting, heart-rate-increasing, suddenly-get-hot-and-itchy mistake was running an UPDATE query without a WHERE clause. Luckily there was a backup from the last 15 minutes and the data didn't change very often which meant I was able to fully restore the data within 10 minutes without anybody knowing.

Nothing too bad but I've had some close calls. They are and remain close calls because I've heard enough horror stories in this industry that everything I do that could screw things up gets a good going over.

6
  • 1
    Don't you guys making DB mistakes ever call rollback?
    – Jé Queue
    Oct 25, 2010 at 23:02
  • @Xepoch, that only helps if you started a transaction
    – CaffGeek
    Jan 13, 2011 at 19:47
  • 1
    @Chad, should we generally not assume all data calls should be implicitly transactional?
    – Jé Queue
    Jan 13, 2011 at 20:29
  • 2
    @Xepoch, never assume.
    – CaffGeek
    Jan 13, 2011 at 20:56
  • 1
    @Char, I have a personal preference for maintaining all DB interactions within transactions. Those who don't, they carry risk or have business cases which don't care. The specification assumption and MANDATEs are that we will begin and maintain code projects with transactional DB management.
    – Jé Queue
    Jan 13, 2011 at 23:31
3

For a robot I work on, we keep a log file of every run. We have the robot and a simulator, both which generate these log folders. The simulator's log files are useless, but the robots log files are very useful and are stored forever.

Because the robot itself has limited hard drive space, they're transferred to another computer and stored there. This computer happens to be the main operating computer for communicating with the robot.

Well, I had just started working on the robot a couple months earlier and didn't know this. I thought the logs on the operating computer were the useless ones generated by the simulator and deleted them. We lost all of the old logs because there wasn't a proper backup.

1
  • 8
    Damn... I want to work on robots! All I do all day is build lousy web sites.
    – Dan Ray
    Oct 26, 2010 at 12:14
2

Built a version of our software on my computer instead of the build computer as it was faster (about 4 hours faster, which meant that it could go into QA that day instead of the next) and the publisher wanted it asap.

I had a special define for debugging on my computer which caused a bug that wasn't detected in QA. They only detected it at the end of 4 weeks of validation testing, but it was bad enough to fail validation.

2

Drop a production database by accident instead of a development version. I didn't see what server I was working. Fortunately, the last backup was produced 4 hour ago and the user didn't made a lot of changes on the system, so no big data loss.

1
  • I had a co-worker do this. Only he thought he was dropping his local copy not the prod one. We were all working then heard him go 'OH SH**!!!!' That was a fun afternoon.
    – Tyanna
    Jan 13, 2011 at 19:38
2

The biggest mistake I ever made was not having my work in source control off of my development machine. My hard drive crashed and I lost weeks of work. It's a lesson hard-learned and something I'll never let happen again.

1
  • not my mistake, but related. Company I worked for had one of their webservers compromised, giving an intruder root access to the version control server (DMZ? Whazda?), where he promptly executed an "rm -r /"
    – jwenting
    Apr 20, 2011 at 6:26
2

Using == instead of equals for string comparison in Java

5
  • 13
    That is the biggest mistake you have ever made?
    – Chris
    Oct 25, 2010 at 14:19
  • This is much more of a problem in Java, where == is a reference comparison. In C#, == does a value comparison on strings, same as .Equals() That said, there are some differences, see Jon Skeet's answer here: stackoverflow.com/questions/3678792/… Oct 26, 2010 at 0:56
  • 2
    @Tim: that still doesn't answer how this was "the biggest mistake you've ever made"... Some more context about the consequences at least would be nice... Oct 26, 2010 at 5:28
  • If the most upped answer is about typo, how is my answer here different? If you've been beginner, making this mistake is almost impossible to debug and could frustrate you the whole day.
    – nanda
    Oct 26, 2010 at 7:00
  • @Dean: It wasn't the biggest mistake I've ever made... I was just clarifying the difference in behavior here with C# vs Java Oct 26, 2010 at 13:02
1

I told my Director that just because he is OLD and has more years of experience than I have does not mean that I will respect him. The only thing that I would respect is ABILITY. I didn't have to face the consequences of such a pathetic statement may be because I was a junior Programmer than. Looking back this seems to be my bisggest mistake. Where was my common sense\humility ?? :-(

4
  • 2
    I agree partly with you, just because someone has more experience (in years) doesn't mean that (s)he's a better programmer. There are a lot of programmers out there, which can write 'xx years of experience' on their resume but have no clue what they are doing.
    – Bobby
    Oct 25, 2010 at 12:28
  • They're more likely to have picked up a stupid mistake along the way, like those listed here, that will make the a more careful developer though
    – johnc
    Oct 26, 2010 at 0:00
  • To be clear, the mistake wasn't having that attitude. The mistake was SAYING that you have that attitude.
    – Dan Ray
    Oct 26, 2010 at 12:13
  • I've wanted to say this to many people over the years.
    – Reasurria
    Dec 20, 2017 at 5:43
1

This happened to a colleague this month.

He was fixing a bug that occurred when you send an SMS to a lot of cellphones. Usually these messages won't actually be sent to save on costs but due to a misconfiguration in our database they did get sent.

Cost: one months wage of SMS traffic.

1
  • that's cheap! :)
    – jwenting
    Apr 20, 2011 at 6:28
1

On a manager's request, I copied /etc/sudoers from one machine to another (even though it wouldn't have fixed the problem at hand, but that's beside the point). Unfortunately, I used sudo to move the copied file into place, without noticing that its owner and permissions were completely wrong. At that point, nobody had a root shell open, nobody could sudo, and nobody could log in as root because its shell was set to /bin/false. And the machine was in a remote data warehouse...

1

I wanted to create metadata files for each file in a directory (i.e. for every file somedir/foo.bin, I'd want to create a file somedir/foo.bin.meta). For some stupid reason I decided to create the files by opening and closing a file stream via Python:

for fn in os.listdir(path):
    open(os.path.join(path, fn), 'w').close()

For some even stupider reason I thought it was clever to test this script against a directory on my harddrive with several gigs of personal data on it. Only after running it I realized that I had forgot to actually modify the filename prior to passing it to open and that I had just truncated every single file in that directory (ouch).

Luckily that was on a personal computer and didn't cause any damage to anything important, but I still learned my lesson.

1

I was working for a company who handles one of Sweden's biggest telecom companies customer support. On our server we had some software that regulated the incoming call queue, how many calls we could take and so forth. If it wasn't working we weren't getting any calls and customers weren't getting any support.

Aaaaanyway, I did a (minor) change to the software. I could have waited for a service window to change it but I thought "Hey, it's going to take one minute to restart the service, why bother. Fortune favors the bold". So I restarted it, unfortunately the thread was locked so I couldnt replace it. Starting to panick I decide to quickly reboot the machine (I was remoting in). Unfortunately in my panic I click "install updates and shutdown" instead :P

Needless to say there wasn't much customer support in Sweden to be had for the next half hour before we managed to restart it.

A colleague was worse then me though, he we debugging the automated voice response system one night and rerouted the number to his mobile. Only thing he forgot to switch it back, had the day off the next day and left his mobile at work. We we're wondering why it was ringing constantly that day :)

1

Argh, 11pm, lab shutting, quick, quick...

$ enscript -o midterm.hs midterm.hs
$ submit midterm.hs
Error: submission is empty

God no! FFS. I meant PS you ****!

1
  • Apologies for language style, it is needed for authenticity.
    – Orbling
    Jan 13, 2011 at 22:27
0

Sometime before when coding the JS, I was not able to distinguish between "1" (one) and "l" "small L". Even when I was typing this answer both looks almost same in editor!

1
  • Why the down vote?
    – Gopi
    Oct 26, 2010 at 6:24
0

I made a site templating language, in an early version of which all HTTP parameters were implicitly expanded as variables. This meant that you could write a URL like:

page.vis?body=<body><p>Oh dear</p></body>

And it would do exactly what it looks like it does.

Luckily I have no users (sigh), so I guess it wasn't too much of a security risk.

0

Definitely TSQL mishap - too many TSQL statements in one editor window. Was a bit tired, and people were interrupting me left and right, and I issued a command that updated 47,000 records address, city, state, and zip - oops. Had it fixed in about 25 minutes, though.

Not the answer you're looking for? Browse other questions tagged or ask your own question.