What have you learned from the bugs you helped discover and fix?

Question

I liked the core of this question, and wanted to re-ask it in a way that made it less about 'fun' and more about 'What do these past mistakes tell us about how we can write and test software better?'

As an SDET, I'm always looking for anecdotes about new and interesting ways that programs can fail. I've learned a lot from these tales in the past, and would like to get that from the intelligent people in this community as well. I'd be interested in hearing what the issue was, how it was caught, if you think there was anything that could have reasonably done to catch it earlier or to avoid the same issue on later projects, and any other interesting lessons you took away from this bug. Please only write about bugs you personally were involved with, ideally on a project you worked on (e.g., no "10 years before I was born, this happened and it was FUNNY!" answers).

Please vote up answers that are thought-provoking or could change how you develop or test in some way, so this isn't just 'social fun'. Try to avoid voting up something just because it was funny.

Edit: Please also focus on concrete examples of bugs or classes of bugs, not general discussion of what you've generally learned. They don't have to have been super-difficult or extremely weird, just something that made you go, "huh".

Answer 1

OK, here is one thats totally irrelevant to desktop-land, because it applies to embedded micro processors where you need to delve into the gory parts of low level interrupt handlers. That should be enough warning.

I was once on a project where the commercial off the shelf compiler and run-time system had been purchased and was being used for a very large very expensive piece of equipment. Two years of development later, and things were heading to a wrap up.

The only trouble was, the code would run for hours and hours (or minutes), and then fail in the most peculiar manner. Most failures where never the same, and the run-times were not the same either. It was not a classic "heisenbug" (where observing changes the effect you see). It was just some time-related strange crash.

I was assigned to the team to fix it. For various reasons unrelated I also resigned, and spent the time working out my notice period on trying to resolve this. Mostly this involved single stepping, or attaching a debugger and waiting for a fail - then trying to extract the machine state from the registers to figure out any common cause.

In the meantime there were many theories proposed (all of them completely wrong - but wind-bags liked to talk and suggest things to look at; so long as they did not actually have to do anything).

An actual bug in the s/w that was developed was ruled out - the complete lack of reproducability and a lot of blood and tears led to the conclusion that identical inputs led to different times-before-crash.

After leaving, a 3-man bug-hunting team spent another 5 MONTHS before eventually finding the cause: The commercial run-time system so generously provided had a bug in the interrupt handler that was the core of the run-time task scheduler. (Yes, it was effectively an RTOS).

If turned out that they had been shipping this for years with the bug in. On this processor, you had to explicitly re-enable interrupts before returning from the handler. Somebody did this, THEN popped the stack, then returned from int. The cure: swap 2 instructions. (pop, then re-enable int, then return).

The lesson from this: It's always good to understand how something works, and spend a little time digging into the guts of things provided as "known working library just use it and don't think".

Since then I've delved a little into the important guts of every embedded environment - processor start up, shut down, interrupt dispatch, you name it. (Even this did not save me years later on another strange one where the processor/ memory interface had a speed mismatch, this took 12 months to track down due to a peculiarity of the processor.)

Answer 2

I've learned you shouldn't put code into production on a Friday afternoon.

Answer 3

I've spent much of the last 23 years doing things that involved having to read and comprehend code so I could fix bugs, add features, or port it to another operating system. Here are a few things I've learned...

Write your code for the next poor soul that has to work on it.

In commercial software, unless you're writing a one-time-use throwaway program, somebody, sometime, will have to mess around in your code. If your code is hard to understand, it will make their life much harder, and their opinion of you will be much less kind than, "Wow, this guy must have been really smart to write such obscure code."

A lot of people think code should be self-documenting. I'm cool with that when it's possible, but it's not always. And I figure more information is always better than less, so I often add comments anyway. If comments save you a couple of minutes a few times a day, that can be the difference between making it home on time and having to explain to your spouse why you're staying late at the office again.

When this really hits home is when you are the poor soul having to fix some software you wrote five years before, and you can't figure out what you were doing.

Avoid cleverness. Or if you must be clever, explain it in comments.

I'm writing this while in the middle of fixing a bug in some old code where file names were getting truncated unexpectedly.

The original author could have worked with the file path strings using strcpy() and so on, and everything would have been fine. But for reasons I can only guess at - I suspect they were a misguided attempt at speeding up the code - he's doing some funkadelic stuff with pointers and remembering locations of specific characters and so on. And yes, he probably saved a few microseconds at the beginning of saving... a 27 GB file to a hard drive, which typically takes about 10-20 minutes. Any optimization he got from that cleverness has about as much practical effect on the program execution time as a pebble on the surface of Pluto perturbs the interior of your pancreas.

And, of course, it doesn't work. Actually, it works great as long as the file name is 29 characters or less. But due to the cleverness, at 30 characters, whack! Chopped off filename. Bug. And it could have easily been avoided by simply using standard C library functions.

There are times when you need to write clever code. If you find a performance bottleneck, sometimes clever code is the only way to fix it. But in that case, make sure it really works, and for heaven's sake, add some comments to explain what you did and why. The next poor soul who has to look at it will thank you.

Be human, and humble.

One of my favorite code comments ever was in the GTK+ sources. I can't find it - they may have fixed the underlying problem - but it basically ran, "This code works but it's fragile. So many people have added so many things to this and related routines that nobody really understands how it all works any more. We really ought to rewrite this someday." I think very kindly of those folks.

Answer 4

It has helped me to see that different people make the same mistakes over and over again. Learning that has helped me to debug issues faster because you start to see the same patterns. It also keeps me from making some of the same mistakes.

The errors I would see the most were related to fence post errors and buffer overruns. A lot of people learn Java in school and seem to have no concept of a NULL terminator; they take "String" for granted. I have learned that if they come from a Java background, then you must give pointer/malloc/free lessons.

The other thing I've really learned is to not be afraid of what I call "Jenga" code (spaghetti code). In the game of Jenga, pieces are taken out of the tower until its so unstable you don't want to touch it. Some source code becomes this "unstable" tower that nobody wants to touch. The right thing to do is to learn what it's doing and fix it.

Answer 5

The first one that comes to mind was related to a tiny 10-line function called deep in a collision detection mechanism for a game engine. Programmer after programmer tried to find an elusive collision detection bug, we reviewed all logic and algorithms, we discussed and verified that this was correct. And still we had this bug.

The solution was eventually found: We corrected an one-line comment that described this function as working on a [begin,end> range to what it really did, working on a [first,last] range. We then corrected the assumption made in the few spots using this function.

Lesson learned: I force myself to completely ignore comments when I read code.

Answer 6

I grew up in Michigan, with long cold winters, where everyone slips on the ice. Maybe I'm just weird, but I happen to not like slipping on ice and falling down. So I, like everyone there, developed the skill of walking carefully on different types of ice, so automatically that we don't think about it.

The harsh lessons of past bugs are like the times when I fell. Off-by-one loops, putting the wrong data type on the call stack, screwed up memory management, and all the other bugs that each took hours to solve, were like whacks of the skull onto concrete or a skinned hand, and with each such agony I developed the skill of double-checking certain kinds of code before moving on. Eventually, I came to habitually double check everything. Such checking is so automatic, it's like driving a car or doing anything I've done all my life.

These days, half of those lessons don't matter anymore, due to smarter compilers and better languages. But the other half keep me from wasting time on the same old stupid mistakes, so that I can in the future waste time on more interesting new kinds of bugs.

It's like when someone says "be careful!" when you set out to do something new, but you don't know exactly what it is you should be watching out for. The experience of solving a difficult bug teaches you an example of what exactly to be careful of, at least in the land of coding.

Answer 7

A while ago, I had some CPU-intensive data manipulation code that would run quickly under certain circumstances but far too slowly under others. I checked for several different problems and didn't find anything. I ignored the obvious possibility--that a certain callback was running every time the data was updated--because I had disabled that.

After a while I ran it through a profiler, and to my great surprise I found that the majority of the time was spent in the callback that I was sure I had disabled for this operation! A bit of quality time with the debugger revealed that, buried within the library I was using, there was a check that, under certain circumstances, would ignore the callback being disabled and fire it anyway!

Once I knew what was happening, it wasn't too hard to code up a workaround. But I learned that it's important to always verify your assumptions with hard data, and always check the obvious things first, even if you think you know that can't possibly be the problem.

Answer 8

One lesson I've learned from a number of bugs is that threads should avoid sharing state, as much as possible. As a corollary, a single thread relying on inversion of control is often significantly more maintainable than multiple threads that can block. I've run into bugs like:

• System eventually crashes on buffer allocation, or buffer release.
• Media decoder task can't easily be killed on EOF detection from input.
• Quickly closing / opening connections to different servers causes crash.

Most of these were fixed by refactoring from blocking code executing in several threads, to an architecture in which the threads were separated according to the state that they needed to work with, and that communicated with each other strictly through message passing. When new bugs were found in the refactored code, they were almost always much easier to reproduce than the ones we used to have.

Answer 9

I've found that people who claim that it's a compiler bug always have a bug in THEIR code and it's not the compiler's fault. I'm sure there are exceptions, but the odds that you will see one are infinitesimal.

Answer 10

If you allocate a single byte, allocators typically round up to a word or more. A subsequent use of a full word goes undetected. You need memory debuggers like Valgrind to reveal it. Only after you use several words instead of bytes, you get runtime errors.

Answer 11

If you're going to set up several different enums, especially if they contain names that are very similar, then be sure to typedef the enum and use the proper named enum types in all your prototypes and variable declarations.

If you assume that using an enum is just like using an int, only you get to use words instead of numbers, then you're using them the wrong way.

Answer 12

Make sure you can't be reached until a week after the code goes life. That way someone else has to suffer the stress of fixing the critical production problems.

So as soon as the life date is known, book a vacation starting a few days before that date and ending at least a week after it. Book it to some far away place so it's cheaper for the company to have some other slob solve the problems than to recall you and pay for the long distance airfare for the flight home.

p.s. I've never actually done this myself, but have more than once been the guy who had to solve the problems because the person who should have done it was on vacation.

Answer 13

I learned to actually stop and take a break. Sitting there starting at the computer waiting for the bug to pop out and say "HEY IT'S ME" 99.99999% of the time isn't going to happen (maybe in the future.....le sigh...) but It's best just to get up and do something else. Maybe it will come to you...maybe it won't. But either way coming back on a fresh mind helps alot.