Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Yesterday, Twitter was hit with a "Cascading Bug" as described in this blog post:

A “cascading bug” is a bug with an effect that isn’t confined to a particular software element, but rather its effect “cascades” into other elements as well.

I've seen this kind of bug, on a smaller scale of course, on some projects I've worked on. They can be difficult to identify in dev/test environments, even within a test driven development environment.

My questions are...

What are some strategies you use, beyond the basic TDD and standard regression testing, to identify and prevent the potential trouble points that might only occur in the production environment?

Does the presence of such problems indicate a breakdown in the software development process or simply a by-product of complex software systems?

share|improve this question

4 Answers 4

Without knowing the specifics of the problem (and it would be instructive to all of us if Twitter could provide more info on this, though I understand why they wouldn't) I imagine it's one of those butterfly-tornado problems that are near to impossible to predict, and that only manifest themselves in a real-world situation, where you have a realistic number of users and configurations.

Yes, integration testing should have found this, and I imagine Twitter have a 'test lab' where they simulate millions of simultaneous users sending messages. However, some particular configuration, some odd setting, some happen-chance of barely-interconnected events, conspired to create a problem.

I think Twitter showed their maturity by immediately rolling back, and not attempting to hack their way out of the problem, as some might have been tempted to.

I think the only thing you can really do from things like this are to learn from them - to make the integration tests better, if possible. I don't think nessecarily there's anything they didn't do which they could have done.

I'm in the UK, and currently witnessing the effects of a cascading bug on NatWest, one of the largest banks here. Now, banking software is really well tested, yet these bugs still happen.

I think, at the end of the day, we do the best we can with the best tools and practices we know, but software is a complex and complicated thing, and sometimes bugs will happen. How we deal with them when (not if) they happen defines our competence as engineers.

share|improve this answer

Dude, don't fear the [Chaos Monkey]! -- Bill S. Preston, Esq.

Seriously, as Netflix, Amazon Web Services, Twitter, and others have found out, these problems arise spontaneously in complex systems. There is some research to suggest that it is impossible to know for sure that such problems do not exist (i.e., that they are akin to Turing's Halting Problem). The only "best practice" seems to be to randomly knock parts offline frequently, and make sure the system as a whole continues to function. It's painful at first, and is best done on non-production systems until you've figured out how to handle many of the resulting failures, but it seems to work. Netflix calls the tool that does this on their system "the Chaos Monkey" :-)

share|improve this answer
    
I agree, but in most organizations you have to have some level of CYA to keep your job after such a disaster. Basically, if you've covered the "known unknowns" you should be OK, well, most of the time. –  jfrankcarr Jun 28 '12 at 14:48

I would argue that it is both a by-product of complex software systems AND it indicates a breakdown in the software development process. Those two states are not mutually exclusive.

Defensive programming is the catch-all term to help prevent these types of bugs. Some of the techniques from concurrent programming and not trusting any outside variable (I'm exaggerating) are useful as well.

From my experience, system level code reviews are best at catching these types of issues. By "system level", I mean someone who is thinking about the entire system / module instead of just the code that changed. First, the reviewer should spot when a variable / object is used in a way that isn't guaranteed to remain consistent. Second, the reviewer should catch the situations when a variable is setup to be used in that situation. The former is easier to spot than the latter. Both require an experienced reviewer who understands the module(s) in question.

share|improve this answer

I suspect the code in question wasn't covered by unit tests. (or at least not adequately). Good unit tests cover handling exceptions gracefully as well as normal conditions.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.