Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Jeff Atwood recently wrote a blog post about Netflix's implementation of a "Chaos Monkey". It is a very high level article. I am curious if anyone has actually implemented this technique for testing a system.

I guess what Im really trying to ask is: What strategies do you implement to ensure your architecture can survive a part of the system crashing ?

share|improve this question

migrated from Jul 1 '11 at 6:34

This question came from our site for professional and enthusiast programmers.

And perhaps, which strategies do you use to test such an architecture? – rlb.usa Jun 29 '11 at 17:07
...Well, I guess we're waiting for someone on the netflix team to come along and answer this lol – Earlz Jun 29 '11 at 18:00
I'll probably toss a bounty on this. Could have some very interesting anwsers – Robotsushi Jun 29 '11 at 18:24
Agreed... But belongs on programmers, not SO. Voted to migrate. – djacobson Jun 29 '11 at 18:36
up vote 5 down vote accepted

Isolation and graceful degradation are the general strategies. (Another term you might see that is similar to isolation is decoupling, though I tend to see that on a smaller scale, such as in OOD/P. The concept is the same.)

You isolate different parts of a system from each other so that if one is down, the others can still respond to requests. Like the Netflix blog said, if searching wasn't working, streaming would still be fine. This just means that searching and streaming where separated enough that a bottleneck or incapacitation of one did not affect the other.

With graceful degradation, if the best implementation of something is not available, you have something else fill in. Again from the Netflix post, they have a system for looking at the things you've watched and liked and then working out personalized recommendations of other things to watch. If that system is down, they fall back to showing recommendations of things that are popular overall. The point is to have a Plan B, Plan C, etc. to do or show something when Plan A fails rather than showing nothing or an error.

One common client-side example of graceful degradation (whether implementation is common or not) involves the use of javascript on websites. If the browser's javascript is disabled or simply unavailable, the site's pages should still operate successfully without it. It may not be as fast or slick, but it should still work rather than become unusable.

These are very general ideas, though. Just about every project would implement them differently, depending on the services and subsystems they provide, and the dependencies between them.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.