What is the best way to go about testing that we handle failures appropriately?

Question

we're working on error handling in an application. We try to have fairly good automated test coverage. One big problem though is that we don't really know of a way to test some of our error handling.

For instance, we need to test that whenever there is an uncaught exception, a message is sent to our server with exception information. The big problem with this is that we strive to never have an uncaught exception(and instead have descriptive error messages).

So, how do we test something what we never want to actually happen?

Answer 1

Simulate the failure / exceptional situation.

Your mock objects are crash test dummies and you already know what can possibly go wrong, it shouldn't be hard to emulate it (you know because you've already prepared for it, if you don't, well, you'll soon find out ;). And... it's always amazingly fun to physically unplug your database server while your processes are running. Doesn't really scale as a testing method, but it's awesome even if you do it just once.

Consider this trivial / overly simplistic example:

• Class A does something, and throws an OMGSomethingBroke exception if something goes awry.
• Class B calls Class A, and in case something breaks, catches the exception and logs it (or whatever).

What you need to do is create a mock of Class A, let's call it Mock A that does nothing more than throw the OMGSomethingBroke exception. Mock A and Class A should implement the same interface, and that's about it. You've emulated the failure, and you can now move on to testing Class B. Have it call Mock A instead of Class A, and test if it handles the failure gracefully.

Testing can be exceptionally simple if your design is solid (yes, both puns intended).

---

Just found a very interesting Udacity course, Software Testing (CS258) - How to Make Software Fail:

When writing software, destruction can be just as valuable as creation. Learn how to catch bugs and break software as you discover different testing methods that will help you build better software.

The course has a wider scope than your question, obviously, but the core message in its summary applies: Breaking stuff is fun! (Well, and a good way to test software).

Answer 2

This is one of the many reasons why modularization is a good idea.

Your test is supposed to test the behaviour of one bit of code, and one only. In this case, it's the code that sends email when a different bit of code generates an error. So what you do is hook up a mock version of the other bit of code that does create an error, to your actual, production-grade error handler, and then verify that it does send email. (You can also replace the email sending code with a mock version, too, so it doesn't actually send the mail but just reports that it would have done so - makes the test much easier and faster.)

That way, everyone is happy: Your error handler is tested, your production code never causes errors (that you know of), and you still know hat would happen if it did. If your code base doesn't support such flexible hook-ups, well this is one major reason why it should.

Answer 3

So, how do we test something what we never want to actually happen?

Sometimes, you can't, sometimes you don't want to. Boeing doesn't intentionally send a prototype jet into a nose dive 100 feet above the ground to test various "can't happen" type events. A nuclear power plant isn't going to intentionally drive the power plant past the point of no return to see if the tertiary backups work.

This one seems testable. Throw an exception you know isn't caught. Except via a catch-all (e.g., catch (...)), you almost certainly are not catching an exception of type BogusExceptionForTestPurposesOnly. So throw it in some test code. Just make sure this test code never makes it's way into the deliverable.

Answer 4

Add a special "break the rules" request to debug builds and invoke it. For example add a "uncaught throw" request and make it throw a unique condition you think should be uncaught.