Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm looking for a library that I can use to help mock a program component that works only intermittently - usually, it works fine, but sometimes it fails.

For example, suppose I need to read data from a file, and my program has to avoid crashing or hanging when a read fails due to a disk head crash. I'd like to model that by having a mock data reader function that returns mock data 90% of the time, but hangs or returns garbage otherwise. Or, if I'm stress-testing my full program, I could turn on debugging code in my real data reader module to make it return real data 90% of the time and hang otherwise.

Now, obviously, in this particular example I could just code up my mock manually to test against a random() routine. However, I was looking for a system that allows implementing any failure policy I want, including:

  • Fail randomly 10% of the time
  • Succeed 10 times, fail 4 times, repeat
  • Fail semi-randomly, such that one failure tends to be followed by a burst of more failures
  • Any policy the tester wants to define

Furthermore, I'd like to be able to change the failure policy at runtime, using external knobs or switches that can be removed (along with the rest of the library) prior to shipping the code. (If the component is loaded by a dependency injection framework, we could provide the mock as one implementation in the testing release and withhold it from the shipped build.)

In pig-Java, I'd envision a FailureFaker interface like so:

interface FailureFaker {
    /**
        Return true if and only if the mocked operation succeeded.
        Implementors should override this method with versions consistent
        with their failure policy.
     */
    public boolean attempt();
}

And each failure policy would be a class implementing FailureFaker; for example there would be a PatternFailureFaker that would succeed N times, then fail M times, then repeat, and a AlwaysFailFailureFaker that I'd use temporarily when I need to simulate, say, someone removing the external hard drive my data was on. The policy could then be used (and changed) in my mock object code like so:

class MyMockComponent {
    FailureFaker faker;

    public void doSomething() {
        if (faker.attempt()) {
            // ...
        } else {
            throw new RuntimeException();
        }
    }

    void setFailurePolicy (FailureFaker policy) {
        this.faker = policy;
    }
}

Now, this seems like something that would be part of a mocking library, so I wouldn't be surprised if it's been done before. (In fact, I got the idea from Steve Maguire's Writing Solid Code, where he discusses this exact idea on pages 228-231, saying that such facilities were common in Microsoft code of that early-90's era.) However, I'm only familiar with EasyMock and jMockit for Java, and neither AFAIK have this function, or something similar with different syntax.

Hence, the question: Do such libraries as I've described above exist? If they do, where have you found them useful? If you haven't found them useful, why not?

share|improve this question
    
Did you ever find a solution to this? –  Gary Rowe Mar 22 '11 at 11:29
    
Sorry, I'd forgotten I'd asked the question - and I was able to work around the problem in the first place. I remember I said I was going to edit the question, I guess I'll do that. –  Michael Ratanapintha Mar 22 '11 at 17:32
add comment

1 Answer

You're in danger of introducing test code into production code

I don't know of any libraries that provide this exact facility, probably because it is something that operates at a higher level of abstraction than they provide. That is, you'd normally use, say, jMockit to provide these responses during the test phase.

You definitely don't want to have special "test only" switches in your production code.

Remember that the objective of test code is to verify that the correct paths are selected for given conditions. To that end the mocking frameworks you've mentioned provide you with a repeatable set of test conditions that guarantee a failure when you want it and allow you to verify that the program will react correctly to it.

So how do you go about stress testing with an unstable environment?

Before you start your stress testing you're already confident that your application will behave correctly but there may be load or thread related issues that you've not anticipated.

In that case run up your application in a container that supports access to the test classpath (e.g. Jetty) and provides a test configuration (e.g. Spring) that injects the random failure mock versions of your classes at startup. Your application is then running on a configurable failure platform and you can hammer it with the stress tests of your choice.

If you need to fine tune the failures, simply adjust the Spring configuration to inject fewer or greater mock failure classes, or change the mocks as required.

Bear in mind that mocking frameworks (particularly jMockit) are much slower than a hand-crafted mock implementation of an interface. In the interests of maintaining a representative environment for stress testing, you should probably avoid the framework approach.

Once your testing is over and you've updated your tests to trap all those additional nasties then you can leave the stress test mocks sitting in your test classpath safe in knowledge that no test code has leaked into production.

share|improve this answer
    
I'm not saying you should have test-only code paths in production code. Indeed, in the C apps I typically work with, I would just have debug and ship builds of my app, and turn on the debugging "mock" at compile time with #ifdef DEBUG . If one uses a framework like Spring, then yes, one can use dependency injection to inject the mock code at load time. In neither case is test code going into production. (This is not to say that I'm not convinced by your argument that mocking libraries are the wrong place for this function - I just don't agree with your initial assertion.) –  Michael Ratanapintha Mar 3 '11 at 21:57
    
(And if it was unclear from the original post that "I'm not saying you should have test-only code paths in production code", please point out what's wrong and I'll edit.) –  Michael Ratanapintha Mar 3 '11 at 21:58
    
@crosstalk What triggered my "avoid test code in production" response was seeing the phrase "... using either code internal to the program under test..." in the OP. –  Gary Rowe Mar 3 '11 at 22:29
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.