Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I have recently been looking at some testscripts which looks a bit like

...
try {
  receiveSomething();
  // something was received even though it shouldn't
  failTest();
} catch (TimeoutException e) {
  // nothing should be received
  succedTest();
}

The problem I have with these types of tests is

  1. They are not deterministic. You don't know if nothing was sent on purpose or if everything has crashed.
  2. Its very hard to simultaneously test something else, which might actually, in this case, send something.

My thoughts are one the one side, how can this types of tests be designed better, and secondly can this be an indication of a bigger design smell of the software that is being tested?

EDIT

To clairify, these test scripts are used for black box-testing of event-based, complex software, not unit testing, and my feeling is that the 'doing nothing' event is a very ambiguous one. :)

share|improve this question
    
as soon as test and code allow for a proper mocking of the data source used in receiveSomething(), there would be nothing wrong with this snippet of test –  gnat Feb 14 '13 at 12:14
    
@Theodor: you may edit your question to make more clear what you really meant - seems most of the other folks misunderstood that. –  Doc Brown Feb 14 '13 at 14:55
add comment

4 Answers

up vote 9 down vote accepted

From what you wrote, I guess your situation is more like this

startAsynchronousComplexOperation();
try {
  receiveSomething(); // listen for events from that async operation above

  // something was received even though it shouldn't
  failTest();
} catch (TimeoutException e) {
  // nothing should be received
  succedTest();
}

and the object under test is not "receiveSomething", but the asynchronous operation. Thus the outcome of that test depends on the speed of the test machine, if a lot of other processes run on the test machine etc., and this is non-deterministic.

IMHO, this is indeed a design problem, and not just a "test smell". If your "complex operation" crashes, or just "takes too long", not only your test will give a "false negative". Think about how to deal with that situation in production. Is there no requirement to determine for sure that your "complex operation" is still alive and did not crash? So you may think about adding some kind of "watchdog" or "keep-alive-packet-sending" mechanic to your program to get control over that. And once you have such a mechanic available, you can use it in your tests like this:

startAsynchronousComplexOperation();
while(isComplexOperationStillAlive())
{
    try {
       receiveSomething(); // listen for events from that async operation above
       // something was received even though it shouldn't
       failTest();
   } catch (TimeoutException e) {
   }
}
if(didComplexOperationCrash())
    failTest();
else
    succedTest();
share|improve this answer
1  
Interesting leap and could be spot on, but it doesn't answer the question, as they are both currently written. –  pdr Feb 14 '13 at 13:40
1  
@pdr: I am pretty sure it answers the question as it was meant. "receiveSomething" obviously waits for an event - from where, if not from a part of the code the OP forgot to add to the example? And why else does he think the test is non-deterministic? –  Doc Brown Feb 14 '13 at 14:17
    
Exactly. These scripts are testing event-based, complex software :) The 'doing nothing' event is a very ambiguous one. –  Theodor Feb 14 '13 at 14:32
1  
Upvoting because of correct assumptions about the question :) –  Silver Quettier Feb 14 '13 at 15:09
    
+1 for good answer to question he meant to ask and eventually did –  psr Feb 14 '13 at 20:01
add comment

They are not deterministic. You don't know if nothing was sent on purpose or if everything has crashed.

It doesn't matter, in the context of this test. All that matters is that, under the preconditions defined, nothing is received. Other tests can simulate other preconditions and check their results are as-expected.

Its very hard to simultaneously test something else, which might actually, in this case, send something.

Good! You shouldn't try to plug other tests into this test. This test is doing exactly what it should be doing. One set of preconditions => One result.

This is the very definition of determinism.

share|improve this answer
    
I see your points here. Ok, so testing a feature like this might be ok, if it was designed this way. But would a more robust design be to explicitly and not implicitly communicating that nothing was to be expected, e.g. send a State.PASSIVE message? –  Theodor Feb 14 '13 at 12:17
    
Maybe. That's a code decision rather than a test decision. Does it make your CODE more robust? If so, do it and test it accordingly. If not then test what you have. –  pdr Feb 14 '13 at 12:30
    
@pdr: I guess the situation of the OP is a little bit different, see my answer below. –  Doc Brown Feb 14 '13 at 13:32
    
+1 for good answer to question originally asked –  psr Feb 14 '13 at 20:00
add comment

They are not deterministic. You don't know if nothing was sent on purpose or if everything has crashed.

A deterministic algorithm as defined in Wikipedia is as such:

is an algorithm which, given a particular input, will always produce the same output

But the algorithms to build the test hardly matter, pretty much because it is supposed to test the proper state after invoking a specific behavior. In other words, the only thing that needs to be deterministic is the code module that you are testing. The test itself is actually definining what the determinination should be.

Its very hard to simultaneously test something else, which might actually, in this case, send something.

As others pointed out, that is a GOOD THING. If I have a test for module A, that has dependencies on Modules B and C, then if TestModuleA fails, how do I know it is ACTUALLY an issue in module A?

Your test is supposed to just verify that Module A provides the correct state for invoking an operation of A. You should have seperate tests for the dependencies to verify their correct behavior too.

share|improve this answer
add comment

Very hard to answer this without some context.

Were the test case to be for the function point ("System Will not respond to any request after 'Quiesce' command issued") then yes this would be a perfectly valid test case.

Although I would have specified that a valid response was received immediately before the shutdown request - but these type of "operational" tests are very difficult to automate as its unlikely you would be able to issue such system commands from an API.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.