Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Recently at work we've been having some differences of opinion with regard to Parameterized testing. Normally we use a TDD-style (or at least try to) so I understand the benefits of that approac. However, I'm struggling to see the gain parameterized tests bring. For reference, we work on a service and it's libraries which are exposed via a RESTful interface.

What I've seen so far is tests that are, at least using JUnit within Eclipse:

  • Lacking in detail - when a test fails it's very hard to see the parameters which caused it to fail
  • Often complicated to create
  • Tend to be created after the code has been written - strictly not a drawback as such but do people set out with parameterized tests in mind when they start a piece of code?

If anyone has any examples of where they are really useful or even any good hints for using them that would be fantastic. I want to make sure I'm not just being obstinate because I personally don't choose to use them and see whether they are something we should consider being part of our testing arsenal.

share|improve this question
1  
The problem is not with the idea but with the clunky library. In C# the syntax is way friendlier, when you use say MbUnit. Yes, it is a good idea. Add your own code to make this process easier - read stuff from files - whatever works. Also look at how MsTest handles this. –  Job Jul 10 '11 at 14:07

5 Answers 5

up vote 4 down vote accepted

The problem with testing any piece of software is that complexity blows up pretty quickly. The fact is, you can't test all possible combinations of parameters passed to your methods. Phadke advocates a Design of Experiments (DOE) approach, which allows generation of the likely list of parameter values that need to be tested.

The idea is that, even though you are not testing exhaustively, most defects cause a "fault region" to occur rather than an isolated point fault. The DOE approach Phadke advocates using orthogonal arrays samples the parameter space finely enough to hit all possible fault regions.

Isolated faults will probably not be identified, but these are generally fewer than fault regions.

The DOE approach gives you a systematic way of choosing the parameter values to vary.

share|improve this answer

They can be useful for ensuring that your code handles not just the happy path, but also the edge cases. After you know your code works with normal variables, parametrize the test case and make sure nulls and 0's, empty strings, large numbers, long strings, weird Unicode characters, etc., also work fine.

share|improve this answer

There are at least two flavors of parameterized tests, at least in JUnit 4.8. Those are: Parameterized Tests (@RunWith(Parameterized.class)) which requires a data source, which generates/reads predefined parameter configurations, and Theories (@RunWith(Theories.class)) which, given one or more set of possible inputs per argument type can exercise specification of given methods. It looks more-less like this:

  • specify some possible values (@DataPoints) for string arguments (like null, empty string, non-empty string, really long string)
  • specify some possible values (@DataPoints) for Animal class arguments (like null, Dog instance, Cat instance, Bird instance)
  • prepare @Theory which accepts a String parameter and an Animal parameter. it will get executed with every possible combination of the possible parameter values (in given example that'd be 4x4 = 16 combinations, including (null, null))
  • if method under test cannot accept some combination, use Assume.assumeThat static imports to filter out invalid combinations (eg. when you want to check method's behavior for non-empty strings, one of the first lines would be "assume that is not null"

As written before - it doesn't make sense to test every possible combination of every method (it explodes test sets, imagine testing a method with 5 parameters, each one having just 5 possible values: 5**5 -> over 3000 test runs!), but for mission-critical methods (like API methods) I'd encourage it, just to be on the safe side...

share|improve this answer

Generic example:

  • Methods with string arguments. Use parametrized tests to test different inputs and their expected outputs. Is much more practical to have a list of pairs (input, expected) than to write a TC for each pair.

  • Apply same scenario on different arguments. We have a scenario that works with the Animal object and the have a lot of subclasses such as Dog, Cat, Bird. Create a list of the available animals and test the scenario on them.

Concrete for webservice:

  • From the string arguments example above. Test what happens with different arguments of the same type but different values.
share|improve this answer

One case where I use lots of parameterized tests in a TDD-ish manner is writing parsers -- I can start with a list if input and expected output and then write the code so it passes all test cases.

But I've seen a few horrors of parameterized testing. No, Virginia, your test suite should not need it's own unit tests.

share|improve this answer
    
Ideally parametised tests should be of the form "does item(n) in the actual output match item(n) in the expect output" or similar, and in that case no testing needed. But for anything more complex I'd prefer to seen a clean parametised test or two with their own test cases than the usual handwaving "my (test) code is obviously correct". If that was true you wouldn't be writing test cases at all. Obviously it's possible to go overboard with the complexity and I'm not arguing that there's no line, but I think there are cases where testing tests is a good idea. –  Мסž Mar 28 '11 at 22:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.