Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Let's say I have 4 (A,B,C,D) parameters with 3 possible values, also 81 unique combinations. With e.g. orthogonal array, I will end up with 9 test cases, each combining 3 pairs. But that means that if one test case fails, I do not know which combination in the test case is faulty, right? Whether values of A,B or A,C or BC. Just wanted to be sure I understood it well.

edit: Corrected (thanks GlenH7).

share|improve this question
up vote 2 down vote accepted

Have a look at the Wikipedia article on Orthogonal Array Testing as well as the linked PDF from 51 testing

Some of your numbers are wrong.

Original version: 3 parameters (factors) with 4 values (levels) each, results in 4^3 combinations or 64 unique combinations, not 81.

Updated version: 4 parameters with 3 values each results in 3^4 combinations, or 81 as you asked.

Original: The minimum number of test cases to check all the pairings is factors * levels or 12 runs, not 9. Note that if you had differing number of levels, you would need to calculate that in a different manner. The explanatory PDF goes into more detail.

Update: The minimum number of pairings drops to 9. However, if only one test fails, you'll still be able to identify the faulty pairing by comparing it against the ones that cleared.

I think that the incorrect numbers you were using threw off your reasoning and your question about knowing which pair failed. With the those 12 minimum runs (also listed below), you will know the pairing that failed.

Original version:
0 0 0
0 1 1
0 2 2
0 3 3
1 1 0
1 2 1
1 3 2
1 0 3
2 2 0
2 3 1
2 0 2
2 1 3
3 3 0
3 0 1
3 1 2
3 2 3

Updated version:
0 2 0 0
0 1 1 1
0 0 2 2
1 0 1 0
1 2 2 1
1 1 0 2
2 1 2 0
2 0 0 1
2 2 1 2

share|improve this answer
Yes, I put it vice versa,should ve 4 variables with 3 values, of course, sorry! But the point stands the same. – user970696 May 1 '13 at 17:42
Do you think this is applicable to performance testing? If P(A, B, C) is my "performance function" - e.g. how much RAM / CPU it took to to execute a function with given parameters A, B and C, then would having this result help interpolate the answer for any possible combination of A, B and C? – Job May 1 '13 at 18:26
@Job - orthogonal array testing is based upon the premise that defects appear most frequently during pairing of Factors (parameters) and varying the values in those pairings. You could use this approach to more quickly discover if the pairings of the parameters were associated with performance issues. But you (obviously) cannot extend this approach to prove the absence of performance issues. It's simply an approach that's more comprehensive in its coverage than random selection of test cases. It's not exhaustive coverage. – GlenH7 May 1 '13 at 18:49
I've used this technique to find a particularly hard-to-resolve embedded device bug. We had several different possibilities, and enumerated a very similar set of possible scenarios. We ran 9 of the devices overnight with the suggested setup parameters, and BAM! we had isolated our problem very quickly. – Peter K. May 1 '13 at 20:00

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.