Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

"A/B testing" is defined as "a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates".

The point here, of course, is to know which small single-variable changes are more optimal, with the goal of finding the local optimum.

However, one can also envision a somewhat related but different scenario for testing the response rate of major re-designs: take a baseline control design, take one or more completely different designs, and run test samples on those redesigns to compare response rates.

As a practical but contrived example, imagine testing a set of designs for the same website, one being minimalist "googly" design, one being cluttered "Amazony" design, and one being an artsy "designy" design (e.g. maximum use of design elements unlike Google but minimal simultaneously presented information, like Google but unlike Amazon)

Is there an official name for such testing?

It's definitely not A/B testing, since the main component of it (finding local optimum by testing single-variable small changes that can be attributed to response shift) is not present. This is more about trying to compare a set of local optimums, and compare to see which one works better as a global optimum. It's not a multivriable, A/B/N or any other such testing since you don't really have specific variables that can be attributed, just different designs.

share|improve this question
BTW, the question was indirectly brought up in my mind by these:… and – DVK Oct 28 '11 at 2:06
+1 for reference. – Kyle Hodgson Dec 6 '11 at 5:13
This question appears to be off-topic because it is a "name that thing" question. "Name that thing" are bad questions for the same reasons that "identify this obscure TV show, film or book by its characters or story" are bad questions: you can't Google them, they aren't practical in any way, they don't help anyone else, and allowing them opens the door for the asking of other types of marginal questions. See – gnat Oct 25 '13 at 12:16

I would consider this to be A/B testing as your variables are design A and design B.

You are comparing two or more variables of the same type albeit two complex one.

As long as your goal is to decide whether A is better (you need to define better) than B it doesn't really matter how complex the components of the two variables are.

share|improve this answer

This is still A/B testing.

As you may know, amazon has been doing a major A/B test of exactly this kind on their website. Internally, they refer to it as an A/B test.

(My wife has had the newer, cleaner design for months while I still have their age-old, cluttered design.)

share|improve this answer

The official term is multivariate testing.

Even though you state that they are just different designs, each of those designs is effectively a new variable.

For websites, there are a number of commercially available tools that can aid in the assembly of such tests and the subsequent number-crunching that is required to determine the outcomes.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.