Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Many books on startups/kanban strongly advocate the use of AB testing to validate product features.

I haven't had any experience with this but it sounds like a great idea for some projects.

My question is, how does one go about AB feature testing in practice?

Say you want to test how registered users interact with a specific feature.

Do you create tables to set what batches of users see what features? What about non-registered users. ect.

Say it's determined the feature isn't needed based on your tests. Do you go back into the code base and cut that code out?

share|improve this question

closed as too broad by Robert Harvey, Corbin March, GlenH7, gnat, Dan Pichelman Sep 12 '13 at 20:55

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs. If this question can be reworded to fit the rules in the help center, please edit the question.

Put the question you are asking in the title, not your Google Search. –  Robert Harvey Sep 11 '13 at 17:02
@JeffO It's not so much a UX question as how do developers handle/manage turning on and off different feature sets for AB testing. –  Andrew Walters Sep 11 '13 at 18:32
@AndrewWalters - do you want a code example? –  JeffO Sep 11 '13 at 21:05
That would be nice, I have ideas but they seem messy –  Andrew Walters Sep 11 '13 at 21:47

3 Answers 3

Since you asked about non-registered users, you can alternate between each user that access the feature. Doesn't matter if they're registered or not. This provides a little more randomness (Not that you can't randomly put users into two groups.) and accounts for the frequency each user accesses your app.

You could also A/B test each individual user depending on what type of features you're examining. Do they prefer using the menu bar or right-clicking for some other menu? Both are available to everyone, but you can track which feature gets used the most.

There's no need to keep unused code whether it fails an A/B test or not. See Here

share|improve this answer

One simple way is to randomly assign people to each group the first time they hit the page, and store the group they were assigned to in a cookie. People may end up switching groups if they clear cookies, or use other browsers, but in many cases the numbers doing this may be low enough not to matter.

The nice thing about this is that you don't have to mess with real user registration or keep user level records. All you really need are reports that show behavior grouped by the value of the cookie.

The downside is that you only have statistical information about each group. You can't go back and figure out if a particular user was in a particular group. But if you're just testing which version of a particular feature gets better results, this is likely fine.

share|improve this answer

I think my original question was too generic, like "How to do TDD".

I was hoping for some real world experience based answers like "I use xyz product for testing" or "I've implemented testing using this design" ect.

Here's what I've found.

There are several companies offering solutions for enterprise users:

And a few that seem to target smaller business

I haven't tested these tools but most have these

  1. Heatmaps: Nice to have but there are other tools that do the same
  2. AB Testing

The testing is done by creating a bit of HTML/javascript in a WYSIWYG editor on their site and pasting some javascript into your page. The javascript will request a content block and render it to your page, similiar to how adds are often served.

At first I thought these would be limiting because you could only test small content changes. Marketing slogan A vs Marketing slogan B ect.

Now I realize if I want to test more complicated changes involving userControls ect. I can render both to the page and use the test's javascript bit to hide/show one.

I think when the time comes I'm going to go with one of these services instead of coding my own testing framework. Will save the hassle of keeping track of what users see what, how to render the different content ect.

If you are considering building your own, here's some slides I found of things to consider: AB Testing Framework Design

And finally some general guidelines on testing from smashing magazine


  • When doing A/B testing, never ever wait to test the variation until after you’ve tested the control. Always test both versions simultaneously. If you test one version one week and the second the next, you’re doing it wrong. It’s possible that version B was actually worse but you just happened to have better sales while testing it. Always split traffic between two versions.
  • Don’t conclude too early. There is a concept called “statistical confidence” that determines whether your test results are significant (that is, whether you should take the results seriously). It prevents you from reading too much into the results if you have only a few conversions or visitors for each variation. Most A/B testing tools report statistical confidence, but if you are testing manually, consider accounting for it with an online calculator.
  • Don’t surprise regular visitors. If you are testing a core part of your website, include only new visitors in the test. You want to avoid shocking regular visitors, especially because the variations may not ultimately be implemented.
  • Don’t let your gut feeling overrule test results. The winners in A/B tests are often surprising or unintuitive. On a green-themed website, a stark red button could emerge as the winner. Even if the red button isn’t easy on the eye, don’t reject it outright. Your goal with the test is a better conversion rate, not aesthetics, so don’t reject the results because of your arbitrary judgment.


  • Know how long to run a test before giving up. Giving up too early can cost you because you may have gotten meaningful results had you waited a little longer. Giving up too late isn’t good either, because poorly performing variations could cost you conversions and sales. Use a calculator (like this one) to determine exactly how long to run a test before giving up.
  • Show repeat visitors the same variations. Your tool should have a mechanism for remembering which variation a visitor has seen. This prevents blunders, such as showing a user a different price or a different promotional offer.
  • Make your A/B test consistent across the whole website. If you are testing a sign-up button that appears in multiple locations, then a visitor should see the same variation everywhere. Showing one variation on page 1 and another variation on page 2 will skew the results.
  • Do many A/B tests. Let’s face it: chances are, your first A/B test will turn out a lemon. But don’t despair. An A/B test can have only three outcomes: no result, a negative result or a positive result. The key to optimizing conversion rates is to do a ton of A/B tests, so that all positive results add up to a huge boost to your sales and achieved goals.


share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.