Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Only 1% of our clients use particular features or scenarios. Do you think we should spend as much time testing these features and scenarios as we spend on our central features and scenarios, or we should test mostly the "happy path" for these features?

UPDATE

Maybe I should have asked the question in a different way. Do you think a company needs to spend more time testing the central features of its products? For example I assume the Windows 8 team tested the the boot sequence more in depth than they tested the driver for a very uncommon graphics card. The cost of a bug in a boot sequence is huge since no one would be able to use the OS. But a bug in a driver of an obscure GPU will hurt only a small client base.

share|improve this question

closed as primarily opinion-based by gnat, mattnz, MichaelT, Kilian Foth, GlenH7 Mar 18 at 14:09

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

    
If the feature is not used enough by enough customers to be economic to test, is it economic to fix it? Consider the cost to fix before deployment vs after deployment. Ask the product owner / sales / marketing department - they are the only ones who can give you the answer for the product and feature. –  mattnz Mar 17 at 2:13

3 Answers 3

It comes down to cost/benefit.

Except for trivial programs, it is usually not physically possible to test all the permutations of possible inputs for program correctness. (If you could, you could replace your code with a simple look up table).

Since you can't test everything, you (or management) need to prioritize. I'd probably do something like:

priority = odds of hitting it * the pain you'll see if the failure occurs

  • "happy path" - 80% of your customers use these features every day. A failure here will be noticed by a lot of people, very quickly. High chance of hitting it, not sure about the pain (depends on the feature).
  • Catastrophic failures - low chance of hitting it, but extreme pain if hit
  • odd ball features - low chance of hitting it, and hopefully low pain if hit.

Your "1% feature" could fall anywhere in the priority scheme. If this is your biggest customer and they represent 1/2 the income of the company, their 1% bug might have a higher priority than all 500 other customers combined.

share|improve this answer

Let's pretend I published an encyclopedia. Only this one is special. I went and replaced the definition for sarsaparilla with a picture of a monkey-butt.

Most of my users will be happy that they can look up popular things like monkeys and butts, but the one person who desperately needed to know what a sarsaparilla was would find themselves in a bit of a pickle.

So it depends. Are you okay with your program potentially crashing on 1% of your user base when they misclick something? Are they likely to do so? If they are, you might want to spend some time in that section of the program.

share|improve this answer

You should test all features of your application. You don't want to present any failures to your clients just because you don't have the inclination to test as thoroughly as you should.

That said, there is a very practical purpose for not testing those features: time. If these are features that haven't been touched by the new code (either directly or by supporting code), it is highly unlikely, but not impossible that they will be broken with a new compilation.

UPDATE

Let me be very clear, I am always going to recommend testing everything in an application. Management may not allow me the time necessary to fully test. If that is the case, then I would put more emphasis on testing the most used components of the system.

share|improve this answer
    
so... you're saying... maybe? (and not sure why the last part is focused on compilation) –  haylem Mar 16 at 19:28
    
No, I am say you should test everything if you care about the possibility of failures occurring in front of you clients. –  Adam Zuckerman Mar 16 at 19:30
    
yeah but even after your update I don't see much added-value: everybody will always "recommend testing everything". You seem to simply say what the OP already knows: that given time constraints, you may not be able to test everything and need to prioritize. To me that just seems like a "show of hands" answer with you choosing one side, but without much argumentation (note that I haven't voted you up or down, I'm just curious how you'd develop your answer). –  haylem Mar 16 at 19:35
    
I think my first sentence pretty much answers his question. My second explains my position of why. If you want a really long winded explanation, I can provide that. –  Adam Zuckerman Mar 16 at 19:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.