Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

If you only had time to test 5 features of a product that had 10 features total, how would you decide which features to test?

Another Question:

When is it okay to ship a product with a bug

share|improve this question

migrated from stackoverflow.com Mar 11 '11 at 7:19

This question came from our site for professional and enthusiast programmers.

    
How big is the company? If there is a business/product and test team then, these will be their responsibility. –  StuperUser Aug 12 '11 at 17:51
add comment

10 Answers

up vote 4 down vote accepted

My first question would be what the purpose of the product is. Until I know the purpose of the product, I wouldn't know what to prioritize. Second question would then be to find out what the milestone goals of the release, again to figure out what is important.

Then from there, I would test the features that are most important for accomplishing the purpose of the product. If the product is a word processing program, features involving input/output would be high on the list. Saving and usability features would also be high. Things that would be down on the list could be optimization code.

share|improve this answer
add comment

For both questions, consider cost and benefit.

If there are ten features total, how much benefit is gained by testing each one? If only three of those ten are new features, I'd probably want to test those three and the two most important existing features, or the two existing features that are most likely to be impacted by the new code. As others have said, there's not enough information to give a specific answer, but that's the approach any sane person would choose. Also, consider the cost and benefit of slipping the schedule long enough to allow proper testing of all the features.

The same goes for shipping with a bug. All products ship with bugs, and most products ship with known bugs. The question is: what's the impact of leaving a given bug in the product. How much will it cost to fix it? How long will fixing it take? How much could not fixing the bug cost in terms of sales, reputation, liability, etc.? Also, what can be done to mitigate any of those costs?

share|improve this answer
add comment

I think there is no point trying to give a general answer to such questions (in other words, a general solution would be less than ideal in most of the cases). As others have noted, one should know more about

  • the individual features,
  • their importance for the user and
  • their implementation, especially its level of risk.

Based on this, one can decide which features to focus the tests on if testing resources are scarce.

However, one must also know

  • the mission/business/life criticality of the product,
  • its estimated user base and
  • its method of deployment.

E.g. for an experimental, noncritical product used only by a couple of users in-house, including an auto-update facility, it can be acceptable to ship untested features. If the users understand this is experimental stuff, they tolerate bugs better, and patches will be installed automatically. However, for a life critical health care product built into custom hardware, without an easy way to deliver software updates, shipping a feature without adequate testing would be a fatal mistake: it could lead to loss of life, and probably the demise of the producer company itself via the ensuing lawsuits.

share|improve this answer
1  
+1 on emphasizing when risks outweigh the benefits. –  Aditya P Mar 11 '11 at 8:37
add comment

I would ask the business team for the product to outline risk, impact and other factors and then for them to decide the priority for which to test.

These questions involve something that is not the responsibility of a programmer.

A Product Manager would decide testing priority and also when to release with known defects.

share|improve this answer
    
Why the downvote? A programmer should not be deciding the scope of a release, it is the responsibility of a business/product team, with the input of development. –  StuperUser Aug 12 '11 at 17:48
add comment

As a developer neither of those questions is up to me. My job is to develop a single use case at a time delivering it tested. It is up to a project manager to ensure my efforts are on the most important task at that time and when a bug is sever enough to keep software from shipping.

If a bug is that severe then I should be fixing it and not working on new code which may introduce a new bug.

share|improve this answer
    
That might not be a bad way to dodge the question, but there are two risks: 1) it looks like you're dodging the question; 2) a smart interviewer could easily rephrase to move the decision into your domain: "While debugging, you discover a longstanding flaw in the guts of a product. How do you decide between working around the flaw but leaving it in the code and fixing the flaw?" –  Caleb Aug 13 '11 at 19:32
add comment

Interesting no one is thinking about value to the business of the software / features. I think an interviewer would be looking for someone who understands that commercial software development (in general) has to provide some sense of revenue creation / cost avoidance to be worthwhile. What is the

<business speak> value proposition </ business speak>

behind the software and its function would be one sensible response to the question.

share|improve this answer
add comment

If the point of the task is to choose five out of ten features, then estimate either:

  • what features are most important or
  • what features cover the most lines of code.
share|improve this answer
add comment

I would test the 5 features that are most likely to be used the most, as that would mean the least encountered bugs for users in general.

share|improve this answer
    
From a user-centric point of view this is a good answer. In the real world, however, not all bugs are created equal. If there's a feature that is only used once a month, but when it fails it costs the customer (or you) 10's of thousands of dollars, that's the one you should test. –  Bryan Oakley Aug 12 '11 at 17:43
add comment

$ python
>>> import random
>>> random.sample(range(10), 5)
[9, 2, 7, 4, 1]
share|improve this answer
add comment

I would decrease the features to 5 and test them fully. There's no point in writing untested code. It presents more problems than it solves.

This is what i would reply at least :)

share|improve this answer
    
+1 for "not accepting the rules" or trying to change them –  k3b Mar 11 '11 at 7:24
2  
Are you suggesting that you can fully test something? –  Jacob Mar 11 '11 at 7:32
1  
Depending upon the role applied for . How they react to this answer can tell you a lot about the company. –  Aditya P Mar 11 '11 at 8:44
1  
+1, A good example for this approach would be release of Windows Phone 7. –  decyclone Mar 11 '11 at 9:01
2  
Deciding the scope of a release is the responsibility of a business/product team, with the input of development. –  StuperUser Aug 12 '11 at 17:48
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.