Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is it dangerous to substitute unit tests for user testing?

A co-worker believes we can reduce the manual user testing we need to do by adding more unit tests. Is this dangerous?

Unit tests seem to have a very different purpose than user testing. Aren't unit tests to inform design and allow breaking changes to be caught early? Isn't that fundamentally different than determining if an aspect of the system is correct as a whole of the system? Is this a case of substituting apples for oranges?

share|improve this question

closed as not a real question by Jesse C. Slicer, StuperUser, Mark Trapp, Justin Cave, MainMa Oct 25 '12 at 18:56

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

Needs more context. – Caleb Oct 25 '12 at 15:50
Yes and no. Great unit tests can replace large amounts of manual testing. No amount of unit tests should be used to replace all manual testing. Unit tests may replace most (90%) regression tests leaving only 10% regression as "smoke tests". This is predicated on Great Unit Tests. Bad unit tests will give no actual guarantees and be a total waste of your time. – Jimmy Hoffa Oct 25 '12 at 15:53
@JimmyHoffa Why not post that as an answer? – marco-fiset Oct 25 '12 at 15:56
@marco-fiset I try to defer posting things as answers unless I'm certain my answer is both direct and complete. I don't feel like going into the detail for this question which would require citing evidence due to being such a controversial topic. – Jimmy Hoffa Oct 25 '12 at 16:00
Unit testing done right, tests units of code as atoms, completely isolated from other tests. This is impossible to do manually (well, too hard to do practically). Manual testing can replace automated integration tests (which test how these atoms work together). – StuperUser Oct 25 '12 at 16:19
up vote 5 down vote accepted

Manual tests and unit tests are designed for different purposes*, so the answer is "yes" - substituting one for the other is a dangerous practice.

Unit tests pin down your assumptions about your code in the form of assertions that you make throughout your tests. Along with automated integration tests, unit tests help ensure that your software does what you think it should do. When done correctly, they should give you a high level of confidence in the correctness of your software. Automated testing is not as time-consuming, and much less error-prone.

However, it is often impractical to eliminate all manual testing. Specific areas of concern include integration with other products that you must use, such as installers, web browsers, and operating systems. At the minimum, someone needs to install your product before shipping, and see if it starts up, or open your product in the browser to see if the visuals remain in the right places. All of that can be automated, but the effort to make it happen does not make it worth the trouble, especially since we are talking about running through a ten-minute checklist.

If we are talking about the specific "manual tests vs. unit tests" distinction, the two concepts do not slice the space of software testing in the same dimension: in addition to unit tests, you need integration tests, regression tests, and stress tests. These are usually automated, but they can be done manually. Moreover, even the unit testing can be done manually with a small help of a software driver. That's why substituting one for the other is not a good comparison: if your manual testing covers unit testing, regression testing, stress testing, and integration testing, then replacing it with automated unit testing is not a fair exchange. If the gap is filled with some other mean of automated testing, then you can shift some burden from the manual testing to unit testing, but you would not be able to eliminate manual testing completely.

* As Jimmy Hoffa correctly notes, there are companies who (mis)use manual testing in the areas where unit testing is a lot more appropriate. In cases like that, switching to unit tests is not dangerous - in fact, it is highly desirable.

share|improve this answer
It seems like your answer starts addressing unit tests and manual tests and ends up discussing automated tests and manual tests as it's final point. – MushinNoShin Oct 25 '12 at 15:59
@MushinNoShin That's right - I needed to fill the "gap" between the unit vs. manual, because they slice the testing space in different dimensions. The manual/automated dichotomy seemed better suited for addressing the "gap", because unit tests are nearly exclusively in the "automated" camp. – dasblinkenlight Oct 25 '12 at 16:04
That sounds like you're answering a different but easier question. – MushinNoShin Oct 25 '12 at 16:12
@MushinNoShin Please see the edit, I tried to address the gap explicitly. – dasblinkenlight Oct 25 '12 at 16:27
You start by saying manual test address something different than unit tests. Though many companies do manual tests for regression which is exactly what unit tests are for. – Jimmy Hoffa Oct 25 '12 at 18:34

Automated unit tests answer the question 'Does it do what I thought I asked it to do?' while manual testing answers the question 'Does it do something that seems sensible to a user?'.

Different questions. And if you substitute the methods for testing one for the other, you'll tend to get sub-optimal results.

share|improve this answer

When you say "Manual testing" I take it you mean an entire system/program? If so this is not the same as unit testing and is more akin to user testing. Unit tests (should) target a very specific piece of code (a method/property on a class) and test it in isolation. Unit tests are also tests written by developers for developers. Manual testing (I will assume you mean user based testing) is for the users of the system often done (if your lucky) by testers for the product owner/sponsor. You should end up with a lot more unit tests than user tests because they are testing a smaller degree of scenario. So, I would say yes it is dangerous to substitute unit tests for manual testing because in doing so you probably will end up testing less specific things which could lead to more bugs slipping through.

Its also worth noting that unit tests are quick. Do you have time to run 1000's of manual tests everytime you make code changes?

share|improve this answer
I'll clarify the original question to "User testing" from manual testing. Nice catch, thanks! – MushinNoShin Oct 25 '12 at 16:29
@MushinNoShin thought so, most of the testers I have worked with unfortunately are still doing just manual testing! – nashwan Oct 25 '12 at 16:33

Unit tests are important, not the be-all and end-all of testing. The primary purpose is not to find bugs as you write and run the test, but to highlight when expected behaviour changes in the future (and consequently regression bugs).

They certainly don't serve the same purpose that testing with users does.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.