Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm working on an ASP.NET MVC website which performs relatively complex calculations as one of its functions.

This functionality was developed some time ago (before I started working on the website) and defects have occurred whereby the calculations are not being calculated properly (basically these calculations are applied to each user which has certain flags on their record etc).

Note; these defects have only been observed by users thus far, and not yet investigated in code while debugging.

My questions are:

  • Because the existing unit tests all pass and therefore do not indicate that the defects that have been reported exist; does this suggest the original code that was implemented is incorrect? i.e either the requirements were incorrect and were coded accordingly or just not coded as they were supposed to be coded?

  • If I use the TDD approach, would I disgregard the existing unit tests as they don't show there are any problems with the calculations functionality - and I start by making some failing unit tests which test/prove there are these problems occuring, and then add code to make them pass?

Note; if it's simply a bug that is occurring that can be found while debugging the code, do the unit tests need to be updated since they are already passing?

share|improve this question

4 Answers 4

up vote 4 down vote accepted

Short Answer: Most probably unit tests are not well designed to cover the critical calculation scenarios. In addition, there might be business rule modifications/changes that are not covered in the existing unit tests, because of million reasons that one may face in a tightly scheduled development environment.

As was also mentioned:

Automatic unit testing doesn't imply that the final tests will cover all cases and code will be absolutely correct and bug-free. It is easy not to see some complex use-cases and possible bug.

There might be million reasons why it is NOT done right. My point is that you need to cover these cases with product owner, and then go through each unit test and verify the implementation by applying correct naming of each unit test, as well as by adding the missing once.

If I use the TDD approach, would I disgregard the existing unit tests as they don't show there are any problems with the calculations functionality - and I start by making some failing unit tests which test/prove there are these problems occuring, and then add code to make them pass?

TDD (Test-driven development) will for sure help, but i would NOT discard the current set of unit tests as far as they do what they supposed to be doing.

share|improve this answer

Using TDD and automatic unit testing doesn't imply that the final tests will cover all cases and code will be absolutely correct and bug-free. It is easy not to see some complex use-cases and possible bug.

TDD on the other hand makes it easier to introduce changes for those not-thought-before cases. Simply repeat TDD as if you were developing:

  1. Write new test case, that fails
  2. Implement code, that will make this test succeed
  3. Refactor code to remove duplicities and introduce good abstractions while keeping all tests passing
share|improve this answer
    
I'm not clear on whether I would need to add more unit tests if I change/fix the code in order to verify that it has been fixed and now works? I'm guessing I would create unit tests (that fail first) which have all the data stubbed necessary to reproduce the problem, and then add code / change existing code until it passes? –  Theomax Sep 23 '12 at 9:44
3  
It's also well worth auditing the existing tests that exercise this area of the applications. Tests are just code after all, so they can contain bugs. –  chooban Sep 23 '12 at 9:52
3  
@GregBair unless the tests that are there are right but not comprehensive... edge cases are a killer and may well be missed by the tests precisely because they are edge cases... –  Murph Sep 23 '12 at 15:16
1  
@GregBair I'm not sure I agree - if the tests are for the results of calculations and the issue is edge cases then you're adding more tests for more case and the old tests shouldn't fail because the rules they're testing still apply –  Murph Sep 25 '12 at 12:31
1  
@GregBair but the key point here is that you have specific scenarios that are not being correctly identified by the tests. One has to assume that the tests are right just not comprehensive. So write the new tests, fix the code and then see what happens... –  Murph Sep 25 '12 at 17:43

This is exactly why you would want to consider Behavior driven development and Acceptance Test driven development - unit tests are not a guarantee about whether the system is doing what it is supposed to - it checks only whether it does what it was designed to do. The flip side is that product owners/Business analysts cannot review these tests very easily to verify that they are correct.

On the other hand, if you use acceptance tests, then they can be very easily verified by Business analysts or other non-tech folks. That can be quite useful.

In your particular case, I would say the product owner should take the decision on whether these are bugs or not - if they are, then obviously existing tests are either incorrect or at the very least insufficient - and you will have to write new (failing) ones and maybe even delete some of the old ones (which are expecting wrong things) before writing code that passes the new tests.

share|improve this answer

I do not believe that unit tests necessarily test requirements: 1) unit tests operate against very low level components (methods/classes/interfaces/et cetera) tested as stand-alone units; 2) requirements are supposed to be tested while the test engineer designs integration/system tests during the requirements design phase; 3) functionality, requirements and specifications are tested (again) once the system is in an operational state; and 4) using an iterative software development approach, many phases of the development lifecycle are revisited for tightening up requirements, design, and development -- processes that re-occur.

Why unit testing is not appropriate for testing requirements. Unit testing will not necessarily uncover a requirement issue for a number of reasons. One, if a requirement was poorly communicated, chances are the requirement was incorporated into the software design, and subsequent development; as a result, if the unit testing follows suit, correctly following an incorrect requirement, the unit testing will come up with nothing. Two, if the requirement was properly documented, and the development team misinterpreted the requirement, similar to item 1, the misintepretation will carry through to the unit testing. Three, as far my experience goes, unit testing is done by the developer, and is part of the development lifecycle -- unit testing is low-level and will not likely represent a requirement, unless each complete requirement is represented by its own method, which is very, very unlikely.

Example. Consider unit testing in terms of methods. Unit testing largely verifies that given the pre-condtions, post-conditions hold after execution of the method under test. The object of unit testing is to test/verify the methods, et cetera, of a software system, starting from the most atomic or primitive methods, progressing towards the most complex ones. By testing each low-level method, you verify whether it works, a no brainer. At this stage, the developer does not need to re-test the low-level method again as a stand-alone unit. While moving upward in complexity to higher-level methods, unit testing verifies these methods, demonstrating that the primitive methods, from which the higher-level ones are composed, work together. As far as I know, this is pretty much where unit testing stops.

How requirements are tested. In a well planned project, the test engineer (an integrated / system tester), starts designing tests while the requirements are written. Through this activity, requirements get verified, before system design, and long before a stitch of code is written. Once the system becomes available in some useable form, requirements are verified via functional/integration/system testing.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.