Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We've built up a substantial number of unit tests for our main program over the years. Several thousand. The problem is that we don't have a clear idea of what tests we have because there are so many. And that's a problem because we don't know where we're weak on tests (or where we have duplicates).

Our app is a reporting engine. So you can have a template that is used to test parsing (do we read all table properties), merging in data (did we keep the correct table properties in the merge), formatting the final page (is the table placed correctly on the page), and/or the output format (is the created DOCX file correct).

Add to this what we need to test. Take the padding around a table cell (we use Word, Excel, & PowerPoint for the report design). We have to test the padding across page break, for a table inside a cell, vertically merged cells, horizontally merged cells, a vertically & horizontally merged cell that contains a table with a vertically & horizontally merged cells in the inner table, where that table breaks across a page.

So what category does that test go in? Table padding, page breaks, nested cells, vertically merged cells, horizontall merged cells, or something else?

And how do we document these categories, name the unit tests, etc.?

Update: A number of people have suggested using coverage tools to verify that we have full coverage. Unfortunately that is of limited use in our case because the bugs tend to be due to specific combinations so it's code that has all been tested, but not in that combination.

For example, we had a customer yesterday that started a Word bookmark at the end of a forEach loop in their template (a Word document) and ended it in the beginning of the next forEach loop. This all used code that has unit tests against it, but we had not thought of the combination of a template expanding a bookmark start to be started 25 times, then ended 10 times (the two forEach loops had a different number of rows).

share|improve this question
    
It seems your question is really, How do we know we've tested a particular scenario? –  Andy Wiesendanger Oct 4 '11 at 18:59
    
Yes! And also where are the tests for any similiar scenarios. And from that we get the #2 need - reading what is covered helps us find what we've missed. –  David Thielen Oct 4 '11 at 19:21
add comment

4 Answers

Generally, I tend to mirror the source tree for my unit tests. So, if I had src/lib/fubar, I would have a test/lib/fubar which would contains the unit tests for fubar.

However, what you seem to be describing are more functional tests. In that case, I would have a multi-dimensional table that enumerated all your possible conditions. Then, the ones that have no tests are either nonsensical or need a new test. You can, of course, then put them in sets of test suites.

share|improve this answer
    
We presently mirror the source tree. But we have two problems. First, for table formatting, there are over 100 different tests. Keeping track of what exactly is tested has become an issue. Second, very different functional areas need to test tables - the parsers, the data substitution, the formatting, and the creating the output document. So I think you're right, in a sense it is functional testing of a given property. –  David Thielen Oct 4 '11 at 17:44
    
Which leads to the question, where do we store the table of tests? I'm thinking a spreadsheet in the root source directory??? –  David Thielen Oct 4 '11 at 17:44
    
I would store it in a version controlled spread sheet in the test directory. If you have a lot of things you need to test, breaking it down into meta structures would be beneficial. Try thinking in terms of what general thing is being tested instead of what or how. –  Sardathrion Oct 5 '11 at 8:17
add comment

There is no reason for a unit test to be only in one category. All major unit test toolkits support the creation of test suites, which bundle tests for a particular category. When a particular area of code has been changed, the developer should run that suite first and often to see what has broken. When a test concerns padding and breaks and nesting, by all means put it into all three suites.

That said, the point of unit tests is to run them all the time, i.e. they should be small and fast enough that it's feasible to run them all befor committing any code. In other words, it doesn't really matter what category a test is, it should be run before committing anyway. Suites are really just a crutch that you use if for some reason you can't write tests that are as fast as they should.

As for coverage, there are very good coverage tools that tell you what percentage of lines were actually exercised by running your tests - this is an obvious pointer to what kind of tests you are still missing.

As for naming, there is no particular value in expending effort on the names of unit tests. All that you want to hear from your tests is "5235 of 5235 tests passed". When a test fails, what you read is not its name, but the message, e.g. the String in the assert() that implements your success critrion. The message should be informative enough that you have an idea of what's wrong without reading the body of the test. The name is unimportant compared to that.

share|improve this answer
    
Agree 100% on all you say (our build machine runs all tests on a check in). Our big problem is tracking what we are testing. And code coverage isn't a lot of help (see update above). –  David Thielen Oct 4 '11 at 17:46
add comment

One way of knowing whether you are weak on tests is traceability. Usually for tests, this takes the form of coverage.

The aim is to measure what parts of code are exercized by your tests, so that you can see code that is not covered by your tests. It is up to you (and the coverage tool) to define what a "part of code" is. The least is branch coverage.

share|improve this answer
add comment

In .NET, I tend to mirror, or at least approximate, the namespace structure for the source code in the test projects, underneath a "Tests.Unit" or "Tests.Integration" master namespace. All unit tests go in one project, with the basic structure of the source code replicated as folders within the project. Same for integration tests. So, a simple solution for a project might look like this:

Solution
   MyProduct.Project1 (Project)
      Folder1 (Folder)
         ClassAA (Class def)
         ...
      Folder2
         ClassAB
         ...
      ClassAC
      ...
   MyProduct.Project2
      Folder1
         ClassBA
         ...
      ClassBB
      ...
   ...
   MyProduct.Tests.Unit
      Project1
         Folder1
            ClassAATests
            ClassAATests2 (possibly a different fixture setup)
         Folder2
            ClassABTests
         ClassACTests
      Project2
         Folder1
            ClassBATests
         ClassBBTests
      ...
   MyProduct.Tests.Integration
      Project1 (a folder named similarly to the project)
         Folder1 (replicate the folders/namespaces for that project beneath)
            ClassAATests
         Folder2
            ClassABTests
         ClassACTests
      Project2
         Folder1
            ClassBATests
         ClassBBTests

For any AATs or AEETs that are coded with a unit-testing framework, this changes a bit; usually those tests mirror a set of steps, that will test the functionality of a new use case or story. I usually structure these tests in a MyProduct.Tests.Acceptance project as such, with tests for each story, possibly grouped by milestone or "epic" story which the story under development belonged to. However, those are really just uber-integration tests, and so if you prefer to structure the tests in a more object-oriented instead of story-oriented fashion you don't even need a MyProduct.Tests.Acceptance or similar project; just throw em in MyProduct.Tests.Integration under the scope of the highest-level object under test.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.