Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I cannot count the number of times I read statements in the vein of 'unit tests are a very important source of documentation of the code under test'. I do not deny they are true.

But personally I haven't found myself using them as documentation, ever. For the typical frameworks I use, the method declarations document their behaviour and that's all I need. And I assume the unit tests backup everything stated in that documentation, plus likely some more internal stuff, so on one side it duplicates the ducumentation while on the other it might add some more that is irrelevant.

So the question is: when are unit tests used as documentation? When the comments do not cover everything? By developpers extending the source? And what do they expose that can be useful and relevant that the documentation itself cannot expose?

share|improve this question
4  
Never thought of using unit test straight as documentation. I think the unit tests are often unreadable, because many developers don't take time to write them clearly. –  superM Jun 28 '12 at 7:37
9  
Comments may be wrong. –  user1249 Jun 28 '12 at 7:47
2  
I know you are asking specifically about unit tests, but I'd also note that integration/system tests as really useful documentation too, just at a different level –  jk. Jun 28 '12 at 11:49
2  
I've seen unit tests that would be better characterized as “unit experiments”. Their dependence on external factors was just so much as to make them near useless. They were also very unclear. (Yes, I have a long-term goal to refactor them to be better, but I do other things too…) –  Donal Fellows Jun 28 '12 at 14:26
3  
@Ant unit tests invoke the real code and documents the expected response and compare it to the actual response. Whether the code invoked is correct or not, is not the point - the tests documents how to invoke it. –  user1249 Jun 28 '12 at 14:49

6 Answers 6

up vote 12 down vote accepted

They're NOT an ABSOLUTE Reference Documentation

Note that a lot of the following applies to comments as well, as they can get out of sync with the code, like tests (though it's less enforceable).

So in the end, the best way to understand code is to have readable working code.

If at all possible and not writing hard-wired low-level code sections or particularly tricky conditions were additional documentation will be crucial.

  • Tests can be incomplete:
    • The API changed and wasn't tested,
    • The person who wrote the code wrote the tests for the easiest methods to test first instead of the most important methods to test, and then didn't have the time to finish.
  • Tests can be obsolete.
  • Tests can be short-circuited in non-obvious ways and not actually executed.

BUT They're STILL an HELPFUL Documentation Complement

However, when in doubt about what a particular class does, especially if rather lengthy, obscure and lacking comments (you know the kind...), I do quickly try to find its test class(es) and check:

  • what they actually try to check (gives a hint about the most important tidbits, except if the developer did the error mentioned above of only implementing the "easy" tests),
  • and if there are corner cases.

Plus, if written using a BDD-style, they give a rather good definition of the class's contract. Open your IDE (or use grep) to see only method names and tada: you have a list of behaviors.

Regressions and Bugs Need Tests Too

Also, it's a good practice to write tests for regression and for bug reports: you fix something, you write a test to reproduce the case. When looking back at them, it's a good way to find the relevant bug report and all the details about an old issue, for instance.

I'd say they're a good complement to real documentation, and at least a valuable resource in this regard. It's a good tool, if used properly. If you start testing early in your project, and make it a habit, it COULD be a very good reference documentation. On an existing project with bad coding habits already stenching the code base, handle them with care.

share|improve this answer
    
May I ask why I was downvoted? What ticks you off in there or what do you disagree with? –  haylem Jun 28 '12 at 7:59
1  
The (IMO) best part of your argument is written in the smallest font - the best way to understand code is to have readable code in the first place. I'd change that to "readable and working code", but I agree. Then if you look at unit tests again - running tests are working code (and like all code, should be readable), so it's actually pretty good (if often too local) documentation when done well. –  Joris Timmermans Jun 28 '12 at 12:38
    
@MadKeithV: thanks. I updated for "readable and working code" and pushed that bit higher up. –  haylem Jun 28 '12 at 14:15

One interpretation is that unit tests are "executable documentation". You can run the unit tests against the code and it will tell you whether it's still performing as it was when the tests were written to pass, or not. In that way the unit tests "document" the functionality of the system at some point in time, in an executable way.

On the other hand I have also only rarely actually read unit test code as "documentation" to understand functionality. A single unit test is too localized, specific and abstract in a way to be able to tell you much about the actual system that's behind the class being tested.

share|improve this answer

If by documentation you mean I want something to find out how the code works then unit tests are perfect little examples of how units of the code works in both expected, edge cases, and mistakes (aka bugs) cases. Also, your tests could be created before the code is written thus underlying what the code should do from a business/requirement point of view.

Are they replacing documentation? No.

Are they a useful addendum to documentation? Yes.

share|improve this answer

I see unit tests as:

  • a way to prove the documentation is correct (assuming the documentation matches the api implementation).
  • a way to demo to a developer how to use a particular feature; unit test fixtures/unit test themselves are usually small enough that one can quickly learn from it.
  • and obviously to spot any regression bug.

To some extend they can be seen as a complement to an existing documentation but not as the documentation.

share|improve this answer

I'm going to answer your question by asking you another.

How often when working with a new API/routine have you fired up help to look for a code example of the thing you're trying to use? Failing that you switch over to google for an online search of code samples?

That is exactly when you would use unit tests as documentation.

  • In fact the unit tests can be a little more rigorous than normal code examples, because you should have multiple tests (examples).
  • Hopefully your unit tests illustrate proper usage. E.g. They clearly show all essential dependencies either via normal objects or mock objects. (Otherwise they're not particularly good unit tests.)
  • NOTE: If your comments or "normal documentation" provides code examples, you're actually violating the DRY principles. And those code examples can easily become incorrect over time, whereas there's significantly less chance of that with regularly executed unit tests.
  • If the unit tests are thorough (usually a big if), then they should provide additional information:
    • All known edge cases clearly illustrated.
    • All expected exceptions that can be thrown.
    • All bugs previously found (this is probably more useful when extending the unit under test than when writing a new client for the unit).
    • All underlying business rules associated with the unit. (if any)

I suspect there are quite a few reasons unit test don't tend to get used as documentation even though they could be an excellent complement to more traditional documentation:

  • I would venture to suggest that often the tests themselves aren't well enough written for the purpose. Other answers have already alluded to tests that are:
    • Incomplete.
    • Confusing. (I've seen test cases that don't call the method under test directly - you go 3/4 levels deep into the call stack before it's called and the preconditions to calling the method are scattered in different places in a complex class hierarchy.)
    • Outdated. (usually tests should fail when they become outdated, but this is not always the case).
  • There are usually plenty examples of use already available in production code whenever the need for an example arises.
  • The unit under test is so well written (self documenting) that the methods don't need examples. I wish!
  • In my experience as a programmer, we tend to be quite keen to jump in at the deep end and RTFM next Tuesday...
  • Documentation and comments that violate the DRY principle.
share|improve this answer

TL;DR Unit tests and API comments are complementary - some things are best described in code, and others in prose.

Unit tests are mainly useful to document special cases and edge conditions which are hard (and cumbersome) to describe in API comments. Also, the API comments are usually directed to people who want to use the API.

If you want to modify the code, there's typically a lot more you need to know, and some of that is difficult to put into comments (and these comments go stale quickly). In that case a unit test also works as documentation.

An example: You have a method m(a, b) that performs a certain calculation. Due to backward compatibility requirements, it must special-case inputs of a=0 and a=-1, but only if b is NULL. Putting that into a comment is complicated, verbose, and likely to go stale if the requirement is later removed.

If you make some unit tests that checks the behaviour of m(0, NULL), m(-1, x) you get several benefits:

  • The description of correct behaviour is clear, misunderstandings are reduced
  • People cannot overlook the requirement when they change the code, unlike a comment
share|improve this answer
    
but for your example, if that behaviour is not documented in the comment at all, a user might get unexpected results for that edge case. Which is not exactly a good thing. –  stijn Jun 28 '12 at 8:01
    
@stijn: True. In that case the best way would probably be to have a brief mention of the special case in the docs, plus the unit tests for the messy details. –  sleske Jun 28 '12 at 8:17

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.