Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We're outsourcing some work to an external developer, so I'm busy writing up a contract about what constitutes a deliverable.

So far I require that the code is shipped with automated tests.

But, what is a reasonable way to specify the detail of tests up-front in the contract in a measurable way?

I'm loathe to say "100% code coverage" because it's been established pretty often that 100% is pretty meaningless, and the diminishing returns above about 70-80% would probably just be pushing up our costs unnecessarily, and possibly even pushing up the complexity of certain things that might otherwise be very simple.

Internally we pretty much leave it up to our developers to decide on the level of tests needed, based on their intuition and experience. With a contractor however there is a fixed price that has to be agreed to up front and we need some way to enforce a certain level of quality.

Any suggestions or recommended reading matter would be appreciated!

share|improve this question
    
Why do you want tests? Is it for code quality or to ensure you have got what you asked the developer for? –  Henry Nov 2 '10 at 2:48
    
Quality, and future maintainability. –  realworldcoder Nov 2 '10 at 2:54
    
make sure you properly assess the coverage you want, contractors charge the same rate regardless of whether they are writing a code or tests. I would not include anything related to tests and coverage into the contract and just consider tests as a part of the product, justifying the required coverage on per-component basis –  bobah Nov 2 '10 at 8:01
    
Do you want the contractor to produce good coverage or to write good and maintainable tests ? –  David Nov 2 '10 at 8:09
add comment

4 Answers

up vote 1 down vote accepted

When subcontracting out, it is up to you to ensure the code being written at least works the way you need it to. For that reason, your team will need to write some automated acceptance tests. Provide those tests to your subcontractor, so they can make sure their code works with it.

Anytime you require percentage coverage in your unit tests, it is up to you to provide the tool which will be measuring the code coverage. I don't know the environment you are running (.Net, Java, Ruby, etc.), but there are usually more than one tool available to measure coverage, and they are not all equal. You also need to specify, or at least agree to the parameters used (i.e. coverage exclusions, type of coverage, etc.).

It would be unfair and unproductive to require testing of:

  • Generated classes/methods (some ORM tools generate classes, .Net UI components generate classes and methods, etc.)
  • System level exception catching code. The code may be required by the language, and good practice, but if testing it requires hacking the platform itself, it's not worth the investment.

Don't require more of your subcontractors than you would of your own team. If you are going to require a certain percentage of unit tests as an acceptance criteria, provide a range like 70-80%. If they beat it, great. I would consider 50% coverage an absolute minimum, with 70% a decent requirement. Anything above 70% may cost more, but you'll have better piece of mind about it.

Just a note about metrics like test coverage. They are just numbers, and anyone can play with numbers. I think your intent is a good one, but anyone who wants to game the system can. The coverage number is a rough indication of the thoroughness of the testing, but not the quality of the testing. In my experience, many programmers who are not used to writing unit tests tend to write integration tests, and merely run the application through the test framework without any assertions whatsoever. Essentially they are just providing themselves a launching point to step through with a debugger. It takes time and training to get unit tests that are useful.

I would require an early initial delivery simply to evaluate the effectiveness of their unit testing, and to help fine tune both your expectations and theirs. That will help both of you to get on the same page, and make future deliveries better.

share|improve this answer
add comment

You can come up with more specific measure. E. g. 100% coverage for methods with cyclomatic complexity >= 5. Etc...

share|improve this answer
add comment

If you want a number, 80% is often used. In fact, where I work presently our continuous integration server (Hudson) displays a yellow light for any project on which the test coverage is below 80%.

The challenge here is that ensuring that a certain percentage of the lines are covered by tests is very different from ensuring that the code is tested in a way that leads to better maintainability.

For one thing, a code-coverage tool can be fooled by a test that exercises the code and ends with Assert.assertTrue(true).

The less malicious concern is that a programmer that doesn't know how to write good tests will not write tests at the appropriate level of granularity, which can lead to a situation where future changes require major changes in the tests, and the tests become a burden to refactoring, rather than a help.

So if I wanted to give a number, I'd use 80%. But this number is only useful when dealing with an honest developer that knows how to write good tests.

share|improve this answer
    
"But this number is only useful when dealing with an honest developer that knows how to write good tests.". Actually, in that case it might be useless as this kind of developer would write those tests even without being told to. –  David Nov 2 '10 at 10:10
    
@David: Not really - without a policy like that, how does the honest and competent developer know how much of the client's money to spend writing tests, and what tests should be in the deliverable? –  David Thornley Nov 2 '10 at 14:40
    
@David Thornley: maybe you're right. I just think that : either you trust your contractor and it will provide a good level of quality (that's why you chose this particular contractor) or you don't trust it and whatever rule you set you are not sure of getting the quality you expect. –  David Nov 2 '10 at 14:54
add comment

Testivus on Test Coverage -- From the Google Testing Blog:

Early one morning, a young programmer asked the great master:

“I am ready to write some unit tests. What code coverage should I aim for?”

The great master replied:

“Don’t worry about coverage, just write some good tests.”

The young programmer smiled, bowed, and left.

Later that day, a second programmer asked the same question.

The great master pointed at a pot of boiling water and said:

“How many grains of rice should put in that pot?”

The programmer, looking puzzled, replied:

“How can I possibly tell you? It depends on how many people you need to feed, how hungry they are, what other food you are serving, how much rice you have available, and so on.”

“Exactly,” said the great master.

The second programmer smiled, bowed, and left.

Toward the end of the day, a third programmer came and asked the same question about code coverage.

“Eighty percent and no less!” Replied the master in a stern voice, pounding his fist on the table.

The third programmer smiled, bowed, and left.

After this last reply, a young apprentice approached the great master:

“Great master, today I overheard you answer the same question about code coverage with three different answers. Why?”

The great master stood up from his chair:

“Come get some fresh tea with me and let’s talk about it.”

After they filled their cups with smoking hot green tea, the great master began:

“The first programmer is new and just getting started with testing. Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.

The second programmer, on the other hand, is quite experience both at programming and testing. When I replied by asking her how many grains of rice I should put in a pot, I helped her realize that the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.”

“I see,” said the young apprentice, “but if there is no single simple answer, then why did you tell the third programmer ‘Eighty percent and no less’?”

The great master laughed so hard and loud that his belly, evidence that he drank more than just green tea, flopped up and down.

“The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.”

The young apprentice and the grizzled great master finished drinking their tea in contemplative silence.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.