Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm working on a legacy system (by that I mean it was written without tests). We've tried to test some of the system by writing integration tests that test functionality from the outside.

This gives me some confidence to refactor parts of the code without worry about breaking it. But the problem is these integration tests require a deploy (2+ minutes) and many minutes to run. Also, they are a pain to maintain. They each cover thousands of lines of code and when one of them breaks it can take hour(s) to debug why.

I've been writing lots of unit tests for these functional changes I've been making lately, but before I commit I always do a new deploy and run all integration tests, just to make sure I didn't miss anything. At this point I know my unit tests and some of the integration tests are overlapping what they test.

How do I know when my good unit tests are adequately covering a bad integration test so that I can delete that integration test?

share|improve this question
add comment

2 Answers

up vote 12 down vote accepted

The easiest metric is to ask, "when was the last time this integration test legitimately failed?" If it has been a long time (there have been a lot of changes) since the integration test failed, then the unit tests are probably doing a good enough job. If the integration test has failed recently, then there was a defect that was not caught by the unit tests.

My preference would generally be to increase the robustness of the integration tests, to the point where they can be reliably run unattended. If they take a long time to run, then run them overnight. They are still valuable even if they are only run occasionally. If these tests are too fragile or require manual intervention, then it may not be worth the time spent in keeping them running, and you may consider discarding those that succeed most often.

share|improve this answer
3  
+1 for recommending the tests be automated since that leads to the obvious question of "Why kill off an automated test?" –  GlenH7 Jan 30 '13 at 21:12
1  
Yea, I agree with this. Of course, even that still comes to bite you if you don't have good enough unit test coverage. For instance, we currently have an integration test suite that takes about 6 hours to run... but I don't think a test has EVER been deleted due to my company's focus on compatibility –  Earlz Jan 30 '13 at 22:13
1  
Perhaps I should start a new question for this, but do you suggest that whenever an integration test legitimately fails I should figure out how to write a unit test that also fails and get them both passing? –  tieTYT Jan 31 '13 at 5:49
1  
@tieTYT: Yes, that absolutely sounds like a good idea. Unit tests are good; unit tests for stuff that you know has already broken before are even better. –  Greg Hewgill Jan 31 '13 at 7:31
add comment

Units tests are not the holy grail of testing, they are only one tool of many than should be used to test a code base. Therefore no amount of unit tests should ever be considered safe to replace other tests. If you have a bad integration test you should work to make it a good integration test, not replace it with something else, that is like replacing your front door with a perimeter fence and a gate.

share|improve this answer
    
If this project started from scratch that'd make more sense to me. But my first integration test was "check that you can login" and eventually I made a lot of unit tests that "check that you can login". The integration test crashes all the time if you change the html. This example is completely contrived, but isn't it a good case for removing the integration test? –  tieTYT Jan 30 '13 at 22:17
2  
@tieTYT: testing something through the UI leads often to very unstable solution. Nevertheless, testing by the UI is important, it is just sometimes the case that manual tests here produce less effort than trying to automate that test and keep it stable. So when you think this is the case here, you may remove that "integration test" from the list of automated test and add it to your test plan of manual tests. –  Doc Brown Jan 31 '13 at 12:42
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.