Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm getting started on a project wherein I want to have pretty thorough test coverage, and I have the luxury of driving the test strategy. I've settled on a workable plan for unit testing, and I've also settled on using Gherkin to describe features and a port of Cucumber to run the scenarios as end-to-end acceptance tests.

The problem is that I sense that there is a gap in between those two layers. I can test all my units in isolation, and I can test that my features work, but I can think of other things that I'm going to want to test.

I'm also coming from a different project with (poorly-implemented) automated tests that are a very brittle and are a maintenance nightmare, the goal of these tests being to mostly replace manual regression testing. Writing more maintainable tests is a must, but at a higher level I'm not sure that our tests are the right ones.

As an example, given a web application, say there's a form to add an event with start and end dates. As an end-to-end test, we can validate that you can, in fact, add an event. But if your start date is after your end date, then you get an error message, and I wouldn't think that how a trivial user input error is handled belongs in a feature file. On the other hand, there seems to be a pretty strong belief that unit testing the UI isn't worth it; instead, one should do automated integration testing.

So what do I do for this code?

Do I unit test the components related to error messages in general, as well as that this form is going to show them, and skip automating that they actually appear? Do I do the above and then automate that just one error message somewhere shows as intended, and assume the rest will work? Do I try to automate every different potential failure case for every form?

This hits the middle ground of integration testing, of which I am leery. Based on my experience, maintaining a large number of integration tests does not seem worth the value. On the other hand, there is functionality above the unit level and below the feature level that I would ideally like tested, both in the UI and outside of it. And I'm concerned about what kind of confidence automated regression can bring if it isn't hitting everything.

I am perfectly willing to write integration tests, but in the context of when to write tests, what tests to write, and how many to write, what is a good approach to addressing this problem?

share|improve this question
    
No matter how well the components are tested, you still need to test that they are wired together properly. This applies at pretty much all levels of component. Wherever you have an interface (UI or API or whatever) you really need some sort of test to see that stuff gets across the interface intact. –  BobDalgleish Apr 29 at 2:34
    
I agree, but my question is in regards to what extent such tests are necessary. I am starting to get the sense that the goal is to verify that an interface works in general rather than trying to verify that every single use of that interface works properly (to the extent that such is possible). This is a good starting point for writing testable code, but I'm always looking for ways to improve my process. –  Kate Jo Apr 30 at 12:31
3  
Here is a rule of thumb that I use for integration testing: Draw all the components of the system, at a suitable level of detail, and the interfaces between those components. Now, draw a line from a top, outer component all the way to a bottom component through the interface lines. This "line" represents a top-to-bottom integration test. Repeat the process until all components and all interfaces have a line through them. Obviously, these lines are not all implementable as tests, but they should give you a good sense of the testing you need. –  BobDalgleish Apr 30 at 19:19

2 Answers 2

up vote 2 down vote accepted

It all comes down to being cost-effective and pragmatic. Because you are in charge of testing for this project, be careful not to use every type of testing approach or change things that have worked in the past, or introduce a bunch of new tools or practices just for the sake of doing so. I of course am not saying that is your plan, but just wanted to state that.

If the GUI contains a lot of logic, which it shouldnt, then it must be tested. Ideally the GUI contains very little logic (or none, but realistically that is challenging to do). If it does contain a lot of logic, see if it is feasible or cheaper to refactor the GUI code versus creating and maintaining a large base of slow GUI tests. If the GUI contains little to no logic, then you can maybe get away with not testing it at all, or very few very basic tests to make sure the "connections" are all good, as mentioned above. Also since there are only a few, you dont have to worry about their performance and the maintenance overhead isnt as high.

You should integration test the rest of your system (everything exception GUI) at least, though. The system should be designed in such a way that it doesnt require a GUI to drive it, it could be an (automated) command line interface, web service interface, etc.

There should be the highest number of unit tests, then a smaller number of these "core" integration tests, and very few large/ GUI tests.

share|improve this answer

Do BDD. Write acceptance tests to define the important scenarios of the system. You'll have most of the integration tests that you need, plus valuable documentation and proof that your system does what it's supposed to.

Then add integration tests so that you have good integration coverage. (BobDalgleish's comment is a good way of thinking about integration coverage.) I usually find that missing integration coverage helps me find missing acceptance tests.

Then add unit tests to specify details of behavior and technical correctness. You won't have to unit-test some code at all, because it will already be tested by your acceptance/integration tests. You will think of things to test in other code, and you may even want to re-test paths through that code that are already tested in acceptance/integration tests so that the unit tests make sense without reading the integration tests.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.