Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

When developing features with TDD, I create a test for each combination of feature and case. So, one test for creating user successfully, one for validation errors, and one for database errors. I don't test each one with many different types of data, in general, as I write the tests to flesh out features, less to catch errors. I don't think it's time-effective to write out several cases (even with a "data provider") for every single feature I develop in order to validate that the feature works correctly in all cases.

Sometimes, though, there are certain implementation specific bugs to be wary of when implementing the feature. After writing the feature test that allows me to write the implementation, I'll notice that certain edge cases need to be tested more carefully because of the implementation.

How does one organize these tests with the rest of the feature-specific tests? Are the tests I'm describing regression tests?

Edit:

I guess I wasn't clear about what I meant with "different types of data" and "works well in all cases." My main issue is in writing implementation-specific tests--specifically the workflow and organization.

We are testing the interface. We're specifically testing the feature only, at first. But a certain implementation requires more tests.

share|improve this question

2 Answers 2

If you can send in different types of data to your methods, and some of those types have a chance of working incorrectly, then you need to be testing those possibilities. Plain and simple.

However, if that is really the case, I think the actual problem is that your code is not set up cleanly. I smell problems.

What I'm going to do is first talk about how to refactor your code so that it can easily be tested. Then, I'll explain how you might approach testing a system like the one you've described, if you have to (my hope is that you don't).

Code Refatoring (the best solution)

First, lets talk about why your methods allow so many different types to be sent in when they don't work consistently with them. This is an obvious code smell that should probably be addressed. My guess is that you need to refactor your code so that this isn't a problem in the first place. An approach I would take might look like this:

  • I'm going to make sure that all objects sent to this method have a common interface
  • I'm going to have the method only use this common interface to do what it needs
  • The objects themselves will each have their own implementation of the interface that will always work for that type of object.

If you do this, you can now easily test the method and see that it is correctly using the common interface. You now know that this code is working no matter what object you put in. The interface is defined, so the correct action is taken.

You will also test each of the objects that implement the interface, and make sure that they have implemented it correctly for the type of object it is. Once you've tested that, you are done, you know everything is working without a million different tests testing each object with each type.

To illustrate, I've put together some pseudo-code to demonstrate. Lets say your code looks like this:

function output_to_file (shape x) {
    if (x is a triangle) {
        // output to file
    } else if (x is a square) {
        // output to file
    } else if (x is a pentagon) {
       // output to a file
    }
}

Then your tests are going to require you to send every different type of shape to the output_to_file method. If you have a whole bunch of methods like this, and a whole bunch of shapes, you will need a whole bunch of tests. Every time you add a shape, you need to modify your output_to_file code and add tests. No fun at all. Not good code.

Instead, you need to refactor your code to look more like this:

function output_to_file (shape x) {
    text = x.get_text_representation()
    save text to a file
}

Instead of handling the details of each shape, output_to_file just uses a common interface: the method get_text_representation.

Now to test output_to_file all you have to do is check that it is calling get_text_representation and putting the results in a file. If it is doing that, it is working. The output_to_file method now follows the "open/closed" principle. It is open to new types of objects being sent in (as long as they support the interface), but closed to the need for additional changes.

This means of course that our shapes need the get_text_representation function. But that is simple:

class triangle {
    function get_text_representation {
       // return whatever
    }
}

class square {
    function get_text_representation {
       // return whatever
    }
}

//and so on

Now we test the get_text_representation method on each of those objects to make sure that they are working like they should. That method might be used in a variety of places, but we don't really care, as long as it is working right we know that other methods relying on it will work right.

By setting up our objects in a different way, we've eliminated the need to test a whole bunch of different objects/method combinations. We test that the method is using the correct interface, and we test that the objects implement the interface correctly. Nothing more is needed. Yay!

Testing More Cases (the less than best solution)

Now perhaps you've been reading this so far, and you're thinking, "well that is nice and all, but I can't do that, he doesn't understand my situation."

Ok, first, try really hard to refactor your code. I'll be worth it. Did you try? Ok then, I'll give you benefit of the doubt and assume you are in some type of complicated situation that can't be refactored the way it should be (but I still don't believe you).

Here's the deal: you have to test each possible different type of object with each different method. You just have to. If some of those object/method combinations can cause error, you must test. Sorry.

Now, if it seems tedious or repetitive to be testing the same methods with different types of input again and again, you can automate the process somewhat. A lot of people forget that you can code your tests to do repetitive things for you. Let me give an example of how you might set up your test:

test_objects = {new triangle, new square, new pentagon .... }    

function test_output_to_file {
   foreach (test_objects as x) {
      output_to_file(x)
      assert everything is good
   }
}

This is really the best you can do if you can't refactor your code correctly. Is it tedious, yes. Is it clean, no. Refactor your code if you can, this isn't a good alternative.

Conclusion

In conclusion, the problem you are describing -- being able to send many different types of data to many different functions, some of which are not supported or buggy -- is probably an indication that your code isn't clean. Testing isn't the problem, and refactoring your code is the solution.

If for some reason refactoring isn't possible (I find that hard to believe!) then you are going to have to bite the bullet and test every combination. Make some structure that allows you to do that a little more easily.

I highly recommend anyone interested in creating clean, testable code to read the book Clean Code by Robert C Martin.

share|improve this answer
    
Perhaps my question wasn't clear. Either way, this does not answer my question in the slightest. –  moteutsch May 12 at 20:09
    
@moteutsch Please describe your problem a little more. Give us an example of some simple code that might illustrate the issues you are having. If not code, at least describe a problem that has arisen in the past, in detail. –  Damien Black May 13 at 18:25
    
You're right. I hope to get to it soon. –  moteutsch May 14 at 15:57

Any test that is used to validate ongoing development of software can be considered a "regression test", i.e. making sure some new change doesn't break old fixes/enhancements. However I don't think it helps to single out the "edge case" testing you describe as specifically "regression tests". TDD is filled with regression tests.

I do see the value of the edge case testing you describe, and the importance of "organizing" them with the TDD feature-minimal tests that are commonly co-written with the code. One simple principle regarding sequence of running tests would be to arrange them in the same order in which they are added. Of course, noting your "unit tests" tag, there may be a hierarchy of code modules, and edge cases may require testing at some intermediate level between a unit (or single routine) and the whole package (application or library).

A TDD approach would emphasize testing that is closely integrated with builds, so presumably the specifics of organizing the tests finds its natural expression in terms of the build tools you use.


Added: A nice 2009 write-up on distinguishing the test cases that drive development/specification from the unit tests that validate acceptance criteria (such as the correctness of edge cases you describe):

TDD Tests are not Unit Tests -- Stephen Walther

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.