Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

What are some strategies that could be used to automate the creation of unit test cases? What aspects would you need to look at in each class to be able to generate at least a decent test case skeleton?

I realize a comprehensive automatic solution isn't practical, but I'd like to speed up the test creation a little at least by creating a skeleton. I'm not looking for code examples, just perhaps some suggestions for where to start or examples of where something like this has been done so that I can see how they approached it and what might be possible.

I'm particularly interested in methods for creating unit test skeletons in PHP, which doesn't provide all of the tools that other languages afford, like complete type hinting, for example.

share|improve this question
    
Latest Visual Studio is all you need ... –  Job Sep 24 '11 at 17:03
add comment

3 Answers

up vote 5 down vote accepted

Your strategy and skeleton depends, non-trivially, on what kinds of tests you're looking to generate, what kind of coverage you're looking for, and the language/environment you're working in.

It's fairly straight forward to write a test generator which, for languages like C or Java, reads class signatures and automatically generates tests for standard corner cases (passing in 0, 2 random values, MAX_INT, MIN_INT, to an integer argument, nulls for nullables, etc...). You could then run the generated tests, record the results for each test, and manually filter through them to remove irrelevant ones, approve acceptable results for tests which pass (so they can automatically pass from then on), and mark as invalid ones which fail.

You can augment this with tagging/commenting/refactoring of classes to help your generator with extra hints. You might have a tag which lists all the possible exceptions that a method call is allowed to raise, or which gives a reduced range of valid integers for an integer argument. Look at these as short-hand for having to write the tests yourself.

So, here are some components you'll want to look at:

  • A component to automatically parse source code/function signatures/manual annotations, producing standard test cases, or outlines/signatures for test cases which wait for your input to be completed.
  • A perpetually growing/changing language of tags/annotations/comments which may go to any level of granularity (method/class/signature/while loops/etc...) representing hints to the automated test builder. Ideally you should be able to play with this language without having to recode your framework or any chunks in it
  • Automated test runner, with the ability to identify new/old tests and record/test against "acceptable" answers for each test. Ideally this runner will build a database of test runs, results accepted/declined, and current acceptable results for each test.
  • Automated "object faker" which, given a class name and map of names->values, can generate an object mimicking the class, returning customizable data for function calls, accessors, public data slots, etc...

There are lots of test frameworks out there which already include chunks of this functionality for various languages and platforms. While it's fairly easy to start doing this work yourself and grow this kind of framework organically in-house, it's also an endless long-term project which will probably be duplicating existing work. I'd recommend taking significant time to look at what's available first, and then decide whether it's worth the time to dive in.

share|improve this answer
add comment

I haven't had a chance to use it on an application of a meaningful size or complexity yet, but there are tools, including Google's CodePro AnalytiX, that automate the generation of unit tests for Java applications. I also found a commercial product, Parasoft's C++Test, that appears to allow for the generation of C++ unit tests

These applications used heuristics to generate test cases. I'm not sure that there's a single framework that you can use to produce a skeleton, but there are constructs that you can look for. I tend to focus on loops, conditional statements (if blocks, switch/case statements), and exceptions, and create test cases that force different execution paths to be carried out.

I wouldn't focus on speeding up the writing of tests by trying to create a skeleton or template, but rather by improving analysis of the specification and/or implementation and writing high quality tests. Being able to identify what tests add the most value early, writing those, and then filling in holes later would have a greater impact on productivity and quality.


Just to provide some more publicity, Falcon tried out CodePro on a project and wrote a little blurb about his experiences.

share|improve this answer
    
Google's CodePro Analytix sounds interesting. But "Quis custodiet ipsos custodes?" Who tests the tests? This can only be used to backup an existing project by unit tests and will probably not detect failures, it'll rather assume that the defects are correct. –  Falcon Sep 18 '11 at 10:54
    
@Falcon You can't blindly trust any tool - doing so will just cause more headaches. I think the Pragmatic Programmer's tip "care about your craft" applies here. CodePro does contain a test editor to make it trivial to see what values are being passed in and what the expected result is, and then to make changes to it (and then update the generated test code to reflect these changes). –  Thomas Owens Sep 18 '11 at 10:58
    
I just wonder what is more reliable in this case, human or machine. I think those generated tests will cause more headaches than tests written manually. Ideally the test should be written first anyway. But I'll definitely give it a try. I'd love to see a tool which can generate tests based on formal requirements and some metadata glue to interface the system one day. –  Falcon Sep 18 '11 at 12:09
    
@Falcon Yeah, ideally some tests should be written first, but until you have an implementation and can do white-box testing, you don't necessarily see all of the different edge cases that you can see once you have an implementation. If you get a chance to play with CodePro's test generation features, could you post your thoughts somewhere and somehow get me a link? I'm interested to see how well it works and other people's experience with it. –  Thomas Owens Sep 18 '11 at 12:15
    
I'll test it next week with a medium sized J2EE application (120 klocs) that incorporates some really tough business rules and tell you about my experiences here. –  Falcon Sep 18 '11 at 12:24
show 2 more comments

I wrote a generator to speed up unit testing for a .NET project a few years ago. There was a large codebase with no unit tests and it aimed to quickly increase the basic coverage. Here are a few notes that might be of help:

  • My chance was that the core framework on top of which the project was developed provided standard operations and class naming. If you're thinking of writing your own, a standard structure like this will help greatly.
  • Using data-driven testing helps a lot, if your codebase permits. The test framework created a database table for each unit test to store test data, so that each row in that table was a separate test and no additional code was required (Rule of Representation). From this point on, the actual tests could be easily created automatically or entered manually.
  • The resulting unit tests were simple but they served as smoke tests at least. For areas of higher risk, additional manual tests were written.

To summarize, I agree that a generic solution would be impractical (if it's possible). I believe the chances are better if the codebase is suitable for test generation and the test framework can take advantage of its structure.

(As a side-note, there's Pex, but it's for .NET)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.