Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm in the process of choosing new unit testing tools for soon to be started project. There are plenty of them!

The basic requirements I have would be that it can run tests on at least Windows and Linux, it is C++/C++03 compatible, it can execute tests in parallel (e.g., to take advantage of multi-core processors), and it is able to be run from the command line. But these are pretty broad requirements, and haven't narrowed down the field.

To try and do that, I'm looking into potential gotchas and pitfalls of different types of implementations. For example, if I wanted to use a unit testing engine in higher level language like Python, would it still produce output formatted like C unit tests? My main intent here is to use the same storage and report tools for both C++ and Python results tests.

I'm also wondering if it's necessary to have a specific engine to run the tests, or is it enough to have them integrated into our continuous integration engine (buildbot).

Are these two concerns above valid? What else should I be concerned about when evaluating unit testing engines?

share|improve this question

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

1  
Hi Didier Trosset, I've revised your question a little bit just to make sure people don't mistake it for a unit testing engine recommendation question, which wouldn't be on-topic here. Great question otherwise! –  user8 Jan 20 '12 at 20:45
add comment

2 Answers

My experiences involve CruiseControl.NET and Team Foundation Build. I'm a .NET developer myself. These come from my experiences, and only include the experiences where we had testing as part of the build process, so if they don't match your environment, sorry.

I'm also wondering if it's necessary to have a specific engine to run the tests, or is it enough to have them integrated into our continuous integration engine

If you are doing a build every check-in, it is possible to have builds triggered before the previous build is finished running. How do you want to handle situations like this? For example, if it takes 30 minutes to run through the entire test suite, and code gets checked in every 15 minutes, do you want to skip a few builds? Skip some tests? Stack them up, build them in order and get back to Bob that he broke the build with his check-in 5 hours ago?

Team Foundation Build can use multiple cores for builds and tests with 2010 (previous versions could only use multiple cores for C++ builds). CruiseControl.NET can run separate threads, but reading the documentation, it appears that each project can be on its own thread, but that you can't have multiple threads per project (I may be wrong). We've never had a multi-core machine for builds in the environments I've worked in, so I can't speak to how good/fair/bad they are (or wrong I am).

At one previous employer, we included Python scripts in the build, but we didn't have any testing set up to test the Python. NANT was used for the .NET components.

I'm also wondering if it's necessary to have a specific engine to run the tests

I never had time to do it, but at a previous employer (we sold "shrinkwrap" software), we had many bugs that were operating system specific (and sometimes service pack specific), so one of my goals was to set up multiple test machines with different OSes (both 32 + 64 bits and all the various flavors of desktop and server Windows from XP onwards). The build machines were simple and older WinXP machines (or virtuals) because that was all the "hardware" we were allowed by the managers. Instead, all tests ran on the build machine. We also set up validation suites as collections of unit tests, although these were not run for minor updates.

share|improve this answer
    
@Sardathrion, Checking the revision history, "the linux part" was added after I posted my answer. –  Tangurena Jan 23 '12 at 16:16
    
To see the revision history, click on the link in the "edited x hours ago" under the question. –  Tangurena Jan 23 '12 at 16:22
    
@Tanguerna: Ah, thank you for clarifying. Could you edit your question slightly so I can remove the down vote? Voting appears top be locked. –  Sardathrion Jan 24 '12 at 10:08
    
@Sardathrion, done. –  Tangurena Jan 24 '12 at 15:27
    
Done and upvoted -- Although leave a note saying that the Linux requirement was added later otherwise pedantic people like me will down vote you. ^_~ –  Sardathrion Jan 24 '12 at 16:02
add comment

I have used both Google Test and Boost Test. Both are xUnit frameworks, work on Unix and Windows, and are mature. I ended up using Google Test due to Google Mock since I wanted a mocking framework as well. Either should work well under buildbot.

Personally, I want the unit test framework to be minimal and fast. I want many tests and thus want them to run in a reasonable time frame. I personally like to use the main framework for a language instead of a mega tool but that is personal. If you wanted, you could look at continuous iteration testing: every time you save a file, it auto-runs the test cases that file depends on.

You could always write an Interface class to your unit test thus allowing a quick and effective change of frameworks.

share|improve this answer
    
+1 speed of iteration is the key codinghorror.com/blog/2007/02/boyds-law-of-iteration.html –  jk. Jan 23 '12 at 14:30
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.