Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Do you know of any xUnit framework that allows to run tests in parallel, to make use of multiple cores in today's machine?

If none (or so few) of them does it, maybe there is a reason... Is it that tests are usually so quick that people simply don't feel the need to paralellize them?

Is there something deeper that precludes distributing (at least some of) the tests over multiple threads?

share|improve this question

6 Answers 6

up vote 6 down vote accepted

NUnit 2.5 bundled pNUnit which allows the running of tests in parallel.

This release includes pNUnit, an extended NUnit runner for distributed parallel tests. The pNUnit program was developed at Codice Software for use in testing the Plastic SCM and has been contributed to NUnit. For more info about using pNUnit see the pNUnit site.

The JUnit side has parallel-junit as well as amino.

share|improve this answer
    
So the only reason for the other frameworks would be 'not implemented yet'? –  Xavier Nodet Mar 8 '11 at 20:54
1  
@Xavier Yes; that is a fair statement. –  Aaron McIver Mar 8 '11 at 20:56

To answer the second part of your question: Is there something deeper that precludes distributing (at least some of) the tests over multiple threads?

A great deal of code only works when run single threaded. It's trivial to accidentally produce resource contention and deadlocks when writing programs on the assumption that they will be run single threaded. And this works fine because most programs do actually run single threaded. Parallelism is gained by running multiple copies, or different programs at the same time (web scripts being one common example - lots of users accessing a single page means lots of copies of the scripts for that page running at the same time).

Imagine a simple "log to file" class. When you create an instance it opens the file for writing, when you free the instance it closes the file. So, the first test creates an instance and starts running a test. The second test does the same thing in a second thread. And fails, because the second instance can't get write access to the file. But if run one at a time all the tests would pass.

All of this can be coded around, and the simple example could be tweaked to work. But doing that is probably unnecessary for the original program. Having to write thread-safe code just so you can run unit tests is unreasonable for many people. So multi-threaded unit tests should remain an optional extra.

share|improve this answer
    
+1 This should be the excepted answer because it actually answers the why. –  Oliver Weiler Nov 22 '13 at 11:23

If tests need to setup and query a database, tests running in parallel would interfere with each other unless unless there's a separate database for each test running in parallel.

share|improve this answer
    
That's not for the testing platform (xUnit) to care about; that's an implementation detail. –  Aaron McIver Mar 8 '11 at 20:57
    
And not all tests written in a unit testing framework are unit tests, just like the one that accesses the database is not really a unit test, more like an integration test. –  c_maker Mar 9 '11 at 1:57

While JUnit per se may not allow it (I am not intimately familiar with its latest versions though), Maven with its Surefire plugin does have an option to run tests in parallel. I haven't tried it yet though.

I am not strongly pressed to investigate this option, as we have only a bit over a thousand tests and they run fast enough. However, I do know that some of the test fixtures have implicit dependencies between (we have found some such dependencies when some tests broke unexpectedly in the past), so there is a risk in that parallelizing the tests will make some of them fail unpredictably. You may say this is fine as it makes the problem explicit. However, we are dealing with a legacy system, and we have many more important issues to deal with - time is a scarce resource (as usual).

share|improve this answer

Multi-threaded coding is not trivial. Even when done by people who know what they are doing, timing dependent bugs can occur. They are difficult to fix. Having dealt with the one a few thousand case type bugs multi-treading can produce, I would prefer not to have them in my testing framework. The first fix I got appeared to work, but on further testing it was found that it had just become a one in tens of thousands bug.

The techniques to do multi-threading on multi processors are getting better with the advent of multi-processor PCs. However, it will take time before they are in wide use.

Some test suites have dependencies between tests that do not need to be explicitly stated when the tests are run in a single stream. However, on a multi-steam engine, they would need to be explicitly stated. (Where such dependencies should exist is a different question.)

From another standpoint, some things just don't need to be run in parallel. It the process runs adequately fast, it may better to focus efforts on things other than implementing multi-threading.

share|improve this answer

MBUnit is able to run tests in parallel simply by specifying some assembly level attributes.

[assembly: DegreeOfParallelism(6)]
[assembly: Parallelizable(TestScope.All)]

I've been using that project to run selenium tests in parallel quite successfully for some time. Unfortunately the project is not very much alive anymore.

xUnit 2.0 should also support parallel unit testsing but i haven't tried it out yet.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.