Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

In my current project, I am having a hard time coming up with a good solution to create scalable integration tests that have no side effects. A little clarification on the side effect free property: it is mostly about the database; there shouldn't be any changes in the database after the tests are completed (state should be preserved). Maybe scalability and state preservation don't come together, but I really want to push for a better solution.

Here is a typical integration test (these tests touch the database layer):

public class OrderTests {

    List<Order> ordersToDelete = new ArrayList<Order>(); 

    public testOrderCreation() {
        Order order = new Order();
        assertTrue(order.save());
        orderToDelete.add(order);
    }

    public testOrderComparison() {
        Order order = new Order();
        Order order2 = new Order();
        assertFalse(order.isEqual(order2);
        orderToDelete.add(order);
        orderToDelete.add(order2);
    }
    // More tests

    public teardown() {
         for(Order order : ordersToDelete)
             order.delete();
    }
}

As one might imagine, this approach yields tests that are extremely slow. And, when applied to the whole integration tests, it takes around 5 seconds to test only a small portion of the system. I can imagine this number going up when the coverage is increased.

What would be another approach for writing such tests? One alternative I can think of is having kind of global variables (within a class) and all test methods share this variable. As a result, only few orders get created & deleted; resulting in faster tests. However, I think this introduces a bigger problem; the tests are no longer isolated and it gets more and more difficult to understand & analyze them.

It might just be that integration tests are not meant to be run as often as unit tests; therefore low performance might be acceptable for those. In any case, it would be great to know if someone came up with alternatives for improving scalability.

share|improve this question

5 Answers 5

up vote 6 down vote accepted

Look into using Hypersonic, or another in-memory DB for unit testing. Not only will the tests execute faster, but side-effects aren't relevant. (Rolling back the transactions after each test also makes it possible to run many tests on the same instance)

This will also force you to create data mockups, which is a good thing IMO, as it means something that occurs on the production database can't inexplicably start failing your tests, AND it gives you a starting point for what a "clean database install" would look like, which helps if you need to suddenly deploy a new instance of the application with no connection to your existing production site.

Yes, I do use this method myself, and yes it was a PITA to set up the FIRST time, but it's more than paid for itself in the lifetime of the project I'm about to complete. There are also a number of tools that help with the data mockups.

share|improve this answer
    
Sounds like a great idea. I especially like the complete isolation aspect of it; starting with a fresh database install. It seems that it will take some effort, but once setup, it will be very beneficial. Thanks. –  Guven Oct 21 '11 at 17:54

This is the eternal problem that everyone faces while writing integration tests.

The ideal solution, particularly if you're testing on production, is opening a transaction in setup and rolling it back in teardown. I think this should suit your needs.

Where that isn't possible, for example where you're testing the app from the client layer, another solution is to use a DB on a virtual machine and take a snapshot in setup and go back to it in teardown (it doesn't take as long as you might expect).

share|improve this answer
    
I can't believe I haven't thought of the transaction rollback. I will use that idea as a quick fix to the solution, but ultimately the in-memory DB sounds very promising. –  Guven Oct 21 '11 at 17:55

On scalability

I've had this problem some times before, that the integration tests were taking too long to run and were not practical for a single developer to continuously run in a tight feedback loop of changes. Some strategies to cope with this are:

  • Write fewer integration tests - if you have good coverage with unit tests in the system, the integration tests should be more focused on (ahem) integration issues, on how different components work together. They're more similar to smoke tests, that just try to see if the system still works as a whole. So, ideally, your unit tests cover most of the functional parts of the application, and the integration tests just check how those parts interact with each other. You don't need extensive coverage of logical paths here, just some critical paths through the system.
  • Run only a subset of the tests at a time - as a single developer working on some functionality, you normally have a sense for which tests are necessary to cover that functionality. Run only those tests that make sense to cover your changes. Frameworks like JUnit allow you to group fixtures in a category. Create the groupings which allow for the best coverage for each feature that you have. Of course, you still want to run all the integration tests at some point, maybe before commiting to the source control system and certainly at the continuous integration server.
  • Optimize the system - Integration tests coupled with some performance tests might give you the necessary input to tune slow parts of the system, so that the tests will run faster later. This might help uncover needed indexes in the database, chatty interfaces between subsystems or other performance bottlenecks that need addressing.
  • Run your tests in parallel - If you have a good grouping of orthogonal tests, you could try running them in parallel. Be attentive to the orthogonal requirement I mentioned though, you don't want your tests stepping in each others feet.

Try combining these techniques for greater effect.

share|improve this answer
1  
Really good guidelines; writing fewer integration tests is for sure the best advice. Also, running them in parallel would be a very good alternative; a perfect "test" to check if I got my tests straight in terms of isolation. –  Guven Oct 21 '11 at 18:00

Integration tests should always be run against the production set-up. In your case it means you should have the same database and application server. Of course, for the sake of performance you could decide to use an in-memory DB.

What you should never do though is to extend your transaction scope. Some people suggested to take over control over the transaction and to roll it back after the tests. When you do this, all entities (assuming you are using the JPA) will stay attached to the persistence context throughout the test execution. This may result in some very nasty bugs which are very hard to find.

Instead, you should clear the database manually after every test through a library like JPAUnit or similar to this approach (clear all tables using JDBC).

As of your performance problems, you shouldn't execute the integration tests on every build. Let your continuous integration server do this. If you use Maven, you might enjoy the failsafe plug-in which allows you to separate your tests into unit and integration tests.

Also, you shouldn't mock anything. Remember you are integration testing, i.e. testing behavior within the run-time execution environment.

share|improve this answer
    
Good answer. One point: It may be acceptable to mock some external dependencies (e.g. sending email). But you shoould rather mock by setting up a complete mock service/server, not mock it in the Java code. –  sleske Oct 22 '11 at 8:08
    
Sleske, I agree with you. Especially when you are using the JPA, EJB, JMS or implementing against another specification. You could swap the application server, persistence provider or database. For instance you might want to use embedded Glassfish and HSQLDB to simply set-up and improve speed (you are of course free to choose another certified implementation). –  BenR Oct 22 '11 at 8:22

For testing purposes we have used a file based deployment of an SQLite database(just copy a resource). This was done so we could also test schema migrations. As far as I'm aware, schema changes are not transactional so will not be rolled back after a transaction is aboted. Also not having to rely on transaction support for test setup allows to test the behaviour of transactions from your application properly.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.