Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Do you do any performance testing in any repeatable and automated way? Do you trigger it as a part of CI builds? What tools do you use? How do you spot changes in trends?

I'm looking for how-tos and description of working set-ups for end-to-end automated performance testing approaches.

share|improve this question
    
Can you tell us something about your application? Is it a web application, desktop app, is tight performance very important? Or are you just wanting alerts for major regressions? –  Ethel Evans Mar 18 '11 at 0:40

2 Answers 2

up vote 2 down vote accepted

Repeatable - yes. Automated - not fully. We have automated the generation of test data for our performance tests, and the test driver. The tests themselves are triggered by hand whenever necessary. Such as before releases, or platform changes (right now we are migrating our legacy project from Java 5 to 6, and we are about to conduct performance tests to compare the results of different JVM settings). So they are not part of our CI builds. That would in fact be technically challenging as we need to regenerate a lot of test data each time, and the process, albeit automated, is time consuming.

We use mostly tools developed in-house for the testing, tailored specifically for our app. The app is a frontend to an ancient mainframe server, which is the backbone of the whole company. The test data needs to be generated on the mainframe, so there is lots of communication overhead. Also, it can't be really randomized at the moment - we use a set of predefined templates combined with data coming from Excel sheets to create new test data. The test driver then sends requests to our server in standard HTTP, and a proprietary XML based format.

share|improve this answer
    
Care to elaborate on automation of test data generation, and test driver? –  Yuriy Zubarev Mar 18 '11 at 0:26

We are just getting started with load and performance testing. Right now, we're doing simple tests: Run operation X for Y iterations, possibly with some level of randomness to inputs or things that could happen during operation (e.g., close the connection occasionally and make the program recover). Then automatically verify various interesting facts - e.g., that all Y operations completed successfully in under Z seconds, no errors were logged, etc. Note that there are a lot of things to define before you can test. What is "completed"? What is "success"? How big should Z (the number of seconds) be?

As I said, we are just getting started; we made our first test last week, literally (we began work on the product a couple of months back). We actually are setting up CI at the same time, and haven't put testing and CI together yet. I don't think we will be putting load / performance tests into CI right now, but we could do it eventually for small operations by running them for some number of iterations - probably estimated to be 5 to 10 seconds of work - then verifying that the operations took the expected amount of time, and checking any other aspects of load or performance that we're uncertain about.

Currently, we're doing a nightly run that includes our end-to-end tests (which will take 3 to 4 minutes each) and our load tests.

Our system is currently pretty simple. We're creating special performance test fixtures that are just like our normal fixtures except that they run the same operations over and over, then verifying results by checking database timestamps, counting operations, or looking at logs. Later, we'll have to worry about load-testing web sites, imitating users, etc., and we're still figuring out what tools we can use to handle that problem.

share|improve this answer
    
Have you ever considered manipulating the processor affinity to make the tests run on a dedicated core/processor? That would limit the number of context switches (cache swapping, stack swapping, etc...) and probably give more consistent results. I've read/thought about this concept a lot in the past but haven't had a reason to implement it yet. BTW, +1. It's interesting to hear about how you guys are handling performance testing. –  Evan Plaice Mar 18 '11 at 12:54
    
@Evan, we're not (yet) dealing with a situation where such high levels of consistency are necessary or even valuable, and probably will never get to that point. We're not dealing with a system that needs to be highly optimized, we just need a heads-up about (a) how much traffic we can handle, and (b) if a code change causes a serious impact to our performance that we can't explain (suggesting a bug). I know one of my old teams was always fighting weird inconsistencies in context switches, however . .. sounds like something they might have been interested in trying –  Ethel Evans Mar 18 '11 at 19:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.