Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Unit testing consists of executable code which exercise a certain functionality and then assert some conditions. The results are either Pass or Fail.

  • The objective is always to have all tests passed.

Software performance profiling usually requires help from some automated test/profiling tool, but the results seem to always require human interpretation, because:

  • There are no clear pass/fail objectives.
  • There are many facets (dimensions) in software performance, and one needs to consider the overall performance profile (the full picture; no pun intended) in order to interpret the result and make a decision.
  • There are many pitfalls in software profiling, such as results skewed by profiling overhead, unintended interactions between modules or operating system, test cases not representative of needs of customers.

Now that unit testing has been heralded as a major breakthrough in promoting the credibility of software engineering, is it possible to find the next silver bullet for the software performance issues?

Clarification on "automated to a high degree":

  • What I mean is that unit tests that were written long time ago (and updated as the software is developed) will remain useful indefinitely. During the better part of the software's production lifetime, there is little to no need of modifying the unit tests unless there are big changes to functionality.
  • On the other hand, performance testing seems to require adjustments and re-running continuously, even well into the production stage.

Related:

share|improve this question
    
C, Perl or PHP? The answer to this isn't quite language agnostic, if you're suggesting testing without inserting benchmarks in the code itself. –  Tim Post May 8 '11 at 17:53
add comment

3 Answers

up vote 2 down vote accepted

Firstly, you should not use profiling to determine performance. Profiling is intended to be used in order to identify the parts of your program which take the most time. When measuring performance, we only care about the speed of the whole program taken together. Profiling in that case only skews results. Instead, performance should be tested by benchmarks. The program processes some task, and we just measure how long it takes.

Secondly, you don't pass or fail performance tests. What we want to do is identify when something has made the program run slower. By running a benchmark against different revisions, you can easily obtain performance data for each revision. From there is would be fairly simple to automatically raise a warning on significant performance degradations.

http://speed.pypy.org, has done something like this. You can see the performance of PyPy as charts which show its performance. You can look at each benchmark and how it has changed over various revisions. You can actually see some revision which markedly decreased performance but were later fixed.

share|improve this answer
    
Thanks for the "instant winner" answer. Is there a reason why this isn't done on a larger scale, in a wider selection of languages? (Also, +1 for pointing out profiling != benchmarking) –  rwong May 8 '11 at 19:04
    
@rwong, speed.pypy.org exists because pypy is concerned with being a fast implementation of python. I suspect that relatively few other projects care enough about performance to take the effort. –  Winston Ewert May 8 '11 at 19:10
    
@rwong Of course it is done in many projects, but probably most of them are internal proprietary software. My shop always does performance benchmarks as part of the regression testing. As Winston said, it's not a pass/fail test, but if we saw the performance degrade by 10% or more, we'd investigate. –  quant_dev May 8 '11 at 20:22
    
"Secondly, you don't pass or fail performance tests." I don't understand where you're coming from with this. Yes, you can fail performance tests, if performance is specified as a requirement. If you have a requirement that it takes 30 seconds to go from receiving a message to producing output, then you can create test cases (automated or manual) to ensure that given the input, you produce output in 30 seconds or less. Not doing so is failing the test. Is this not what you mean? –  Thomas Owens Sep 13 '11 at 23:07
    
@Thomas Owens, certainly there are some performance tests that you can pass/fail. But the common case, at least in my experience, is that we don't have a hard requirements limit. Rather we just want to keep tabs if the performance changes. –  Winston Ewert Sep 13 '11 at 23:34
add comment

@Winston has some good points, but I must disagree in one context: when the specification (feature) includes service-level agreements (SLAs) then you should absolutely automate performance profiling (aka stress testing) if at all possible, and the software must pass by meeting or exceeding the metric specified by the SLA.

share|improve this answer
    
Most SLAs are against a complete package - Unit tests cannot confirm the performance of a complete package, only the unit. If the designer added performence requirements at the UT level, then you could, but I have never seen it done, so you can't. –  mattnz May 9 '11 at 8:05
    
@mattnz: I use TDD, so I don't test units, I test features. –  Steven A. Lowe May 9 '11 at 15:08
add comment

Why worry about Unit performance at all? Whats the problme it solves? Unless you have a historical problem with a particular unit, in which case perfformance just becomes another functional requirement. I suggest that spending one more micro second thinking about performance is too much - you will spend all your time micro-optimising code that hardly ever runs, and then wonder why you are shipping late, over buget, incomplete and buggy code.

Better slow than never I say.......

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.