Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I just came across the Caliper project and it looks very nice. Reading the introduction to microbenchmarks, one gets the feeling that the developers would not suggest to use the framework if the benchmark takes longer than a second or so. I looked at the code and it looks like a RuntimeOutOfRangeException is actually thrown if a scenario takes longer than 10s to execute.

Could you explain to me what the problems are with running larger benchmarks?

My motivation for using Caliper was to compare two join-algorithm implementations. Those will definitely run for quite some time and will do some disk IO, yet running the entire database would make it hard to do the comparison, because the configuration of the algorithms and the visualization of the results would be a pain.

share|improve this question

migrated from stackoverflow.com Jun 25 '12 at 22:12

This question came from our site for professional and enthusiast programmers.

    
@Robert Harvey and comingstorm: I know that microbenchmarks are great for profiling code with short execution times, and I also see how they can be very useful for unit tests. What I'm really interested in though, is whether there is a specific reason why using the same framework would not work for longer running tests. –  Konstantin Weitz Jun 25 '12 at 22:30
    
See my updated answer. Tests that take that long to execute are generally not unit tests (integration or acceptance tests, maybe). –  Robert Harvey Jun 25 '12 at 22:32
    
My guess is that TDD is the reason why the "micro" is in micro-benchmarks. If you want to try using Caliper for longer-running benchmarks instead, you can try changing the time limit and see if that's the only thing holding you back... –  comingstorm Jun 25 '12 at 23:07
1  
In other words: if you're willing to experiment a bit, I'd guess there's a good chance you can make it work -- even though your use case is probably a bit different from the one Caliper was designed for. –  comingstorm Jun 25 '12 at 23:16
    
> to compare two join-algorithm implementations You might be interested in this paper -- 'Statistically Rigorous Java Performance Evaluation' and the JavaStats benchmarking scripts provided by the authors. –  igouy Sep 28 '12 at 15:26
add comment

3 Answers

up vote 4 down vote accepted

One of the reasons for using caliper rather than just naively writing your own benchmarking code is that caliper goes a long way to avoid some of the pitfalls of writing micro-benchmarks. For example, a good benchmark should first run code in a warmup phase that ensures all relevant code gets JITed. It should also ensure that doing twice as many iterations takes twice as long; if not, it suggests that the JVM is optimizing away the work.

All of this means that to properly benchmark code, caliper needs to run lots of iterations. For a task that should take something on the order of microseconds, this is reasonable. But if a single iteration will take more than 10 seconds, this means that the caliper run will take a very long time indeed. My assumption is that the Caliper team decided that because of this, a task that takes more than 10 seconds to run was far more likely to be the result of programmer error than an intentional choice to write a benchmark that would take hours to run. By bailing early, it helps programmers realize quickly that they need to fix a bug, rather than having their machine spin away for hours.

share|improve this answer
add comment

One idea with test-driven design is to run all your unit tests when building a project, which allows you to catch functional regressions quickly, before committing the changes that caused them. For this to work, though, all your tests need to complete quickly.

If your benchmarks all run quickly, they can be included in your test framework (I don't know if that is what Caliper is supposed to be for, but Google is heavily into TDD...). In any case, the whole point of adding benchmarks to your test-driven infrastructure would be to catch major performance regressions quickly -- so, for that kind of usage, you'd want to flag taking too long as an error.

share|improve this answer
    
I see how it is useful to flag long running benchmarks as errors. But couldn't one just introduce a parameter that can optionally specify what 'long' is. –  Konstantin Weitz Jun 25 '12 at 22:32
    
Yes -- and, I'd expect there to be such a parameter. But if 10s is your institutional limit on unit test execution time, it might make sense to build it in as a fallback sanity check for your micro-benchmarks. –  comingstorm Jun 25 '12 at 22:36
1  
Seems like it would make more sense for the benchmarker to keep track of how long the same test has taken in the past and not throw the red flag until there's been a statistically-significant number of runs and the test time exceeds some number of standard deviations above the mean. –  Blrfl Jul 3 '12 at 23:28
add comment

Many code slowdowns occur in small, tight loops. If you know that's where your time is being spent, a micro-benchmark can help you optimize just that small bit of code.

Unit tests need to complete in a reasonable amount of time. It's rare indeed for me to write a unit test that takes longer than a second or two to execute, since I might have two or three hundred such tests or more in a test assembly.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.