Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Today I was looking at some blogs on Ruby and Python. A Python loyalist wrote a few lines of code with a loop to show how Python is faster than Ruby, with a similar code. I also found a blog of a Ruby disciple who says that it is wrong and he too submitted a code sample and benchmarking results.

I am wondering whether background programs or background processes may sometime hinder the results and we may not get exact benchmarking scenario.

Please correct me if I am wrong.

share|improve this question
1  
Benchmarking the execution speed of loops in Ruby in Python is utterly missing the point. Benchmarking the productivity the programmer that uses them -- while much more difficult -- is entirely the point. –  Adam Crossland Nov 16 '10 at 21:43
    
@Adam: Yes, you can just Google and find some samples. –  RPK Nov 16 '10 at 22:03
    
Maybe the Ruby disciple just wrote better Ruby than the Python loyalist - better to let Ruby disciples wrote Ruby and Python loyalists write Python shootout.alioth.debian.org/u32/… And remember the phrase "time / space tradeoff" –  igouy Dec 12 '10 at 0:47
add comment

2 Answers

up vote 4 down vote accepted

Yes they can.

Normally just common sense in taking results from a couple of benchmarking runs, having some idea of what to expect, and being aware of what background services might fire up on your machine (beware indexing/search services in particular) and slow it down will generally avoid being seriously mislead by results. I've found this to be more of a problem with automated performance testing where there is no human in the loop to go "WTF... ? Oh the stupid foo service just started an update".

For purely compute benchmarks, I've found running the benchmark process at a higher than normal priority can usefully give more consistent, stable results on "busy" machines. This may not work if I/O is a factor though (depends whether your OS' notion of priority just affects CPU scheduling or I/O too).

share|improve this answer
    
I think for this type of tasks, command-line is best. –  RPK Nov 17 '10 at 11:55
add comment

It's possible.

But if that's the case, running the benchmark again would almost certainly produce different results. That's why benchmarks are always run multiple times, to make sure that there are no background processes that are skewing the results.

share|improve this answer
1  
More accurately: Multiple runs make the effects of background processes statistically insignificant. –  greyfade Nov 16 '10 at 21:51
    
removing outlier timings is usually the best way of eliminating the effect of background processes.... of course this still needs multiple runs :-) –  mikera Jul 15 '11 at 14:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.