Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm no computer scientist so please bear with me.

Lately I wrote a python program that has to perform some expensive spatial searches while holding lot's of data in memory. I tried to improve the performance, that is tried different approaches (see my question here for example).

First, and maybe not so surprisingly, I found great performance differences when running the same code on two different machines (up to 6 x faster on one machine).

Then however and very much to my own surprise, running the exactly same code on the same machine on two different days showed great performance differences.

To give you a rough idea what I mean by great.

Machine 1, day 1: 11 minutes

Machine 1, day 2: 2 minutes

There was no reboot in the meantime by the way.

So my question really is: what could be possible reasons for this difference? How can I can I make sure my system is ready for high performance when I start running my program?

share|improve this question
2  
Are the same (other) programs running each time? –  thegrinner Oct 4 '13 at 13:11
    
Well, yes there are some other programs running in the meantime. –  LarsVegas Oct 4 '13 at 13:12
1  
Turn off your virus scanner... –  Steven Schlansker Oct 5 '13 at 5:26
    
cold start vs warms start –  jk. Nov 14 '13 at 14:33

4 Answers 4

up vote 8 down vote accepted

This has a lot to do with the OS, the other processes running, and the amount of memory being used. The OS will let each process have a certain number of CPU cycles before it switches to the next process. If there a lots of other processes, it will be longer between each time your program gets access to the cpu. Also, since your program holds lots of data in memory, when the cpu switches to a different process, this data may have to be moved out of memory. This will require it to be temporarily be written to disk which is a slow process (relative to other operations in a computer).

Basiclly there are many factors that can cause the difference in time. However in most systems, you can set your program to run with greater importance than other processes. This will differ based on the OS you run. That could help to speed up the running time because your program will be given a higher priority to run over other processes.

share|improve this answer
3  
+1, but just to this, if your program connects to a network (or the internet) for any reason, that could cause this as well. –  Morons Oct 4 '13 at 13:17
5  
Also it may be that the program was loaded into to memory, or need some form of compilation on the first run.. so that made it slower. It not uncommon for the the first run of code to be slower than all subsequent runs. –  Morons Oct 4 '13 at 13:18
    
Since it's a python program, there was definitely some form of compilation on the first run, and the second run probably used the cached .pyc file(s) to avoid it. –  Useless Nov 14 '13 at 15:11

First, and maybe not so surprisingly, I found great performance differences when running the same code on two different machines (up to 6 x faster on one machine).

Then however and very much to my own surprise, running the exactly same code on the same machine on two different days showed great performance differences.

Performance measurement is a surprisingly hard topic and huge differences such as what you observed are very common. There is a lot of known measurement bias such as:

  1. OS noise, as suggested by DFord's answer
  2. OS policies (such as scheduler, memory mapper and IO-cache)
  3. Environment variables, even unused ones
  4. Memory alignment
  5. Memory localisation

Each of these parameters may greatly influence the performance of your program. If you want to prepare your program “for high performance” then:

  1. Devise strategies to sanitize environment so that you gain some control over parameters 1-5 above.

  2. Devise tests to determine which combination of parameters do well.

While this is not really the question you asked, I guess you are interested in performance measurement. Here are a few references I suggest to you:

http://compilers.iecc.com/comparch/article/96-02-165

Boehm commenting about execution speed for the same binary varying by a factor of 2 depending on cache placement

http://www.inf.usi.ch/faculty/hauswirth/publications/asplos09.pdf

Producing Wrong Data Without Doing Anything Obviously Wrong! It is very easy to read, contains surprising results, and give a very good feeling of the difficulty of performance measurement and a few methods to produce (more) reliable results.

http://plasma.cs.umass.edu/emery/stabilizer

“Stabilizer is a compiler and runtime system that enables statistically rigorous performance evaluation.” There is a scientific publication claiming to prove that -O3 and -O2 in gcc are not distinguishable, you will find plenty of interesting references in there.

As a final word, you just discovered with your measurements that wall clock does not provide a useful information when it comes to measure performance of programs, so that you will never get a useful answer to a question such as “how long runs my program?” and you have to consider more precise statements.

share|improve this answer

Radical, unexplained, apparently random changes in execution time are a consequence of the chaotic nature of such complex systems. and by complex systems, i go so far as to include, BUGS, as part of the complexity. You may be hitting a major bug somewhere in your programs, a bug that only manifests itself in certain situations and slows your program seemingly randomly ( until you figure it out )

see this bug in SQL server for an example of tracking this down.

http://support.microsoft.com/kb/2570501

so, the point is, you add more processors, your programs gets slower. until you isolate and correct this bug, a bug in SQL server itself.

share|improve this answer

Using two different machines to compare a program is out of question. There are so many factors that affect the results (different hardware, different software, caches, other processes running, etc.) that what you get has no value.

When using the same machine, you can do some things to make the results more reliable. A few examples: (1) Always reboot before starting a new benchmark. It's the only way of ensuring that all caches are cold. (2) Run the same benchmark multiple times (eg. at least five) and average the results instead of relying on a single execution. (3) Consider switching into single user mode or stopping all the rest running programs/services (unless they're absolutely necessary), since they might affect the execution time of your program.

If you want to know more about how hard proper benchmarking is I recommend A Nine Year Study of File System and Storage Benchmarking

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.