Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I know that profiling is useful to identify bottlenecks and determining what parts of the code require how much time to execute. The latter isn't always very easy to track in the midst of other paths being executed, so once I decide what I want to optimize it might be problematic to see the improvement in numbers. This is especially true in desktop apps which run constantly and it is difficult to: execute the same path and execute it the same number of times to have reliable comparison.

It won't help me if before optimization the function ran X times and took 500 milliseconds, and after optimization it run Y times and took 400 milliseconds.

In such cases, can I somehow use a profiler to determine improvement or do I have to resolve to other options?

share|improve this question
3  
So you trust the profiler to identify the bottleneck but not to see if the bottleneck is still there? – Anthony Pegram Feb 18 '13 at 18:14
    
@AnthonyPegram I guess I've phrased myself incorrectly. I've modified the question to make it clearer. I am not talking about big bottlenecks but rather about not-easily replicable scenarios where the gain might not be as large as in your average bottleneck. – Maurycy Feb 18 '13 at 18:39
2  
@MaurycyZarzycki If you can't reliably see the problem (or the effect of the problem) then how do you know you've identified a bottleneck at all? – Caleb Feb 18 '13 at 18:43
1  
You can, however, calculate the per-execution time for both versions by dividing time by the number of executions. That's really the metric you want to compare. – Blrfl Feb 18 '13 at 18:48
    
Download the 30-day trial of Intel's (buggy) vTune and profile your app with that. It should show you in far more detail than you need what's going on – James Feb 18 '13 at 19:52
up vote 5 down vote accepted

once I decide what I want to optimize it might be problematic to see the improvement in numbers.

One has to wonder whether you've really found a bottleneck, and also whether you've really eliminated one, if you can't see the effect in the profiler.

This is especially true in desktop apps which run constantly and it is difficult to: execute the same path and execute it the same number of times to have reliable comparison.

This is one of the things that unit tests can help you with. You should be able to recreate any situation that interests you in a test, even (especially!) those which might be very difficult to replicate in the wild. You can also repeat that test as many times as you want to. If you run the profiler against your test, then, you should be able to see the effect of whatever improvements you make. If you can't measure the effect of the improvement, you really have no grounds for claiming that you've improved the code at all.

share|improve this answer
2  
+1 "If you can't measure the effect of the improvement, you really have no grounds for claiming that you've improved the code at all." – Michael Shaw Feb 18 '13 at 19:44
    
You are right, pretty much. I am too used to measuring performance relatively to a different implementation, not as how it relates to the whole application performance. I guess it comes from my game development background where performance is important everywhere. And my microoptimization deviation. – Maurycy Feb 18 '13 at 20:12

In such cases, can I somehow use a profiler to determine improvement [...]?

YES!

It's very helpful to just repeatedly run the profiler over your code as you are tuning. It also points out not only how the times are changing as you tune but also how the dynamics are changing (reduced branch mispredictions, cache misses).

Also as you go, you can watch how the hotspots kind of reshuffle. For example, after optimizing your top hotspot, it might drop to 8th place and the system puts a focus on a new hotspot (and it might not even be what was shown as the second hotspot initially due to changing dynamics, especially if you are shifting memory layouts for cache-friendliness and exploring alternate data structures).

Building Intuition

I find that these kinds of sessions also help me understand the nature of the hardware at an intuitive level just a little teeny bit more each time just by watching the cache misses, branch mispredictions, and the way the hotspots and shuffling with each change. I'd never claim to be able to predict the nature of the hardware though (why I lean so heavily on the profiler) but profiling repeatedly does help me kind of develop some sense of intuition, even more so when my efforts make things slower (learning from mistakes).

Building that intuition helps me design more efficient code the first time around without degrading maintainability, and also helps me understand how to design my interfaces better in a large-scale system in a way such that optimizations can be applied without changing the interfaces (only local implementation details). Understanding how efficiency works at the computer architecture level can help you write cleaner code, since hard-to-maintain code often comes about from dystrophy of repeated iterations resulting from unforeseen changes that degrade maintainability until the code is eventually refactored. Understanding performance a little better upfront can help mitigate that kind of degradation by getting your designs more correct the first time around in places where you know in advance that you'll be dealing with some very tight loops over very large inputs.

What Not to Optimize

For people who are kind of reluctant to use profilers (too proud after getting decent results without one for years, e.g. -- been there, done that *), you might find that you are reasonably successful at spotting the number one hotspot in your system on your own without the aid of such precise measuring tools.

* As an embarrassing personal story, I was one of those stubborn types insistent on not using a profiler, and went for almost 20 years without using one. It's because I was getting pretty decent results without one and knew my way around the code, a kind of classic case of man vs. machine. The thing that changed it and forced me to use one was a conflict scenario where accusations were made among a team about who was causing bottlenecks in the codebase based on superstitions about what language features may or may not be slow. I finally reached for a profiler then to try to settle the matter objectively, only to realize I found a tool that was almost as invaluable as a debugger through the process. It is a natural inclination for a veteran in the industry who has done well without a profiler to look down upon one, but once you embrace one, the whole superstition against them will start to seem really silly. It's not about our ability to write efficient code. It's about our ability to do that more productively.

Yet the greatest immediate value of a profiler, perhaps, is in telling you what not to optimize. Here is an example of a personal profiling session when loading a text-based format (OBJ) format containing 16 million polygons:

enter image description here

Drilling into the #1 hotspot, it shows the times are significantly dominated by string parsing with functions like sscanf. Without the profiler, I would have never guessed that the mesh engine took so little time. I would have been tempted to at least tweak and tune functions to create polygons and so forth otherwise. My hunches there would have been that the mesh system contributed at least a good 35-50% of the time with the other in string parsing, when the mesh engine is actually only contributing about 12%.

So the profiler immediately gave me a tremendous boost to productivity by showing me that all those parts I suspected would have taken at least a bit more time here don't need to be optimized. That's something that only the accuracy of a profiler can provide (alternative human measurement techniques generally don't come close, even if they can do an okay job at pointing out what to optimize).

If you use a profiler for a good number of years, optimization will cease to seem like some absolute, independent metric. It'll start to seem like a productivity metric. You start to appreciate what the profiler does for you productivity-wise more than just pointing out bottlenecks. When the profiler shows you what doesn't need optimization, you can write such code to be as easy to maintain as possible without a single performance worry.

Testing for Correctness

It's critical to lean on your unit and integration tests before you go too deep, and run them with each optimization attempt, since it's horrible if you find after 3 iterations worth of optimizations that your tests fail after you get excited over the improvements, only to then realize you introduced a glitch and have to back them out (something I've done a few too many times to care to admit).

Example Session

Here's a recent profiling session I applied with VTune. I made no algorithmic improvements, only micro-optimizations like multithreading, SIMD, and the bulk of the times dropping as a result of memory optimizations (fixed allocator, more contiguous layouts, shuffling data members in structs to reduce padding, hot/cold splitting, etc). Towards the end there most of my attempts to speed things up were unsuccessful and I had to back out my changes half the time, and the few improvements that were successful were getting increasingly diminished returns (with noise making it difficult to even tell if I made a genuine difference). That's usually a good sign to kind of stop and rest and let the code breathe for a while.

T-Rex (12.3 million facets):
Initial Time: 32.2372797 seconds
Multithreading: 7.4896073 seconds
4.9201039 seconds
4.6946372 seconds
3.261677 seconds
2.6988536 seconds
SIMD: 1.7831 seconds
4-valence patch optimization: 1.25007 seconds
0.978046 seconds
0.970057 seconds
0.911041 seconds

I like to open up notepad when I do this stuff and just record the times from VTune with each step, also with some "checkpoints" recorded where I did something significant (like the SIMD optimization). It helps me keep track of the times and how they relate to each iteration of change.

Also when you go through an optimization crunch like this, it's useful to start a new branch in your version control system and record the time differences in your commit logs as well.

Start Off Big

One thing you want to do is make sure you use a large enough input with big times showing up initially (bigger than you would initially be tempted). Otherwise you can find that after you optimized things 10x, e.g., the times are getting too small to effectively measure without noise messing them all up, and that can tempt you to reach for a bigger input to inflate the times again, but that makes it harder to keep track of the improvements. So initially it's good to make a test that takes like 30 seconds, 60 seconds, even if it's annoying to wait that long, since you might end up optimizing to the point otherwise where the times are getting too small to effectively measure.

Performance Test

This is especially true in desktop apps which run constantly and it is difficult to: execute the same path and execute it the same number of times to have reliable comparison.

This might be overkill but I've found it incredibly useful here for very coarse integration testing and also performance testing to be able to write scripts for your application at a high-level, almost macro-like fashion (expressing a user-end action with a line of code). It's how I tend to do profiling sessions -- I run a little lua script which looks like this (pseudocode):

start application
load some file
perform some operation, possibly multiple times
quit application (this also reports the total time the application ran)

I use LuaJIT for the purpose (mainly because its FFI makes it possible to directly call exported C functions from Lua) though that's just a personal preference. With that, it's easy to just write a few lines of script that drive your complex application and run a profiler like VTune over it with command arguments that specify the script to run. Users might also appreciate this ability to externally drive the application and make it perform batch jobs.

If you end up doing this, it's also invaluable for bug reports. If a user reports a bug with a minimal number of steps and possibly content to reproduce an issue, you can start off writing this kind of high-level macro script to reproduce the bug and make the test fail. Then you have the simple task of making the test succeed, and can include it as part of your CI process.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.