Is it always wrong to optimize before profiling?

Question

I see the mantra of "profiling before optimization" repeated again and again here, on SO, and elsewhere. Although I certainly use profiling tools, I'm only occasionally surprised by the results. It seems like, as often as not, the profiler is just giving the same information that you could reasonably deduce by knowing the likely execution path of your program, understanding how your architecture works, and having a good idea of what optimization techniques the compiler can employ for you.

Because of this, I generally find that when I'm developing, I see areas of the code that I can sense are going to be bottlenecks (often times when writing the code I will think to myself "hey, this is going to be a critical part of the code, and needs to be fast, but I will use a slower implementation first to prove the concept, then optimize it later") and I just go ahead and optimize these areas before I bother doing much profiling.

Is this really such a bad practice, or is it just the natural result of gaining experience doing optimization?

Edit

I just wanted to point out that I do use profilers, I'm just wondering how bad it's considered to be to pick off the low hanging fruit first, rather than waiting for the profiler to prove something is slow.

Edit 2

Thanks for the insights so far everyone. People have posted a lot of good comments. I have to say that I still have mixed feelings about some of the advice here, but it's given me a lot of insight into why people make the statement.

Answer 1

I think the golden rule here is "Everything in moderation".

If you are fairly certain a piece of code is going to prove to be a bottleneck, its not a horrible practice to do some initial optimization. At the very least, its a good idea to take steps to make sure it will be easy to refactor later.

What you want to avoid is going overboard by sacrificing time and readability in the name of micro-optimizations before you've seen actual data to justify such an effort.

Answer 2

The point of profiling before optimizing is that you need a baseline to determine how much improvement the optimization gave you. The "surprising" information I get from my profiler will be along these lines:

• I knew that method was called a lot, but I didn't realize it was called that much.
• Thread monitor contention almost never slows things down where I expect it.
• I'm generating that many instances of XYZ? It should only be about n...

That said, many times the profiler merely confirms my suspicions. Good scientific method involves healthy does of monitoring and experimentation. It's a bit difficult, but I do try to figure out more systemic problems with the profiler. I could do the obvious change and get a 5% improvement, or if I approached the problem differently I might be able to get a 25% improvement. (Most optimizations don't yield such a large improvement, but on occasion they do.) Of course, I wouldn't know how much my optimization improved without a baseline measurement.

Answer 3

The line between "optimizing" and just "sensible design" is sometimes fairly fine, but other times pretty obvious. Just for example, you don't need a profiler to be pretty sure that if you're sorting a few million items, it's worth using an O(N log N) algorithm rather than an O(N²) algorithm. IMO, that just falls under being reasonable sensible, not optimization though.

There are also some things you might as well do, simply because they might provide a benefit, and the cost is minimal to nonexistent. To use @unholysampler's example, writing ++i instead of i++ may have some minuscule cost if you're accustomed to typing it as a post-increment, but (at most) it's temporary and trivial. I wouldn't spend any time rewriting working code for the sake of possibly saving a nanosecond, unless the profiler had shown that I really needed that time, and stood a reasonable chance of saving there. At the same time, when I'm just typing in new code, I'd work at habitually using the form that's likely to be better, because once you do so habitually it's free. It frequently won't gain anything, and even when it makes a difference it often won't be large enough to notice or care about -- but it's still free, so there's not reason not to do it.

Cases like those are fairly obvious, and in my mind would really fall under sensible design rather than what I'd think of as truly optimization. Most other cases of "optimization without representation" would be considerably harder to justify though. If you're going to spend any significant time or effort on the optimization, you should have something much more solid than a "gut feel" to justify it.

I should add that part of the reason I say that is that I think profiling code is extremely useful, even when your goal isn't optimization. A profiler gives a high-level overview of a piece of code that can be extremely useful even when you don't particularly care about optimization at all. Just for example, if you see 10x as many calls to allocate a resource as to free that type of resource, it's a pretty good clue that there's a real problem, even if the code currently seems to run just fine. When you get down to it, a lot of code has a lot of things that should match up (not necessarily 1:1, but somehow or other) and a profiler can show mismatches like that much more quickly than most other tools.

Answer 4

"I can sense are going to be bottlenecks" (sic)

The problem with this statement is observer error. Just because you think it may be bad doesn't mean it IS bad. The profiler will give empirical evidence and keep you from spending time in an area that may give no improvement. That's why you should start with the profiler before optimizing. Let the program and profiler PROVE that it is a slow section first.

Answer 5

There are known performance killers and known best practices which avoid them. I don't think you need to profile to determine a cursor is slower in SQL Server than a set-based operation. If you start out knowing this, you will write code that performs better from the start without the need to profile the two options every time you write code.

If you are ajusting existing code, it is better to profile not only so you can be sure that the code you know is inefficient in general is working badly, but it can also show up problems you didn't know you had. I remember profiling one thing where I suspected the stored proc could be optimized (it could) and found out that it being called hundreds of times by the application when it only needed to be called it once.

Additionally, you have the benefit of being able to prove that you did in fact improve performance when you profile. I personally save these before and after figures and use them in my performance appraisal write-ups and in discussing my achievements when job hunting. It's nice to have actual figures to show how good you are at tuning.

Answer 6

The reason people say that is because you won't know what needs to be optimized before your profiling is finished, so you could end up wasting a lot of time for no benefits.

Still, if something leaps out at you as a bad idea (e.g. your coworker chooses to use deep recursion instead of a simple iterative loop) you can fix it. But you should be focused on moving forward, not naval gazing on old code. There is a stage in the process where that is appropriate.

Answer 7

I think everyone tries to write good code from the beginning. That sounds like what you're doing.

What I think should be done in the beginning is to just keep the design simple, especially the data structure. Often people start off assuming they need more sophisticated data structure, redundant data, and detailed notification techniques because they are worried about performance. In my experience, those things cause the problem they are supposed to avoid.

In spite of good coding practice and good design, performance problems creep in, and you need to remove them periodically. These are almost never things you could have guessed, and most profilers are not very good at finding them either. What's more, the optimization level of the compiler seldom has any effect on them, because mostly they are not tight compute-bound loops. Mostly they present as innocent-looking (or even invisible) function calls that, if you randomly snapshot the stack, are in the middle of it, and are consuming way more wall-clock time than you ever would have imagined, as shown by how often they appear there.

Here's my favorite example.

Answer 8

There are two kind of optimisation that can (relatively) safely be done before profiling.

1. Algorithmic optimisation: choosing an algorithm with better average (or worst) complexity. This can even (should ?) be done before beginning coding. You'll still have to check that with your real data set the selected algorithm is the correct one, but it is a good idea to start with an algorithm that is expected to fare better, isn't it ?

2. Data structure optimisation: using a correct way to layout you data, or a data structure with better locality can increase your performance, but it will have impact on the algorithm that can be used, so it is easier to do such an optimisation before coding (and thus you cannot use a profiler if there is no code). For exemple, when programming a video game, it is generally better to use struct-of-array (SoA) instead of array of struct (AoS) to store data as it will benefit from data locality, cache coherency, ...

Answer 9

There is the 90/10 law which states that 90% of the time is spent in 10% of the code and only 10% of the time is spent in 90% of the code. A programmer cannot always anticipate which 10% of the code will cause the most impact if optimised so until you use a profiler to figure this out, any optimisation you do is premature. Better to figure out where the problem is before attempting to solve it, no?

Answer 10

Yes it wrong to optimize before profiling BUT

Good programming, programming that makes code simpler and more straightforward doesn't require profiling. Good programming like moving unneeded initializations out of loops does not need anymore justification than the fact that you are improving the quality of the code.

IMHO the only times you need to profile is when you are specifically looking to improve performance. In order to show the improvement you must baseline first, and then show the delta.

Anytime you add complexity under the guise of optimization without showing proof that it was a bottleneck and the code does improve performance; is just doing bad programming.