Does profiling without benchmarking lead to micro-optimization?

Question

Read first: For a definition of the two terms ("profiling" and "benchmarking") and the need for distinguishing them, please read this answer to a previous question.

I have to admit that until I saw Winston Ewert's answer, I have never thought of the need to distinguish the two techniques. I simply think that "profiling" can be applied at different "scale levels" of software, and that when it is applied on the higher level, the profiling code in the lower levels ought to be turned off in order to lower the aggregate overhead.

After I reflected on the answer, it might have explained why I fell prey to micro-optimization in my previous project.

In an effort to optimize during that project, I implemented a low-overhead profiler (inserted into the source code) which is good at generating accurate profiling results at the millisecond level. I then spent all days tinkering with it, and optimized a lot of code based on the profiler's result. In the end, I was successful in reducing the computation part of the project from several seconds to less than a fraction of a second.

The next thing I learned, to my horror: when the optimized module was used in a larger project, I/O and data conversion completely dominated the module's computation time. The non-computation part is in the range of 1-2 seconds, making my optimization efforts moot.

To this date, I still haven't got a chance to do a true "benchmarking", though I am going to give it a try very soon.

Given that "Did you do profiling?" has become the cliche on both StackOverflow and Programmers.SE, is there a danger that my kind of ignorance is actually prevalent among fellow developers? Does this ignorance lead to micro-optimizations all over the places?

Answer 1

From what I've seen the "did you profile?" question always comes after "why does this run so slow?" so the "benchmarking" has been done and the result was "too slow" and now we're trying to figure out why it's running so slowly so we go and "profile" it.

Real life is usually more complicated. How fast your software is depends on the architectural decisions you make, algorithms you choose, whether or not you've correctly identified and dealt with various bottlenecks and system constraints. Getting stuck optimizing a system that isn't designed for performance is an easy trap to fall into and can suck away huge amounts of time for little reward. On the other hand, not every software has high performance as a requirement.

Profiling and optimizing before you benchmark, i.e. before you know whether or not the performance is adequate is truly falling into the premature optimization scenario.

I like this quote from Wikipedia:

“The First Rule of Program Optimization: Don't do it. The Second Rule of Program Optimization (for experts only!): Don't do it yet.” - Michael A. Jackson

Answer 2

He who does not honor the small is not worthy of the large.

Even though you may not make the specific feature significantly faster, you may safe some cycles which might help the system as a whole. Taking small bits of time off of many links in a chain may add up to being more then just combating the weakest link.

Of course you need to be careful about putting too much time in it. Small tweaks do stack up though. Small tweaks are also best done while your code is fresh. They won't typically show up in a profiler since they are small and are many all over.

First make it work then make it fast. Sometimes it's just fast enough, even though there is plenty of opportunity for improvement.

Sometimes it's not "premature" if it's hard to do later.

Answer 3

Profiling must be against realistic scenarios (call that benchmarking if you like). (Is that a "Dugh"?)

O(n²) solutions severely beat O(n log(n)) ones by a large margin given sufficiently small datasets. That is well known.

The programs that most developers write first-hand do not scale to even one order of magnitude above. It is a project manager's responsibility to make sure that everything is tested against close-to-real-life test datasets and scenarios, and to executions/second per module when need be.

Risk management is not about being pessimist. It's about considering bad and fatal scenarios in the design and the procedures.

Answer 4

Given that "Did you do profiling?" has become the cliche on both StackOverflow and Programmers.SE, is there a danger that my kind of ignorance is actually prevalent among fellow developers? Does this ignorance lead to micro-optimizations all over the places?

I think so, given the kinds of questions and answers that travel about these sites, and the existence of profiling myths.

It is common to hear people putting timing code into their routines, because they are unhappy or puzzled with what profilers tell them. It's also extremely common to hear people doing micro-optimization whether or not they have tried profiling.

I think part of the problem is the word "profiling" itself. It is often conflated with "measuring", when finding performance problems is not at all the same as measuring them, in my experience. Measuring can tell if what you did made a difference, but it is a very fuzzy magnifying glass for finding what to fix.

There's a very easy technique for pinpointing single-thread performance problems quickly. A small but growing fraction of programmers know it. It's based on a very simple observation. While a program is doing something it doesn't really need to do, you can see what it is by just surprising it at random. If it's wasting enough time to be worth fixing, you won't have to surprise it very many times before you spot it. Then you can see, in precise detail, what the problem is. Here's more on the subject.