Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Read first: For a definition of the two terms ("profiling" and "benchmarking") and the need for distinguishing them, please read this answer to a previous question.

I have to admit that until I saw Winston Ewert's answer, I have never thought of the need to distinguish the two techniques. I simply think that "profiling" can be applied at different "scale levels" of software, and that when it is applied on the higher level, the profiling code in the lower levels ought to be turned off in order to lower the aggregate overhead.

After I reflected on the answer, it might have explained why I fell prey to micro-optimization in my previous project.

In an effort to optimize during that project, I implemented a low-overhead profiler (inserted into the source code) which is good at generating accurate profiling results at the millisecond level. I then spent all days tinkering with it, and optimized a lot of code based on the profiler's result. In the end, I was successful in reducing the computation part of the project from several seconds to less than a fraction of a second.

The next thing I learned, to my horror: when the optimized module was used in a larger project, I/O and data conversion completely dominated the module's computation time. The non-computation part is in the range of 1-2 seconds, making my optimization efforts moot.

To this date, I still haven't got a chance to do a true "benchmarking", though I am going to give it a try very soon.

Given that "Did you do profiling?" has become the cliche on both StackOverflow and Programmers.SE, is there a danger that my kind of ignorance is actually prevalent among fellow developers? Does this ignorance lead to micro-optimizations all over the places?

share|improve this question

4 Answers 4

From what I've seen the "did you profile?" question always comes after "why does this run so slow?" so the "benchmarking" has been done and the result was "too slow" and now we're trying to figure out why it's running so slowly so we go and "profile" it.

Real life is usually more complicated. How fast your software is depends on the architectural decisions you make, algorithms you choose, whether or not you've correctly identified and dealt with various bottlenecks and system constraints. Getting stuck optimizing a system that isn't designed for performance is an easy trap to fall into and can suck away huge amounts of time for little reward. On the other hand, not every software has high performance as a requirement.

Profiling and optimizing before you benchmark, i.e. before you know whether or not the performance is adequate is truly falling into the premature optimization scenario.

I like this quote from Wikipedia:

“The First Rule of Program Optimization: Don't do it. The Second Rule of Program Optimization (for experts only!): Don't do it yet.” - Michael A. Jackson

share|improve this answer
2  
+1 for the Quote. It should be printed with a label printer and stuck across the top to every software developers monitor. –  mattnz May 9 '11 at 1:35

He who does not honor the small is not worthy of the large.

Even though you may not make the specific feature significantly faster, you may safe some cycles which might help the system as a whole. Taking small bits of time off of many links in a chain may add up to being more then just combating the weakest link.

Of course you need to be careful about putting too much time in it. Small tweaks do stack up though. Small tweaks are also best done while your code is fresh. They won't typically show up in a profiler since they are small and are many all over.

First make it work then make it fast. Sometimes it's just fast enough, even though there is plenty of opportunity for improvement.

Sometimes it's not "premature" if it's hard to do later.

share|improve this answer
    
I agree that making a lot of small optimizations on "computational" code can add up. But from my experience, I don't think that optimizations can "add up" anymore when I/O and OS effects are combined. –  rwong May 8 '11 at 23:53
1  
Good programmers write optimal (not optimum) code as a matter of routine. They have the scars from the resources being silently and unendingly spent all over the place. –  Apalala May 9 '11 at 0:30
    
have to disagree. Until you know for certain code is a problem, leave it. What happens if you half the time a piece of code takes to run? Does your program do twice the work in the same time? What if that code only runs 10% of the time. –  mattnz May 9 '11 at 1:37
    
This comes back to "is algorithm selection optimisation?" Small tweaks vs not chaining your program to a rock. To people above a certain skill level it's just good design. But to many people optimisation is going back and replacing (eg) bubble sort with quick sort. But those are the decisions where the application performance is determined. Data in memory vs filesystem overwhelms saving a few cycles getting data out of the filesystem every time. –  Мסž May 9 '11 at 2:09
    
It is usually not worth sacrificing anything else for performance for single-user applications. Their performance is almost always dominated by other things, such as user input and network connections. It is pointless to micro-optimize uniformly throughout the program.. –  David Thornley May 9 '11 at 14:24

Profiling must be against realistic scenarios (call that benchmarking if you like). (Is that a "Dugh"?)

O(n²) solutions severely beat O(n log(n)) ones by a large margin given sufficiently small datasets. That is well known.

The programs that most developers write first-hand do not scale to even one order of magnitude above. It is a project manager's responsibility to make sure that everything is tested against close-to-real-life test datasets and scenarios, and to executions/second per module when need be.

Risk management is not about being pessimist. It's about considering bad and fatal scenarios in the design and the procedures.

share|improve this answer
    
The project manager probably only tells the developer that "You should measure this, this, and that." and leaves the details to the developers. How does the project manager find out whether the developer is doing it right or wrong, if it is up to the developer to determine whether the test scenarios and/or the test mechanism is realistic or not? (Is the average project manager usually more knowledgeable about the proper ways of performance testing than the average developer?) –  rwong May 9 '11 at 1:00
    
@rwong: at some point a user sits there and goes "this is slow". Ideally the manager makes it clear to the developer what the user will be looking for. –  Мסž May 9 '11 at 2:10
    
The project manager should gather the user requirements. –  quant_dev May 9 '11 at 5:47

Given that "Did you do profiling?" has become the cliche on both StackOverflow and Programmers.SE, is there a danger that my kind of ignorance is actually prevalent among fellow developers? Does this ignorance lead to micro-optimizations all over the places?

I think so, given the kinds of questions and answers that travel about these sites, and the existence of profiling myths.

It is common to hear people putting timing code into their routines, because they are unhappy or puzzled with what profilers tell them. It's also extremely common to hear people doing micro-optimization whether or not they have tried profiling.

I think part of the problem is the word "profiling" itself. It is often conflated with "measuring", when finding performance problems is not at all the same as measuring them, in my experience. Measuring can tell if what you did made a difference, but it is a very fuzzy magnifying glass for finding what to fix.

There's a very easy technique for pinpointing single-thread performance problems quickly. A small but growing fraction of programmers know it. It's based on a very simple observation. While a program is doing something it doesn't really need to do, you can see what it is by just surprising it at random. If it's wasting enough time to be worth fixing, you won't have to surprise it very many times before you spot it. Then you can see, in precise detail, what the problem is. Here's more on the subject.

share|improve this answer
    
This works in more than just single threaded scenarios. Take a java container and execute a thread dump. If 20 of 23 threads are waiting on a lock chances are that lock is your problem. –  nsfyn55 May 13 at 16:38
    
@nsfyn55: You're quite right. One case where it did not point to the problem was in a manufacturing simulation where there was an asynchronous protocol between processes, and delays were caused by fast actions being prioritized behind slow ones like DB update. For that, I used a laborious logging method, which worked. –  Mike Dunlavey May 13 at 19:11

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.