Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Is there a (quantitative) tool to measure performance of functions based on its input?

So far, the tools I used to measure performance of my code, tells me how much time I spent in functions (like Jetbrain Dottrace for .Net), but I'd like to have more information about the parameters passed to the function in order to know which parameters impact the most the performance.

Let's say that I have function like that:

int myFunction(int myParam1, int myParam 2) {
  // Do and return something based on the value of myParam1 and myParam2. 
  // The code is likely to use if, for, while, switch, etc....
}

If would like a tool that would allow me to tell me how much time is spent in myFunction based on the value of myParam1 and myParam2.

For example, the tool would give me a result looking like this:

For "myFunction" :
 value   |  value   | Number of | Average
myParam1 | myParam2 |   call    | time
---------|----------|-----------|--------
   1     |    5     |   500     | 301 ms
   2     |    5     |   250     | 1253 ms
   3     |    7     |   1268    | 538 ms
...

That would mean that myFunction has been call 500 times with myParam1=1 and myParam2=5, and that with those parameters, it took on average 301ms to return a value.

The idea behind that is to do some statistical optimization by organizing my code such that, the blocs of codes that are the most likely to be executed are tested before the one that are less likely to be executed.

To put it bluntly, if I know which values are used the most, I can reorganize the if/while/for etc.. structure of the function (and the whole program) to optimize it.

I'd like to find such tools for C++, Java or.Net.

Note: I am not looking for technical tips to optimize the code (like passing parameters as const, inlining functions, initializing the capacity of vectors and the like).

share|improve this question
    
I doubt you'll find such tool, therefore most likely you'll have to implement your own. –  BЈовић May 15 '12 at 13:11
    
I agree with @VJovic: such a tool would only make sense for functions which are called with a very small number of different input parameter value combinations, so I would not expect to find a general solution for this. –  Doc Brown May 15 '12 at 13:19
1  
@DocBrown Of course. For example, discriminating on the value of a pointer won't be really helpful. But if I can find such a tools, even if it has restrictions on the type of parameters it can handles (like working only for ValueType, Native type or on enums), I would gladly take a look at it. –  olchauvin May 15 '12 at 13:36
add comment

6 Answers

Your unit tests should cover this. Wrap the calls in a stopwatch and you're good to go.

I'll assume that you don't have any then, so here's one way to get quickly up to speed and get some.

Write a parameterized unit test around your function using Pex and then run that/those tests. Have ANTS running when you do - or just set a stopwatch before/after the call to the function.

Essentially, if I remember correctly, Pex uses a theorem prover to reach every branch and potential exception/boundary condition using the parameters to the function. Check the parameters it comes up with, critically think about some of your own and you should have what you're looking for.

share|improve this answer
    
Yes, I see your point and it's a good one. But the unit tests won't tell me that the users pressed 90% times the same button and had to wait on average 3 seconds because the if that handled this button was the last one checked. –  olchauvin May 15 '12 at 13:55
1  
@olchauvin: Parse your logs for how often users click buttons and extrapolate. –  Steve Evers May 15 '12 at 14:08
add comment

In C++, if this is for just a particular method with particular inputs, it's pretty easy to write a helper class that will time how long it takes to execute a particular scope. You grab the time in the constructor and in the destructor you calculate the current time and save it in a static data structure.

This is a technique I've used with embedded devices for which I had no profiler.

Beware: I haven't compiled/tested this:

typedef std::map<std::pair<int,int>, std::pair<int,int> > timeMap;

class TimeIt {
private:
    static timeMap times;
    clock_t timeCalled;
    int myParm1;
    int myParm2;

public:
    TimeIt(int parm1, int parm2) : timeCalled(clock()), myParm1(parm1), myParm2(parm2) {}

    ~TimeIt() {
        clock_t runtime = clock() - timeCalled;
        if(timeMap::iterator it = times.find(std::pair<int,int>(myParm1,myParm2))) {
            it->second->first++;
            it->second->second+=runtime;
        }
        else {
            times[std::pair<int,int>(myParm1,myParm2)] = std::pair<int,int>(1,runtime);
        }
    }

    static void report() {
        for(timeMap::iterator it=times.begin();it!=times.end();it++) {
             cout << it->first->first << " | ";
             cout << it->first->second << " | ";
             cout << it->second->first << " | ";
             cout << (it->second->second/it->second->first) << endl;
        }
    }
}

Then in your method, do:

int myFunction(int myParam1, int myParam 2) {

    TimeIt timeIt;

    // Do and return something based on the value of myParam1 and myParam2. 
    // The code is likely to use if, for, while, switch, etc.... 
}

And when you want the data:

TimeIt::report();

I'm sure you can do something similar in .NET or Java. The most important bit is to take care when you grab the clock() time. You don't want to do anything of significance in the destructor before grabbing the clock() time.

Obviously if you want a more general solution, you want a real profiler.

share|improve this answer
    
Thank you. At first I was thinking of something similar using a database to capture the inputs. But I'd like to avoid modifying the code if possible. –  olchauvin May 15 '12 at 17:04
add comment

.NET-only answer

To the best of my knowledge, there's no way you can retrieve the actual value of the parameters passed to a method.

Sure, you can get an array of ParameterInfo objects, through which you can get to know a lot about the parameters... but sadly you can't access their values. Granted, there are some ways to investigate the stack trace, but they are not guaranteed to be robust enough under all conditions.

So, no, it appears that such a tool can't even be developed for .NET.

I don't know about other platforms, though.

share|improve this answer
    
Doesn't the debugger know the values of parameters when a breakpoint causes execution to pause? Could the debugger be hacked to do what the OP wanted? –  FrustratedWithFormsDesigner May 15 '12 at 18:14
    
You are sure right that the debugger somehow knows how to retrieve the values, but who knows what it's doing under the covers. I can guess they have some kind of infrastructure that comes into being when we run code in debug mode. One thing is for sure: with managed code you can't do it in a robust way. One might be able to pull it off using the CLR's unmanaged profiling API, though. But I suspect that would be a tough endeavor as well. –  s.m May 15 '12 at 19:08
add comment

Although such a tool could be very helpful, I'm not aware of any that offer line-based hotspot analysis. Since profilers are tricky to hack and easy to get wrong, you are probably certainly better off temporarily splitting your code into different functions just so a normal profiler can tell you what you need to know.

share|improve this answer
add comment

I doubt you'll find such tool, therefore most likely you'll have to implement your own.

However, there are code profilers (for example callgrind to profile c and c++ code), which provides you with some value which represents a relative time spent in a function.

share|improve this answer
add comment

If you want to profile just one function, and you know the small number of input combinations you are interested in, you can add some kind of "switch mechanism" which will make the calls visible in any typical profiler:

int myFunction_internal(int myParam1, int myParam 2) {
  // the main body
}

int myFunction_p1_p5() {
    return myFunction_internal(1,5);
}

int myFunction_p2_p5() {
    return myFunction_internal(2,5);
}

int myFunction_other(int myParam1, int myParam2) {
    return myFunction_internal(myParam1,myParam2);
}

int myFunction(int myParam1, int myParam2) {
   if(myParam1==1 && myParam2==5)
        return myFunction_p1_p5();
   if(myParam1==2 && myParam2==5)
        return myFunction_p2_p5();
   return myFunction_other(myParam1,myParam2);
}
share|improve this answer
1  
That's a nice idea, but I don't really want to impact the code I want to profile. I am looking for something more general that would profile the whole program and not just one single function. –  olchauvin May 15 '12 at 17:00
    
@olchauvin: yes, I understand what you are looking for, but as something more general most probably does not exist at the current time, I wanted to show you a simple solution which may be applicable immediately for some real-world cases. –  Doc Brown May 16 '12 at 5:50
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.