When designing a replacement application, how can I compare its performance to the original, if there are no existing statistics?

Question

I don't believe there are any performance statistics kept for the application I'm being asked to replace, and don't want to ask the individuals doing the work to manually record things like how long it takes them to perform a given task. However, the main objective of my project is to make it easier for the users to get their work done. If I don't know how difficult or time consuming it is for them now, how can I objectively show that the new system fulfills that objective?

Would adding a few high-level logging statements and analyzing the logs be straightforward enough? I'm not sure how much effort it would take to do that much, let alone setup anything more sophisticated. Or, are there existing tools out there for this scenario? It seems like something that would be relatively (if not quite) common.

Answer 1

and don't want to ask the individuals doing the work to manually record things like how long it takes them to perform a given task

Here's the problem.

You basically want to create meaningful metrics, without measuring the only thing that matters. Nearly all of your users won't care about how fast the code itself is unless it causes a noticeable impact on their activities.

A good question would be to whoever is requesting you of this project - sometimes the information you are looking for (task X takes 10 minutes) is used as justification for replacing existing software, especially internal software.

You might be able to do some profiling or basic logging. Depending on the type of rerwite, if you have tests you could compare test runtimes (won't be as effective if you are starting from scratch and/or have none).

If each user action has simple entry/exit points you could calculate the time for the entire action (maybe filling out a purchase request, whatever your app does) and log this. You don't necessarily need to worry as much about the individual parts of those actions.

You could also request whoever is requesting this rewrite to either give you a list of times or a list of tasks they want to compare. This is probably advisable even if you end up finding the metrics yourself. You want to make sure you are putting effort in fixing actual problems - with application rewriting you need to justify yourself continuously, because the "well it worked before why are you rewriting it?" question will always be present (and legitimate).

Answer 2

I've been faced with this, and the answer is:

For someone experienced in performance tuning, the new code can be tuned so almost no code can go faster. Here's an example.

The reason is, there is a minimum length of time the task can take, and it's greater than one cycle. There are many, many programs that can do the task, and one or more of them take less time than all the others, so they are optimal, by definition.

The old code might possibly be just as fast or even faster, but if it hasn't been tuned by someone with that experience, it's not likely.

So if there are no measurements of the old code, all you can do is apply the expertise to make sure the new code is as nearly optimal as possible, so it is almost certain to be faster than the old.

---

A simpler way to put it:

Software is like a wet towel. Most of the time it has not been wrung out.

You don't know how wet the old one was, but if you really wring out the new one, you can be pretty sure it is no wetter than the old one was, and possibly quite a lot drier.