Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

So you've made some code changes that should hopefully speed up some part of an application. But there is just one problem - you don't know how it will perform in live.

Different networks, different machines, different memory, different configuration, different times of day, different users, different server loads etc etc etc

So...

(a) Do you ask them if that bit of the application has got faster?

(b) Do you ask them to monitor that part of the application?

Or...

(c) Do you just ask them to see if they notice any difference in the application in general? After all, you can't be 100% sure you haven't affected something else.

Each of these approaches would seem to have problems (I know this as I've tried them all):

(a) You're asking a leading question. Before such a fix goes life, there is an emperor's new clothes scenario. The developer believes the code should be faster so there is some kind of crusade to get everyone believing it before it goes live. So...it goes live. Some people say it is faster, some slightly slower and others about the same. What then?

(b) Again: Some people say it is faster, some slightly slower and others about the same. What then?

(c) You're no longer asking a leading question, but you start getting hit with a whole torrent of issues that people have known about for months (maybe years) but you've now opened Pandora's box. How can you be sure which if these (if any) are new issues?

Or maybe I should be asking different questions entirely. Any thoughts?

EDIT:

No doubt there are processes leading up to this point that could potentially be improved. But the nub of the question is this:

The change is tested and has been put live. You have a myriad of stats to prove the changes should improve performance. Your boss has given the stats a once over but now wants to make sure the users are happy before signing off the release. What question(s) should you be asking the users?

share|improve this question
1  
About the code performance: profiling on at least the most common hard- and software setups. About Pandora's box: issue tracker. –  thorsten müller May 17 '13 at 9:16
5  
I don't see the issue you are facing. Either a speed-up was requested by at least one customer. Then you should ask them to measure whatever task they are doing before and after. Or it is strictly for good practice; then you can write the pre-change and post-change tests and proflings yourself. Which is it? –  Kilian Foth May 17 '13 at 9:32
1  
If your performance enhancements are that fragile towards setups, maybe you have a case of premature optimization. There is not really much sense in asking people, especially if the answers are that ambiguous that they can't say if it changed and on one and the same setup some people say it's faster and others it's slower. btw: for your point c) unit tests may help to ensure that changes don't affect other parts of the application. –  thorsten müller May 17 '13 at 10:15
1  
If only it were that simple. Take SQL Server. As any DBA/DB developer worth their salt will tell you: Query cost is a product of CPU & IO. You might notice the cost goes down when you tune the SQL but then you run it and sometimes it will actually take longer. Also understand that this assumes the same query plan will always be executed. In reality however, it could differ as the spread of data is different at each customer site. The point I'm making is that if I could rely 100% on what the profilers are telling me, I wouldn't be asking this question at all. –  Robbie Dee May 17 '13 at 12:45
7  
You cannot rely upon user feedback for this sort of thing. If you do not have a way to test/evaluate/profile your code in a controlled environment to collect hard data, then you may need to build instrumentation & reporting into the application itself to send statistics back to a central point for analysis (similar to how MS offers to collect anonymous "user experience" information in some of their products). –  alroc May 17 '13 at 16:06
show 11 more comments

closed as not a real question by Jim G., Yusubov, MichaelT, gnat, Jalayn May 21 '13 at 12:23

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

5 Answers

up vote 4 down vote accepted

What question(s) should you be asking the users?

Here is the next version of our program. Please report any unexpected behavior.

Unless your users are using stopwatches, or your program has some very long-running processes, users are unlikely to be able to provide anything but subjective feedback. And for optimization I'd be more worried about behavior not caught by your automated tests than an uneven speed increase.

share|improve this answer
    
A few smoke tests aside, this is our normal approach but management do like to tick those boxes... –  Robbie Dee May 17 '13 at 15:15
    
Thank you for being the only one to actually understand (and more importantly) answer the question. –  Robbie Dee May 20 '13 at 8:02
add comment

You should concern yourself with data rather than subjectivity. Thus you should be monitoring/measuring/profiling to determine improvements. That monitoring needs to be in both production and test (and dev) environments.

Ideally, you shouldn't be optimizing anything until after you see the results of performance. Otherwise you risk a "premature optimization" which doesn't improve anything meaningful and perhaps was a waste of time and money. It is very easy to see a piece of code and think, "This is terrible, it will never perform well." only to find out later that it is only executed 1% of the time and the real bottleneck is somewhere else.

share|improve this answer
    
But, perception is everything here. After all, an application doesn't load faster because there is a splash screen - it is just a visual distraction to make the user believe otherwise. In my original question, I noted the scenario where it was believed that there was a speed improvement but also a process of propaganda to make this enhancement seem more evident to the user. Fully with you on your 2nd point (+1), tuning should be very much secondary to getting the thing working. –  Robbie Dee May 17 '13 at 21:36
4  
This answers it. If you can't assess the performance through load testing or profiling then you can't optimize it. Not with any certainty. –  Rig May 17 '13 at 23:37
    
@Rig +1 - when I did Testing 101 all those years ago, this was (I remember) a very difficult lesson. For complex systems, you can't hope to test every case so have to be satisfied with a representative sample... –  Robbie Dee May 20 '13 at 8:10
    
Good point. I didn't account for the user perception aspect of it. A splash screen is a great example. But even so, I think that measuring is key. Better to make the experience demonstrably faster in some way rather than trying to convince your users to "trust me, it's faster now." Someone might benchmark it and complain otherwise. :) –  Allan May 22 '13 at 15:14
add comment

Add a configuration switch to choose between old and new coding and put it into the production system. Run it one week with each setting, measure, and compare.

share|improve this answer
    
This is something I have resorted to recently for exactly this reason. The feedback we're getting is mixed and the data can be skewed this way and that. Often much better to have such a switch and let the mob decide... –  Robbie Dee May 17 '13 at 21:38
add comment

Can you not measure performance in a development environment, before and after fixes? If it's 20% faster, it's usually safe to assume it'll be 20% faster in production too.

You should usually be able to find out quite easily the main bottlenecks in your infrastructure from past experience (or experience of other developers) and take that into consideration when profiling.

share|improve this answer
    
We can of course do whatever we like in development/test but for the reasons I've already outlined, this doesn't always scale outside the development office/lab. –  Robbie Dee May 17 '13 at 21:43
add comment

Your problem here is that you are involving users as testers, don't do this.

Your question is quite valid - how does an app perform once it gets to the "real world", with real world CPUs, user load, data volumes... but once you ask this question, you need to get some objective performance figures and users are never going to be much use for those.

So you need to emulate the user load, in a QA or performance test environment. Get a representation of the end-user environment, get some automated test tools that can hammer your system as if 10, or 10,000 users were using it, populate the data sets with masses of dummy data. Then you can run your tests and get some meaningful data. Then you can also re-run your tests with a new version of your system and see what differences there are.

share|improve this answer
    
This takes place as part of pilot where the software is released to a limited number of sites. Sometimes, this is as UAT whilst sometimes this is in addition to UAT. Obviously it would be a bad idea for us to just test it and throw it over the wall to thousands of users and devices. –  Robbie Dee May 21 '13 at 11:20
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.