Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

When I previously asked what's responsible for slow software, a few answers I've received suggested it was a social and management problem:

This isn't a technical problem, it's a marketing and management problem.... Utimately, the product mangers are responsible to write the specs for what the user is supposed to get. Lots of things can go wrong: The product manager fails to put button response in the spec ... The QA folks do a mediocre job of testing against the spec ... if the product management and QA staff are all asleep at the wheel, we programmers can't make up for that. —Bob Murphy

People work on good-size apps. As they work, performance problems creep in, just like bugs. The difference is - bugs are "bad" - they cry out "find me, and fix me". Performance problems just sit there and get worse. Programmers often think "Well, my code wouldn't have a performance problem. Rather, management needs to buy me a newer/bigger/faster machine." The fact is, if developers periodically just hunt for performance problems (which is actually very easy) they could simply clean them out. —Mike Dunlavey

So, if this is a social problem, what social mechanisms can an organization put into place to avoid shipping slow software to its customers?

share|improve this question
2  
This reminded me of a recent blogpost by Jeff Atwood. –  rahmu Sep 28 '11 at 20:26
2  
Commenters: if you like the question, up-vote it. If you have an answer, please leave it as an answer, not a comment. Otherwise, please only leave a comment if you think the question clarified or improved, or if you have a link to a related resource. –  user8 Sep 28 '11 at 22:04
add comment

18 Answers

With correctly written and complete requirements, there is no such a thing as a distinction between bugs and poor performance. Because you specify the performance as a non-functional requirement, poor performance becomes a bug just like any other bug, and will be caught by QA and solved by developers before release.

Is there a social problem? I don't think so. The major issue is that requirements are incomplete. Working for years as freelancer, I never ever saw a non-functional requirement telling that a specific task must perform in maximum N seconds on average. If the manager/customer/stakeholder or whatsoever doesn't bother about performance asset, why I, as a developer, would bother about it, since people who must care about it don't care at all?

There is another factor which influences poor performance: the fact that developers work on expensive PCs which perform well. When you're working for years on a quad-core PC with 8 GB of RAM, a high end SSD, the latest OS, etc., it's very difficult to imagine how your application will run on Windows XP on a dual-core PC with 512 Mo of RAM and an old hard disk filled at 90% and not defragmented for years. Unfortunately, in some countries, the last case is the one we see for most consumers of an app. The larger the gap between developer PCs and consumer PCs, the more complicated it is for a developer to take care of performance of his app.

share|improve this answer
2  
Piggy-backing off of this comment, I think the best way to assure that at least the devs (and testers) are made well aware of these issues is to have them ALWAYS test on either older, lower end of the requirements hardware or Virtual Machines. As a dev, it's hard for me to say these words, but some times we don't work to fix a problem until we experience it for ourselves. Therefore, at least forcing your devs to test their applications on sub-par systems will force them to find efficient solutions because of the poor performance they directly experience. –  RLH Sep 28 '11 at 20:57
3  
I wouldn't suggest that on daily basis, since it will decrease the overall performance of QA department and depress the employees working on slow machines. IMHO, putting performance metrics as requirements is enough, if specified correctly (i.e. on what machine the automatic performance test must be done, with what load, how many times to take an average, etc.). –  MainMa Sep 28 '11 at 21:08
1  
I do work to a requirement that has a formal (non functional) performance requirement. It's real-time life critical software. The specs are "response within x on average, and y 95% of the time". The software is still slow compared to what it could be, but we know when to improve performance as it does become a defect, just like any other thing the system does incorrectly. Leaving developers to decide is a very very bad idea. Most devs, left to there own devices, over engineer in every way, and triply so with performance concerns. –  mattnz Sep 28 '11 at 22:37
3  
Another issue with performance is you cannot test performance on anything but a trivial system until final integration is complete, often long after the software developers have finished their work. Take a phone app - works fine on bare-bones shiny factory new phone, as a few downloaded apps and the memory gets full, and the software developer is blamed for writing crappy software...... –  mattnz Sep 28 '11 at 22:41
2  
The problem here is, you NEVER EVER get "correctly written and complete requirements". Not on an application of any size or complexity. –  CaffGeek Sep 29 '11 at 17:18
show 4 more comments

But it shouldn't take a letter from QA for a programmer to realize that a 3 second lag between keypress and response is unacceptable

Agree it shouldn't. It should take more than that: a proof that obtained lag is relevant for end users.

Given that you provided no context, it looks entirely possible that lag in dev/QA environment is caused by their local issues of slow disk / memory / network access. If that's the case, your QA and dev will be simply wasting their efforts fixing things that just don't matter to end users.

Relying on developers in performance testing is about as productive as rolling a dice to pick a piece of functionality to speed up. Oh and it's about as reliable as that - "developers generally have horrible intuition about where the performance problems in an application will actually be" (Brian Goetz).

  • I've been in a project where lame management once decided their bright marketing guys and smart programmers are good enough to handle performance concerns of the customers. What a great lesson it was. Rejected release, half year of efforts gone to trash bin, and company almost lost a strategic partner. In the end, they invited professionals (experts in benchmarking, statistics, UX, low level optimization, stuff like that) and professionals fixed that mess.

should all programmers do their own QA so they see such issues immediately?

Fact that it's doable doesn't mean it's the way to go. Rather opposite - in my experience this was one of the most reliable ways to lose on programmers productivity. Almost as good as endless meetings and even better than interviewing candidates.

  • As an ex-tester I once thought it shouldn't be a problem to combine development and QA activities. It looked like the difference between routine dev testing and systematic QA won't matter much. I thought that dev/QA separation is merely a tradition in software industry. Learned rather hard way that this ain't so. Separation turned out to be a matter of focus and productivity, and quite a serious one.

If there's a performance issue, just give me a benchmark and set target performance and I'll do my best to hit it. I'm not that good in performance testing but know a bit or two about optimization.

share|improve this answer
show 1 more comment

How much are you willing to pay for better software? How much will the market wait for better software? How little cruft will want adding to the next release?

It is a cut-throat market out there where many compromises are made. If it is truly crap then the market will (or should) fail the product. Maybe there are enough customers who can live with the status-quo?

share|improve this answer
add comment

I remember once in the mid-90s and I was spending some time trying to optimize something and a coworker told me, "This is running on pentiums, who cares?" .... that was an eye opener. Sadly, it was just the tip of the iceberg, I've heard that attitude throughout my career - albeit the "pentium" part has changed over time.

The only way to get the average developer to care is to get lack of performance to be viewed as a bug on the part of the customer. Depending on the application and audience this can either be an easy or a hard task (I've seen both). If the audience doesn't care about the poor performance, the developers never will (quickly, good, fast - choose two).

share|improve this answer
add comment

Make performance a requirement.

share|improve this answer
show 2 more comments

Can you come up with better examples where we can actually pin the blame on programmers? Apart from Eclipse, and one commenter has already pointed out that it's plugins that do it(my first install of each new Eclipse version runs like lightening, but when I add the other tools it starts to slow), your examples may not be programmer and code related but environment related.

The days of running a program on a computer in isolation and determining if it is 'fast' or 'slow' are gone. The other examples you give depend on their environment - the current network congestion, whether the back end servers are overloaded, badly configured network cards, a faulty cable, the number of other people using it in your vicinity, or hundreds of other variables. eg. our hosting provider charged extra for server gigabit connections but eventually we determined it all went through an ancient firewall device with 10Mb ports. These problems shift around and are hard to find.

Agreed, there are lots of things programmers can do (minimizing bandwidth, UI tricks that improve responsiveness and show progress to give the impression it is fast). But when you roll out to the real world there are all sorts of circumstances that you only learn by experience (and you watch your assumptions fall apart in front of you).

share|improve this answer
show 2 more comments

I agree that programmers should be taught better on maximizing performance, etc.

But I think the solution is not giving programmers an almost-dying hardware for daily use. How stressful it will be if your visual studio crashes twice a day, it took x seconds to build the stuff, y seconds to open the solution, z seconds to change windows. I don't think it was very cost-efficient for the company either since hardware is cheap and programmers time is expensive.

Since the next hand that will be handling the code is QA (testing) team, wouldnt it be better to teach them about the importance of performance, have a standard of whats acceptable performance standard, etc as an enterprise standard (well, in perfect world, the performance standard should be in spec but that doesnt happen very often)? (i.e regular enterprise ee-changing page/tab should happen instantly, "save update should happen in x second", if it's a critical app then...). The computer that the QA teams run in should be the typical user computer. (we dont wanna piss them off by giving them a 386, but dont give them a quad core with 8GB of ram for example). Wouldnt they vent to the programmers if they get pissed off enough with the performance?

I think client/project manager should force programmer/qa team/developer/rep of company team to do their presentation on the lowest typical hardware that they have. (the weakest computer in the company for example). If it's rewrite, they will need to gather data on how fast it is to do x process in the old software and compare it with how fast it is on the new software (it could be slower, since new software might involve extra business process but there should be an acceptable window).

share|improve this answer
add comment

Stop buying the stuff, and, comment on the poor performance on any online reviews you come across.

If the devices are still selling then the software is "good enough" not to invest more time and money in. If bad reviews about performance reduce sales significantly then the software is "not good enough" and needs fixing.

The only metrics that interests a producer of consumer goods are sales and profit per unit.

share|improve this answer
add comment

If performance is a requirement, test for it.

Otherwise Wally can write an infinite loop and leave early "It takes a while." He can claim.

Your software acceptance test should have a detailed acceptance test for various operation performance characteristics.

If you don't do this you're not engineering any performance into the product.

The performance ( like the resource consumption ) should get budgeted out to sub-systems. Then the sub-system acceptance tests can check them.

Then you can test early and often for performance. Even unit tests can then check it.

So now developers have it as an acceptance criterion, and can organise their approach to suit it.

Where I'm working now, the performance stress test is 2x bigger than any customer data set we know of. It regularly breaks a new version of the product. Good testing.

share|improve this answer
add comment

First hand experience is important. Give the developers a fast computer to compile and build on, but a really slow overloaded computer (as some percentage of users may actually have) to run their app(s) on. (This can be done in cloud/server based development environments by partitioning VMs or servers by function.) Give them a cell phone with a half dead battery, and require them to use only it for the days initial mobile app testing.

If the developers empathize with the customer with respect to performance and battery life, then they won't regard performance as some semi-bogus management spec to put at the bottom of the priority list. (As in: "hey, I thought it was premature optimization" until too late.)

share|improve this answer
add comment

Writing performant code is hard. It requires a solid grasp of concepts like threading, asynchronous event handling, caching, and asymptotic complexity. Judging by the groups of programmers I've worked with, around 20-40% of any given group doesn't understand those concepts well enough to incorporate performance considerations as a matter of course into their daily work.

However, those programmers are obviously still useful to the company, but they get assigned to tasks not considered performance critical, so you end up with a blu ray player that can play Netflix streams flawlessly without dropping any frames, but it takes 30-60 seconds to open the menu item that displays your queue.

Unless you're some hotshot software company that can afford to fire 20% of your staff and replace them with more experienced (and more expensive) developers, the only real way to fix it is developer training and filing bug reports. I don't know how it is at other companies, but here if we developers see a performance issue that we don't have time or business priority to fix, we're fully entitled to file our own bug report on it. It might take a couple of releases to work its way to the top of the backlog, but they usually do get addressed eventually.

share|improve this answer
show 2 more comments

Ignoring the developers that don't seem to care...

I think that often times the developers that work on the code don't have the tools to measure performance on a continuous basis.

e.g. If it is possible to measure the response time for your app (e.g. its a web based application, or queries a database, etc.) - Are you currently getting notifications (email, SMS, whatever) that indicate the "top 10" worst performing (or over a determined threshold) responses?

In many, many cases - developers are not getting this info from the "real-world" deployments and as a result its very easy to ignore the information that you don't see.

If however, every day/few hours you get an email that indicates that screen "x" takes 13 seconds to load and it is running the following SQL query SELECT TOP 1.... JOIN... OUTER JOIN... OUTER JOIN... CROSS JOIN... you'd better believe that a developer could (and hopefully would) be all over fixing it.

Thus although I'd like to believe that all programmers do take performance seriously I think that lack of visibility to the issue(s) is often the culprit.

Note: I think this is something that both the developers should be requesting access to (or even developing such a feature) and management should be providing/funding such tools.

share|improve this answer
add comment

I agree with others, that we should find ways to make developers care more about the problem, like making them test on slower hardware, and having performance goals. That's all fine, but really, when it comes to performance tuning --

People Gotta Know How - And They Don't

They may think they do, but just look through all the performance-related questions and answers on StackOverFlow and on this forum. It's painful how many show very little common sense about performance.

It's not something to just talk about, people need to learn by doing it. The only time they are in that mode is when they are taking a class, or learning new things from a book or blog.

So the only way I can think of to solve this problem is to get hold of the people who teach programming, and teach them how to do it.

Heaven knows, I've tried on these forums, as in -

Anyone can do it. They just need to actually do it.

share|improve this answer
add comment

The problem(?):

  • The customer (or end-user) does not complain about it (enough)
  • Thus the project(/product) manager does not consider it a requirement
  • Thus the developer does not get the time to fix it.

You have to start at the beginning, educate the customers. But if they buy the iPhone instead of a faster, less shiny phone, the developers are right to spend their time on looks instead of performance. The organization is not the problem.

Of course, some things can help anyway. Waiting for automated tests is annoying, so if you have automated tests the developers have constant feedback about performance issues, and they will be more likely to solve it (as a technical issue, not as a feature).

But you can't do everything. It's optimize or add features, and those who spend the money decide.

But some good news: I've noticed that SaaS/Cloud/Buzzword-applications help a lot here. When people choose between a few similar web-applications and get to test live instead of first creating artificial lists of 'required' features, they are more quickly influenced by responsiveness, and thus performance will get more attention.

share|improve this answer
show 1 more comment

I think the most general answer to this problem is also the most difficult to manage, which is that every programmer should be mindful of performance with anything they do. I realise also that's a bit of a cop out.

Depending on the size of the project, and corresponding team, I believe there can be a lot of value in having dedicated performance programmers.

As an example, I have worked on a team where the project team (including about 30 devs) had at least 2 people dedicated to performance optimisation. This particular app was also quite prone to performance issues as there were multitudes of interoperability components, not only over web services but also in terms of legacy layers with various data mapping and adapter components.

share|improve this answer
add comment

I think you would find that 99% of the time the problem is the scope creep. With the dvr for example. You would think this is easy but then TIVO or a competitor introduces a new feature that is well received. Next thing you know a new feature is on the plate. It may or may not becompatible with the existing product and we are not redoing the existing product that will take to long. So the feature gets jammed in and it sucks performance. Sure the data is there to get the infromation but if there was not thought to getting that information then there is a good chance it will not be easy to get. So now it has a complex process to build that information everytime you go near the program list.

share|improve this answer
add comment

I've said it before, but I'll say it again: I think one of the best ways to handle this is to simply have your developers work on (relatively) trailing edge hardware.

For your typical programmer, most official performance considerations are secondary to simply: "is it annoying when I try to run it?" If they find it annoying, they'll (at least try to) fix it. If they're happy with the way it runs, the very best you'll get is a half-hearted attempt at fixing. This also helps in finding problems early, before they become much more expensive to fix.

If it gets to the point that QA has to enforce rules the developers really don't believe in because they think performance is adequate, chances are pretty good that most of the "fixes" you get will get creative ways of getting around the rules, not real fixes that improve life for the end user.

At the same time, I should add that I've rarely seen this as a problem. If anything, I've seen developers spending far too much time trying to improve performance where I really didn't matter enough to bother (a sin of which I'm often guilty as well).

share|improve this answer
1  
It's easy to fall into the trap of obsessing over things that don't matter, but if a cellphone ships with a UI so slow that incoming calls go to voicemail before the "Answer" button responds, clearly someone failed to improve performance when it did matter. –  Crashworks Sep 28 '11 at 20:41
4  
In which case the company should buy me a decent sword, because I'm going to spend most of my time compiling. –  David Thornley Sep 28 '11 at 20:43
7  
This was suggested in a Slashdot comment (about something) years back. The response was: "developers should develop on fast machines and test on slow ones." –  user16764 Sep 28 '11 at 21:00
1  
@user16764: There is often too little attention paid to giving developers test environments that are different from their development environment. My wife had a very hard time getting both an admin account (to develop in) and a more limited account (for testing), and before that had constant problems with accidentally putting something in a maintenance fix that wouldn't run on an ordinary user account. –  David Thornley Sep 29 '11 at 13:55
show 4 more comments

Sadly, I find the biggest issue is you can't do everything. You have a ship date, and you know it's slow, but you NEED to get features X,Y,Z out to market.

In your mind, slow you can fix later, but the app at least works.

So, you worry about functionality and aesthetics (because users focus on aesthetics all to often). Next release you'll fix performance.

But the PM just gives you a list of Features, and no time to fix the performance.

And the vicious circle continues.

share|improve this answer
3  
It only gets fixed when the PM givs you a "feature" named "improved performance"! –  FrustratedWithFormsDesigner Sep 28 '11 at 20:39
4  
By the time the PM wants to improve performance, the only real way to do so is a rewrite :) –  Job Sep 28 '11 at 22:18
1  
+1 Here's where it really helps to have a competitor with good performance. When PMs see that, they get scared, and ask you to do something about it. –  Mike Dunlavey Sep 29 '11 at 17:02
1  
@Chad, the conditional probability is high, but the absolute one is low. I work on an app that started out as a 16-bit C program for Windows version "barely after I was born". Fast-forward that to today and many years and dozens of programmers later you have got a mix of C, C++, C#, VB.Net and lots of SQL vendors. Rewriting some key parts of the app in F# does not sound like a terrible idea now ... –  Job Sep 29 '11 at 17:59
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.