Does using new techniques hurt productivity?

Question

It seems that as experience with the specific set of tools you have to work with grows, the incentive to try out new things weakens.

When I was new at this programming job, trying out new things, researching online, made me more productive, because I often found a way (or library) that made the task easier that the code framework already in place. So using something new -- to me as well as in the context of the given codebase -- made me more productive.

Now I noticed, that there are more and more instances where, for a given problem, I know that there probably is a better solution "out there", and finding it would -- presumably -- improve the code. However, given my now intimate knowledge of the code base, it is by far easier to use the suboptimal tools we have, and get a solution (including tests) running than find someting new and "better" and "improve" the codebase.

So there is this tension: "do it properly" vs. "get the job decently done".

Is this something that happens to a lot of developers? Is this a known specific problem? (Is it a real problem after all?) Does it actually have to do with increasing levels of experience?

Oh, and note: I still like my job and like to keep it. It's just that it seems the -- always interesting! -- research part get's smaller as I learn the code base and problem sets we face with our app.

Answer 1

It's often risky to try new things. We sometimes get into trouble because we tend to do two things:

1. Overestimate how useful the cool/new/snazzy thing is. We see some cool example, some code thrown online. Very cool we think. Very cool! In X I can do task Y ten times faster. Its clearly superior. We don't yet see all the "unknown unknowns". We haven't been tripped up by the problems that the salespeople of the new thing omit. We don't have enough expertise in the new thing to see the landmines waiting down the road.

2. Underestimate how useful the existing tools/framework/software/things are. We often weren't there when the current system was initially built. We don't appreciate the delicate tradeoffs that were made. Its easy to play Monday-morning quarterback on an existing system, but it works. Its probably got a lot of weirdness due to very specific tradeoffs between keeping it maintainable, working, and performing well. Yeah sure its weird. Maybe most importantly, the team is an expert on the current weirdness and known how to work around the weirdness. They know the landmines and the traps and the pitfalls to avoid. In fact its precisely because we see all the warts in the current way of doing things that we're even interested in playing with new things.

But failing to try new things and acting too conservative is probably even more dangerous. Sure we need to tread carefully, but if we don't figure out how to build the best mouse trap, our competitors a little more willing to try something new will come along and kick out butts! Acting to conservative and failing to evolve can result in inevitable doom especially in a competitive market.

So yes we need to balance maintaining and shipping the current thing with some level of playful/educational experimentation with new ways to solve problems keeping in mind that many of those new ways are dead ends while others may payoff. IMO This is a good reason many companies have 20% time to play with new things. They know a lot of times they don't work out, but many of the ideas that come out of 20% time become gangbusters. Without time to play and experiment you can easily stagnate as a company and really screw yourself over.

Answer 2

It happens all the time. I've said this in other posts, but more times than not, you're not in the business of developing elegant code, you're in the business of shipping a product. As such, you'll be hard pressed to find any manager who's willing to allocate n hours for you to fix something that isn't broken and really (at the end of the day) doesn't greatly enhance the end user experience. You'll be just as hard pressed to find a manager willing to allocate n hours to research (with no clear end goal) other than "there's probably something out there that's better" than what's being done.

Having said that, if there are bottlenecks in your application that you've discovered using profiling tools and such and you can clearly quantify the expected user experience enhancement that fixing them would bring, then you should (fairly easily) get time to do some R&D work to optimize them using techniques that you can find from various sources.

Answer 3

Here's some details:

1. There are deadlines: Programmer's should beforehand have all the details necessary available of the tools he's using. Reading some web site, finding some new cool thing, and then using it in production environment is a big no-no. You just don't have enough experience with the tool, and even more importantly, you just don't have the necessary time to start learning something new. If you want some cool new stuff, learn it half year or year before using it.
2. Make sure you know it properly before using it: Some nice new tool can be fun to use, but googling for solutions and then using them without properly learning them can be very bad. You just don't have enough experience with it. If you needed to google it to figure it out means that you just don't know it well enough to fix it when it breaks half the system.
3. Using the newest stuff is not doing it properly: the new stuff is not proven tech. Next year the tech is probably completely gone, you being the only one in the world using it any longer. Everyone else noticed it just does not work. It takes hard work to avoid the newest silver bullet, but it's worth it.
4. Focus on techniques that are your core knowledge: Every programmer knows only small part of the whole programming knowledge available in the world. The part where the programmer is comfortable enough using it is even smaller. The part which actually works is even smaller. Use only the knowledge which you have properly learned and you know that it works. This makes you faster programmer and results in better code.
5. Don't tweak it endlessly: Perfect code is a good goal, but this does not mean you write some crap first, and then endlessly tweak it to make it incrementally better. Write it perfectly the first time using the all the good knowledge you have. Trust yourself. Don't trust the world. Noone else knows exactly the same stuff you do. Their opinion doesn't matter. You're the programmer, you need to make it work. The world can't do it for you. Once it's done, Stop. Don't touch it until someone complains about it.
6. Spend time to learn new things: Everyone needs to continuously learn new things. Our future depends on it. Just refuse to use it! Learn new stuff, but don't use it until you're sure it's actually working. You'll get chance to use it next year, once it's proven tech. Or you just forgot it, like everyone else -- only the good stuff remains...
7. Do not lose the good stuff: Once you've successfully built big systems and fixed all the problems in them, and learned some nice techniques, the worst thing you can do is just dump that knowledge and go for something new. You already know how it works, what problems there are and how to solve those problems. Throwing it away is just waste of time.
8. Keep up with new tech: one of the new systems is going to be successful. It has something fundamentally better than anything else in the marketplace. You just don't know which one is the silver bullet. If you fail to find it in time, your knowledge will be outdated. The world can make you lose all the good stuff by removing access to the old systems.

Answer 4

I think some of it comes down to having exerience and more in-depth knowledge of how to successfully solve some problems.

When you are new, all problems are also new and you need to research how to do them. But as you have solved the same type of problem repeately the need to research goes down as you know a successful solution to that one.

Then you only tend to research the new problems or the ones where the old tried and true either no longer works (having been deprecated) or is causing a performance or failure problem. As you start to understand a complex system in more depth, you know you don't have the time realistically to use new techniques every time they come along and you have found through experience that a lot of the time the new technique doesn't live up to it's hype and creates more problems than it solved. So you become less inclined to use new tools and techiniques when you don't actually need them to solve the problem.

But less inclined should not mean you stop learning or never use a new technique, just that you are more judicious about when they are appropriate.

Answer 5

Yes, new things hurt productivity

Yes of course. Even for the best case scenario new things require additional time because they are unfamiliar. It can often cost alot more time.

No, new techniques can improve productivity

Any new technique which allows you to more easily express solution will improve your productivity. This can be as simple as moving from large if-elseif conditions to a dispatch table.

Answer 6

Yeah, I've had that happen. Usually you have to do risk-analysis on how much time it will cost to learn the new technique, and can you recover and use an older technique in case the new technique fails to live up to expectations. I prefer to learn new techniques when I can but when the pressure is on and I can't afford to spend time trying new things that might fail, I stick with tried-and-true methods.

In general, I find the best time to learn new techniques is at the begining of a new project. There's usually not too much pressure and if you find something new that works well, you can easily integrate it with the rest of the project, going forward. The worst time to try and learn new things is the last couple of frantic weeks before a big deployment.

Answer 7

Yes it can hurt productivity. My ex was asked to do some boring data processing job once, so she decided that it would be better to write a long program to deal with the data and then run it - which would fix the problem in seconds.

Took her a week to write it of course, but the problem did get solved in seconds after that.

I think the same applies to your question: yes, you can increase your productivity by learning new things, but you would still be better off applying your existing knowledge to the task and overall getting it done quicker. Who cares about finding and learning a new library, if you can write your own in less time.

Don't forget as well, often getting it decently done with existing tooling is a better solution than putting the new stuff in. Every time you add new, you increase the maintenance surface that's required, which in turn slows everyone else down (and can make your code quite messy - I think of the layers of 'new' tech that passed into legacy over time but are still in our code making things horrible. Looking back, it would have been better to just use the old C ways instead of adding all that COM and all that VB and all that .NET and now shovelling HTML into it too)