Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Recently I've been increasingly plagued by what I would have to describe as one of my most frustrating and morale-killing experiences in this profession: Having to sit on a release that has been tested, re-tested, staged, and for all intents and purposes is ready to ship/deploy.

As an all-around solutions guy and not just a hardcore coder, I do understand and have even advocated the need for proper change control. But lately, the tenuous balance between covering our bases and shipping on time has gone all lopsided, and I've had little to no success in restoring it to something sane.

I'm looking for compelling arguments to help convince risk-averse management that:

  1. The dev team should (or must) be able to set its own release schedule - within reason of course (1-3 months should be conservative enough for all but the biggest Fortune 500 companies);

  2. Software releases are important milestones and should not be treated cavalierly; in other words, unnecessary delays/stoppages are highly disruptive and should be considered only as a last resort to some critical business issue; and

  3. External (non-dev/non-IT) entities who want (or demand) to be involved as stakeholders have a responsibility to cooperate with the dev team in order to meet the release schedule, especially in the last week or so before the planned ship date (i.e. user testing/staging).

The above are assertions that ring true for me based on experience, but it looks like I'm now in the position of having to prove it - so I'm asking for something a little meatier here, if such a thing exists.

Can anyone who has had to "sell" the idea of a fixed (or maybe semi-flexible) release cycle to management give some pointers on what arguments/strategies are effective or persuasive and what is not? Aside from the obvious schedule contention and sunk costs, is there any hard data/evidence that would be useful in making the case that shipping is actually important, even in a "corporate" setting?

Alternatively, I'm open to hearing constructive arguments about why schedule flexibility (even over a period of weeks/months) is more important than shipping on schedule; it's hard for me to believe right now but maybe they know something I don't.

Note we have staged releases, and this went through every stage except production. Issues are tracked using a commercial bug tracker and every issue - 100% of them - that was assigned to this release was closed out. I realize it's difficult to believe and that's really precisely the point - it makes no sense that a 100%, feature-complete, fully-tested, approved-by-stakeholders release would be delayed by management for unexplained reasons, but that's what happened, that's what's been happening, that's the problem to be solved.

share|improve this question
    
Good luck. As developers in a previous company we fought this battle from the other way to the middle. We had deployments at a whim because the business needed bug fixes and enhancements "immediately". I wish you the best in your endeavor. –  AndHeCodedIt Mar 10 '12 at 17:51
    
Has your management ever studied the opportunity cost of waiting on a release? –  chrisaycock Mar 10 '12 at 18:11
    
@chrisaycock: I doubt that they've studied or really understand opportunity cost from any angle. However, just saying the phrase "opportunity cost" isn't going to sway anyone; I need something specific to point to. –  Aaronaught Mar 10 '12 at 19:39
    
Why can't you start working on the next version of your software while waiting for the current one to be released? –  krlmlr Mar 11 '12 at 1:52
    
@user946850: I can, in theory. That doesn't change anything that's been said in this question. It doesn't negate the demoralizing effect of having one's hands tied and having worked on something that won't get released, or the massively disruptive effect of dealing with a mid-cycle release. –  Aaronaught Mar 11 '12 at 3:03
show 2 more comments

8 Answers

Go away from that company, they don't understand software. But seriously, software is what makes the system work, imagine if you were making cars, are car factories slow? do they wait on car releases? Are they incremental and bureaucratic? No, they try to do things the fastest and cleanest way possible. You have a management problem and you should work on it as a management conflict, try to speak them into their terms. Ask what risk means to them, then try to minimize it. The feelings of your devs are also important, since they must cope with the decisiones that will be made by your intervention. Will they be happy doing all nighters to ship on time? What would the prizes/penalizations be? Etc. Now i will give you a practical approach to this problem, a bit extreme, but ask for an audit. That will give you tools to talk to management about the problems you see, but from an external standpoint, and will also improve your image with them.

share|improve this answer
add comment

First thing to consider is to make sure your releases aren't high risk.

Ideally, your change requests should have section called like Risk Mitigation, containing instructions on how to rollback to prior version if things go wrong. Risk-wise, no amount of prior testing can substitute a simple option to rollback the change.

  • In risk-averse environment, I can not imagine a way to make irreversible changes painless.
    If many/most of your releases fall into this category, you may want to reconsider your architecture.

The risk of NOT releasing is to be exposed, if you want dev team schedule to be treated seriously.

If your releases deliver fixes for production / support issues and outages, this is fairly easy to do by just listing these and pointing out that production is at risk of repeating these unless it's upgraded.

A really effective measure available to dev team to fight against release slippages it to make sure that risk of not releasing increases over time. This can be achieved with internal releases running at fixed schedule, providing "cumulative" list of solutions to production issues.

  • Simply speaking, guys blocking your release XX need to be aware not only that they incur the risk of having N kinds of production issues to remain unresolved, but also that week (or month) later, they will have release XX+1 in their approval queue, blocking which will incur the risk of N+M kinds of production issues remain unresolved - and that this will also be the case with XX+2 and further "internal" releases supplied by dev team.

The way described above essentially makes tighter and more explicit connection between your application updates and production issues.

Because of that, you may expect more involvement into production issues escalations. You may use these as opportunity to promote your vision of stricter scheduled updates. You may even trigger some escalations yourself to speed up the process of adopting your desired approach.

  • "...recommend to consider upgrading application to newer version (XX or later). The one that is currently deployed to your environment (XX-2) is severely outdated - two major releases behind the current codebase. As a result, details for such simple failures are deeply hidden in logs and undecipherable garbage is reported by application to clients instead of clear failure descriptions - causing users and support to waste time investigating issues that are really quite simple"
     
    Above is a comment I added some time ago in support issue tracker for the ticket related to particular production incident. Soon after that, "stakeholders" contacted dev team asking how to keep track about our releases and how they can help in deploying to their environment. Oh and they also quickly upgraded from XX-2 version, too. :)

When production escalation happens, you better be prepared to present your vision quickly and clearly.

One thing you'll find useful is to pick and track who's responsible for particular release slippage. Basically, you need to be able to quickly and clearly answer to question like "who holds responsibility for the fact that planned release did not happen?" If you're not ready, they may choose the path of least resistance and put the blame on you. As pointed out in comments to another answer, "You could get yourself in hot water that your management is trying to protect you from."

The last but not the least,

it'll take time

(you probably already know that but it looks worth stating explicitly)

What you look for can be described as attitude change, from "it is risky to release" to "it is risky NOT to release". Attitude changes rarely happen overnight, patience might be needed to get the shift completed.

share|improve this answer
add comment

There is a big question mark hanging over your question, and that is why your company doesn't wish to release the software. It's not always just a simple case of risk aversion. Often there are other causes underlying which don't often get passed down from the lofty managerial heights. So my question to you is, "Have you sat down with your boss and asked him why the product release is being held up?".

There is a big difference between managing risk, and avoiding it. There may be legal or marketing reasons for delaying a product release. It could be trust issues between management and the development team. The product may not suit the needs of the customer, even if it is exactly what the customer asked for months ago. It could even be a case of someone in management seriously covering-ass while trying to shift the blame for delay elsewhere, or even a delay by a third party that is required to complete something before your product ships. I could come up with dozens of scenarios. Quite often the developer assumes risk aversion without understanding the other side of the equation that hasn't been made apparent. On the flip side, it may simply be complacency, conflict between individuals within a company, or even a fear that the product will fail in the market for some reason which results in seemingly ridiculous delays.

In all cases, it's important to have more information about the reasons for delays in product release, and to use that information to build a case for releasing your product as soon as you can. Yes, it can be very frustrating, but there are other ways to deal with these situations without risking either confrontation with your bosses, or a burnout. In similar situations, I myself have gathered information, documented whatever progress I've made, and then if worse came to worse, simply shelved the project and moved onto something else. Where I have been able to get a project back on track for release has usually been the result of making a sound business case based on the information I have received as feedback to the questions that I have raised. Where I have been unable to convince the management that the product should be shipped, I have documented the reluctance to ship as simply being a case of project finished, and awaiting release approval by management. I then make sure that my manager signs my document as accepted and understood by management so that my own butt is covered if they should attempt to blame me later for a non-release.

You always need to dot your I's and cross T's to avoid shipping delays being later blamed on you (no matter how unfairly), and also it's important to avoid becoming too emotionally bound to such issues unless you like the idea of a well earned ulcer. By looking at your own situation with a certain professional detachment, you may find a solution presents itself because you have effectively placed yourself at a greater "distance" as it were from the problem. If you can't make your case, you may need to let it slide for a while, but in order to make your case in the first place, you need to seem professionally detached, and have arguments based on information that is intimately connected to your company and its inner workings.

Something else that I've just thought about which might be of some help to you, is to perhaps read Lean Software Development, and see if there is anything valuable in there that might relate to your company's situation, particularly in the first few chapters. I think it was chapter 4 that discusses the issue of delivering as quickly as you can and has something relating to delay costs.

share|improve this answer
1  
There's no question mark. I didn't mention any of these scenarios because there aren't any that are pertinent. Of course what you're saying here would be reasonable given no context, but the question was written with the assumption that people would believe me when I say that I've exhausted all avenues of investigation. –  Aaronaught Mar 13 '12 at 22:01
    
@Aaronaught In the context of my answer, it's not a question of not believing you. Often when we think we've done everything there is yet another option to try, or if not there is value in having others voice something in the hope that it might be directly relevant to you, even if it states the obvious. It may be that someone else finds themselves in a similar situation, and this answer may apply more closely to them (that's also what this site is for). Regardless my last paragraph remains relevant, although I'll offer a slight edit in addition. –  S.Robins Mar 14 '12 at 1:12
add comment

I can't imagine that putting the Dev team in charge of the business the way you describe is a "good thing" It might mask the real problem, but unless the Dev team is in the right position to weigh the inputs from sales, marketing, etc. etc. you risk releasing for the wrong (and potentially bad) reasons.

If I understand your problem- you have completed the project and have a deliverable you can't show to anyone outside of your company. (I hope this summarizes it, I read the whole thread and I'm having trouble putting my finger on it)

Have you tried:

  1. Ask management for a customer or set of customers to act as stakeholders during development? Usually you can find one customer to take intermediate releases and give feedback. They can be great advocates when it is time to release. Even a "limited release" to one customer might be enough to help sway management
  2. Building a release plan including point releases. That is, you can release 3.4 today because the 3.4.1 release is planned for today + 1 month. This along with the good idea you already mentioned that a release cadence helps this message.
  3. Get support, sales, or marketing on your side. Build in fixes that would solve the top 3 support requests and you can get an ally from another department. If you can't get a customer stakeholder as in #1, try it with a few sales people. Offer to be available for sales meetings and bring along the product. When you are on the road, show the sales guy you are with the product (in whatever state it is in) get them pulling for you.

To answer your other question about "why would management sit on a release?" Companies do this all the time in non SW fields and I don't think it is unprecedented. For example, you have a new release that is ready to go, all tested and done. But the old version hasn't been displaced by the competition yet. Once the competition releases a competing product you the old version, management releases the version waiting on the shelf and disrupts the market, sets back the competition and maintains sales and the reputation as a market leader. Releasing too soon tips their hand to competitors who might have a better idea of your roadmap and try to beat you to the endgame.

With all that said... Hanlon's Razor is probably the right answer :-)

share|improve this answer
add comment

I'd say the easiest / most effective way to sell a release is to down tools for a while when its ready to go. Do something else, start a new project, work on bugs for the existing project. Do not do any more to this one until its sent out the door - after all, why bother doing it at all if it doesn't go when its ready.

but in the real world, the actual release is someone else's problem, you should be shipping it to them and then treating that as a release. Once its out of your door, its shipped. If they just sit on it, it's not your problem (in fact, its great, no bugs get reported on it! w00t! Bonuses for a bug-free release all round ;) )

So you need a change control system in place anyway, and a release manager. Then it's all easy (except you do need to manage the change control, so source control plus deployment builds are very necessary. It requires organisation, but if you're organised, its easy).

share|improve this answer
    
I was worried by the first part of your answer [down tools] but when I read the remainder, I think this is a good way to handle it. –  jasonk Mar 14 '12 at 2:11
add comment

First, watch this video:

http://the99percent.com/videos/5822/Seth-Godin-Quieting-the-Lizard-Brain

Executive Summary:

The way you ship on time and on budget is this: when you run out of time or run out of money, you ship. That's it.

The reason most projects don't ship on time is because someone comes along with "just one more feature or fix" to insert into the release, right before you're ready to ship, at a time in the development cycle when your resources are most scarce, and now you have to scramble. This is called thrashing, and it happens because as long as you don't ship, you don't have to worry about people telling you your product sucks.

The way you cure thrashing is by forcing people to do it at the beginning of the product development cycle, not at the end.

Making product changes in the weeks just prior to the release date is very disruptive, for obvious reasons. This is true whether the change is a new feature, a change to the software development process, or an attempt to fix a long-standing bug that's not in the development schedule.

When someone comes to me for a fix or change late in the development cycle, I always ask them this: "What do you want me to not get done so that I can get your thing done?"

If you need a money motivator, it is this: studies prove that the later you wait in the software life cycle to implement a feature or fix, the more expensive it is. Exponentially. It's not hard to prove this.

share|improve this answer
1  
This is all good advice but I don't think it's really pertinent; I definitely did not say and didn't mean to imply that last-minute scope changes are causing delays. What I'm talking about are bureaucratic obstacles thrown up by people who are just resistant any change at all to the production environment, especially anything client-facing. –  Aaronaught Mar 10 '12 at 19:36
    
Mind you, re-reading my question, I guess I didn't do very much to clarify that there weren't scope changes, so I can't really fault this answer either. –  Aaronaught Mar 10 '12 at 19:40
add comment

Joel Spolsky says that a software development team is a scheme for converting capital (money) into working software. Honestly, from your question, it sounds like your team is good at that: you're getting decent software for reasonable amounts of money invested.

Now, of course the next step is to put the software into service. Until your company chooses to do that, there's no way for them to harvest a return on their capital investment. Software sitting on the shelf ready to deploy is kind of like inventory in the stockroom: the costs are sunk, the company's capital money is already spent. There's also an opportunity cost: your group chose to do this project rather than something else.

What's more, the value of newly developed software sitting on the shelf is kind of like the value of a crate of cheese sitting in a restaurant's fridge: It slowly rots. Its value declines because your competitors catch up. It also declines because it gets harder and riskier to put new software into service as its developers move on to other things. After a couple of years on the shelf, it may actually be worthless. In that case, your company has chosen to discard the capital they invested in developing it. It's possible that the owners of the company -- the shareholders -- will be displeased about this.

There's one thing you, as a developer, must figure out. Is your software really, truly, ready to deploy at the end of the release cycle you're proposing? Is it ready to go, or is there a whole bunch more work to do and money to spend as your company puts it into production? Does your company own the servers and software licenses you need to roll out your software? Does your work product include a deployment plan? Have the end-users been trained? Has the production data been converted? If your new software involves a change to the way your company does business, is the company ready to make that change? Have you worked out a scheme for running things in parallel, to make sure the new stuff works as well as the old stuff?

All those rollout costs can easily dwarf the cost of just developing the software. A wise project management team does its best to plan, budget, and schedule all that stuff. Have you done that work? Have you made it clear to management? If not, it's possible they're wise to go slow on your rollouts.

It's possible that a modern agile software rollout schedule is out of sync with the pace of change the rest of your company expects. Your end-users -- your stakeholders -- simply may not be able to absorb change at the rate your team can produce it.

It's also possible your management team is dysfunctional. Has your team developed something the rest of the company doesn't want? Does your new stuff threaten the heck out of your sales or production team? If so, some executive may be torpedoing it by saying it's too risky to roll out. There's nothing you can do about that unless you're the CEO.

You asked for compelling arguments for timely software rollouts. There's one really compelling argument: return on investment. The company chose to invest in this software, and now they're choosing to waste the investment as your software slowly rots on the shelf.

A secondary argument for timely rollouts is the morale of your team. Hurry-up-and-wait is not a good way to motivate creative developers to do good work. You may lose your team if they don't get the satisfaction of seeing their work go into service.

share|improve this answer
1  
Great answer. The funny thing, in my case, is that even the rollout costs have basically been spent. We just need to flip the switch! Except that, metaphorically speaking, we need a union switch-flipper to actually flip the switch and there aren't any available for the next 7 weeks. –  Aaronaught Mar 10 '12 at 23:05
1  
Wow! This problem is known to medical science as managerial craniorectal inversion. Do you need to be working someplace else? –  Ollie Jones Mar 11 '12 at 20:52
add comment

You have clearly described the problems you are facing, however what I am unclear on why the managers are behaving this way. Can you clarify? For instance, you believe it's ready to deploy, does someone disagree, if so who and why? Are they ex governement beaurocrats

This sounds a lot like a capability vs desire alignment problem, you have the desire to release, but not the capablity (authority), those with the authority do not have the desire.

You need to find out why they lack the desire - it is marketing reason (keep it for another year while we milk the old version for a bit more), is it fear/confidence (If it has bugs, it will make us look bad, cost us money ...., lack of resource (Can't afford the cost of shipping/packaging/PR stuff), is it commercial where your some obscure sink to poor excess profits and they don't want you making money (Not as silly as it sounds).

I also suspect theres litle respect between parties involved, therefore reasonable adult discussions are not occuring.

Sorry - not the specific answer you have asked for, hopefully a couple of things to think about though.

share|improve this answer
1  
The second-to-last paragraph is a pretty good analysis of the problem - it's a political issue, not a technical one. Obviously for privacy reasons I cannot go into much more elaborate detail, and I don't think it's really pertinent to the solution, either. Regardless of who is "blocking" and why, I think that the only long-term solution is to convince upper management that the dev team needs to be in control of the process and specifically a process involving firm ship dates planned in advance. –  Aaronaught Mar 10 '12 at 21:34
3  
One point to note: Its not normal for Dev to be in charge of delivery dates. Dev has input into these dates, but the unless dates are set for business reasons, rather than technical reasons, the business is in trouble. –  mattnz Mar 10 '12 at 21:37
    
You do make a good point highlighting the subtlety here - this is more about getting an executive commitment behind regular ship dates than it is about having complete control over the schedule. Of course the business will ultimately decide when and how often to ship, but these should be decided far in advance, with the dev team having significant input, and most importantly of all they can't be up for debate 2 weeks before (or worse, 2 weeks after) the expected ship date unless there's a sound business reason, with the onus on the blocker to justify it. –  Aaronaught Mar 10 '12 at 22:11
    
Any other process, to me, is just chaos. If you don't have any idea when your project is really going to ship then it's nearly impossible to do the proper scoping or design. –  Aaronaught Mar 10 '12 at 22:12
2  
@Aaronaught - It may be as simple as politics/collaboration. You could get yourself in hot water that your management is trying to protect you from. Let me suggest this logic - assume that shipping is not in the best interest of the business in every and all case. Hence there must be some reasons/situations where it is in the interest of the business not to ship. Not knowing the full situation top to bottom doesn't make you wrong, it also doesn't make management wrong either. –  jasonk Mar 14 '12 at 2:19
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.