Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

At my company (3-yr-old web industry startup), we have frequent problems with the product team saying "aaaah this is a crisis patch it now!" (doesn't everybody?)

This has an impact on the productivity (and morale) of engineering staff, self included. Management has spent some time thinking about how to reduce the frequency of these same-day requests and has come up with the solution that we are going to have a release every week. (Previously we'd been doing one every two weeks, which usually slipped by a couple of days or so.)

There are 13 developers and 6 local / 9 offshore testers; the theory is that only 4 developers (and all testers) will work on even-numbered releases, unless a piece of work comes up that really requires some specific expertise from one of the other devs. Each cycle will contain two days of dev work and two days of QA work (plus 1 day of scoping / triage / ...).

My questions are:

(a) Does anyone have experience with this length of release cycle?

(b) Has anyone heard of this length of release cycle even being attempted?

(c) If (a) or (b), how on Earth do you make it work? (Any pitfalls to avoid, etc., are also appreciated.)

(d) How can we minimize the damage if this effort fails?

share|improve this question
    
What do you mean by "crisis job"? –  Lorenzo Oct 12 '10 at 7:31
    
Requests that we are instructed to patch out the same day they are received. Editing the question to make it clearer momentarily. –  Arkaaito Oct 12 '10 at 7:33
add comment

4 Answers

up vote 7 down vote accepted

You can certainly deliver every week - or even more frequently. At the moment we generally release every two weeks, but it isn't unusual to deploy functionality when something has arrived with no notice from one of our partners that would be irrelevant if we waited for the next cycle. At some point in the next few months I'd like us to move to continuous delivery (items are released as soon as is practical once they're 'done') as standard, but we're not quite confident enough yet to go that far.

The critical thing is you need your website strongly covered by automated tests - both unit tests and end-to-end acceptance tests/executable specifications. By implication this also means that your build is fully automated. At the acceptance level we use Robot Framework which is excellent for quickly building up a maintainable test suite thanks to it's keyword approach. For look and feel our onsite tester makes some cursory checks, but we also have a couple of guys in India who do a more thorough check across different browsers (there are sites which help with this sort of thing by taking screenshots for you, e.g. BrowserLab).

We don't fully automate the deployment (the very final step requires manual intervention, this is a concious decision for us) - but we do automate all the things like ensuring that the correct database connections are being used, etc, with short deployment cycles it'd be too easy to make a mistake with this sort of thing.

There's a pretty good recentish book on continuous delivery that you might want to check out, I've skimmed it but not gone through it in detail yet. What I've read so far chimes well with our experiences though: Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

In summary you need a highly disciplined team, a high level of automation and - most important of all - an extremely high degree of trust in that automation. To me it seems that moving to weekly cycles in your case may be a mistake - crisis patches hint at other issues and you should work to eliminate those. Upping the tempo could potential make the situation worse...

share|improve this answer
add comment

If you are continually in "crisis release" mode, I'd say it would be more prudent to take a step back and reevaluate your code and your process. Obviously, there is some kind of failure there that just keeps getting repeated.

While it may not be entirely possible to do this on a true production scale, but it would probably be worthwhile to have at least one senior member and some other subset of developers and testers be dedicated to this evaluation.

The "4 on at a time approach" doesn't sound like a clear-cut winner to me. That means constant context switching, which means far less efficiency.

Remember, if you are constantly hurrying to make changes, you are much more likely to make mistakes in those changes, and break something else while you're doing it.

share|improve this answer
    
if I may add my own comments to @Wonko's great answer... Our company spent several years doing things similar to the OP. Growing from 6 or devs to 16. About 2 years ago, we decided to go Agile. We hired a Sr. Dev w/ Agile experience, switched 2 week iterations, implemented continuous integration, etc... We're still far from a textbook shop and it's been bumpy from time to time but we've really cut down the context switching, which is a huge win. –  DevSolo Oct 12 '10 at 17:26
add comment

Release cycles shorter than one week have certainly been achieved in the software industry. They employ the technique called continuous delivery (also called continuous deployment).

There is a company that releases 50 times a day. This blog post describes how they do it.

share|improve this answer
add comment

Management has spent some time thinking about how to reduce the frequency of these "crisis jobs" and has come up with the solution that we are going to have a release every week.

Looks like your managament are world champions... why not invest in your team spirit intead ? You will see the problems will disapear by themselve.

(a) + (b) IMHO, according the size of your team it should be two weeks max. One week will work for one man shows or very tiny teams (like 2 or 3).

(c) + (d) But regardless the size of your team or project, one of the first thing I do, is automatize the build & deployment. I save days if not weeks of work by doing that in the early days of a project.

Your deployments must be done with one click. From source to staging, then from staging to production. There are a lot of tools to make this possible. From ant/nant to super heavy stuff like Automated Build Studio.

Everything can be automated from files deployments to database upgrade, including backups, notifications, reporting, ...

share|improve this answer
    
Not entirely sure what you're proposing here - are you saying that the problem management should be addressing is a lack of team spirit? Or are you saying that I'm demonstrating a lack of team spirit by trying to figure out how to make this work (and being nervous about the prospects of success)? –  Arkaaito Oct 12 '10 at 7:15
    
I can't give an objective opinion on your situation when described in few lines of texts. However, I observed that lack of team spirit is usually the root cause of many problems in organization like yours. Regardless that problem I suggest you to address, automating the deployment process will improve your experience –  user2567 Oct 12 '10 at 7:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.