Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We are trying to make our releases more frequent. As part of that we have blocked out a day of the week where we told users that we'll be doing updates. Similar how Microsoft has Patch Tuesday.

This is for a web app running completely server side, by the way. So updates are not a huge problem to deploy. Though they can get time consuming.

Occasionally an update has a bug that needs to be fixed before the next update window rolls around. I see two real options: either undo the update and restore a backup of the previous version, or make a fix and deploy it outside the scheduled time.

Making a fix sounds easy, but often times there are other changes that have been made to the code base. And the fix needs testing before it can realistically go out.

Rolling back isn't a bad option, but sometimes that will remove recent features or at least delay them until the next week.

share|improve this question

migrated from serverfault.com Mar 16 '12 at 2:36

This question came from our site for professional system and network administrators.

1 Answer 1

You seem to have a development problem. I know this because you said:
Making a fix sounds easy, but often times there are other changes that have been made to the code base. And the fix needs testing before it can realistically go out.

You need to isolate you production releases in your version control system (you are using one, right?). Create a branch for your release. Bugs that are discovered in the production release get fixed in that branch (and the fix merged back to the main development line). Bugs that are fixed in main development have the fix brought into the branch if practical (or a parallel fix created if not).

This pretty much guarantees that fixes to the running production code will be tiny fixes, and easy to test (therefore quick to roll out if there's a problem), and hopefully makes them less disruptive.


Outside of that here's the best advice I can offer you:

  1. Have some kind of internal test cycle.
    Don't release code that doesn't work, or at least inflict it on a small set of users first.
    (StackExchange uses meta.stackoverflow.com as their scratch monkey.)

  2. Cut releases when you're reasonably confident they work
    (The above-mentioned branching)

  3. Fix bugs before adding features
    (Liberally stolen from The Joel Test)

  4. Don't be afraid not to release something if it's not ready.
    Better not to ship a patch on "patch day" than to break the universe.

  5. Don't be afraid to roll back something if it's not working.
    (Corollary: Always have a back-out plan!)

  6. Set a release schedule you can keep.
    Once a week is HARD. Once a month is tough if you're doing more than bug-fixing. Once a quarter is pretty easy.

For reference, my company releases software the weekend following the 15th of every mid-quarter month (i.e. the weekend after February 15, May 15, August 15, November 15). For various regulatory reasons our testing cycle is pretty rigorous, and this is as fast a pace as we can maintain while keeping software quality where we need it to be.
Emergency bug-fix patches are released as-needed, usually on Fridays (to give us a weekend where we have relatively low volume in case the fix breaks something else).

share|improve this answer
    
Thanks for the input. We were branching, but have since cut back on it since it ate up a lot of time and confused some people. We do test, but sometimes things slip by for one reason or another. And we are fixing bugs primarily, but those can trigger other bugs. Once a week is a little difficult, especially when it takes away from development time. –  David Hogue Mar 15 '12 at 21:27
    
Branching shouldn't be time consuming (how long does it take you to type git branch myproduct-1.3 and git checkout some-branch?). If your version control system makes it hard you may want to consider changing version control systems before getting more aggressive in your development cycle :) –  voretaq7 Mar 15 '12 at 21:43
    
It's not the running of a branch command takes that long. It's when they diverge, especially when the branches are maintained for long periods of time, that miscommunications and mistakes start to happen. –  David Hogue Mar 15 '12 at 21:56
1  
@DavidHogue The branches shouldn't diverge because you should merge any changes made back into the main development line as soon as they're made.They shouldn't really be maintained for a long time (though I know from experience its never quite that simple) –  Murph Mar 16 '12 at 12:32
    
@Murph Even with long-lived branches that start diverging (e.g. FreeBSD's RELENG_# branches) the maintenance in a branch shouldn't be onerous -- it's bug fixes only, or critical pull-up merges from the trunk. When you make your next major release the previous release branch dies off and you don't have to worry about it anymore. –  voretaq7 Mar 16 '12 at 15:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.