Infamous failure due to the lack of release process?

Question

With my fellow engineers we are putting in place a release process in a team that formerly was made only of biologists. I am going to make a presentation to explain the process to them and explain what advantages it brings.

I am looking for one example of a project that failed or got into real trouble due to the lack of release process. Are you aware of any ?

Answer 1

I can't quote you any widely known infamous failures, only stories from my own experience.

Back in the days, we were developing a Windows app, and we had no formal release process. Several times we had real trouble reproducing bugs which occurred regularly on a client machine, but never on most dev machines.

In some cases, this was due to different versions of the same system/environment libraries installed on different machines. In other cases, we couldn't track back which version of the code was actually running on the client machine in question.

Another issue was that often we couldn't tell easily to clients what features / bugfixes were released in which product version. We tried to keep track of these manually, but of course it was error-prone, and sometimes we had to make educated guesses. While I don't remember this having caused any concrete trouble with customers, it definitely felt bad.

In another company later, we already had a documented manual build process. Of course, as we were only humans, every now and then the guy who happened to create the actual build misunderstood or messed up something, like forgot to put a label on the actual version of the project in the SCM, or put the wrong label, or copied the wrong version of some file into the release package, etc. Needless to say, it caused us grief every time down the line.

In the end I decided to write a script to automate the release process. People only had to pass in a few parameters, and it did some parameter checking too. This drastically improved the quality of our builds, and soon producing releases became a non-issue, like it should be.

Answer 2

With some help from Google, I came across this release process error in the military's Early Warning System (to detect nuclear strikes by the USSR):

To test the warning system, false attack data was intermixed with data
from actual satellite observations, put on tape, and run through the
system.  On November 9 [1979], the test tape of this sort was accidentally
left mounted on a secondary backup computer.  This machine was left
connected to the system in use.  When the primary computer failed, a
backup computer was activated, which also failed.  Then the secondary
computer came into play, causing the alert.

The release process should of course have prevented fake data entering a release machine, even if that machine was a 2nd level backup.

Answer 3

Microsoft's DRM system for XP, "Genuine Advantage" broke down when they put preproduction software on their server.