How can I automate production deployments without experiencing extreme anxiety?

Question

At our shop we use SVN for source control and CruiseControl for CI on handling automatic builds and deployments to our development, test, and integration environments.

This all works smoothly however due to hardware and resource constraints our integration environment is not a 2 server load balanced environment like our production environment. While everything else is equal that would be the only difference between our integration and production environments (although a big one!)

Theoretically the difference is a slightly different configuration of our app servers and the deploy script merely would have to drop the build artifacts into two servers instead of just the one, but why am I so nervous to automate our production deployments?!

I am generally not a control freak but I always feel the insatiable need to deploy production to production manually. I have heard from colleagues that this is generally a Really BAD Thing™ but they failed to make a case against it.

I know that when I do it manually I can SEE that I am physically copying the correct files, I am physically shutting down the app servers and ensuring they closed successfully, I am physically starting the servers back up and then physically inspecting the logs to make sure it started up okay and the deployment was successful. It gives me a peace of mind.

What are arguments against this OR arguments for automatic scripted production deployment?

Answer 1

You can get the best of best worlds: peace of mind with process verification and the reliability of automation.

Script the deployment. Then, go through and manually verify that processes are started, files removed, etc. In other words, write your own QA script just to verify that automated steps 1 - X actually occurred.

Answer 2

There are a few obvious arguments against this.

1. What happens if you leave. Is all this information carefully documented, or is it mostly in your head. Automated scripts are a much better place for someone else to take over from.

2. Everyone makes mistakes. There will come a time when the person doing the deployment is tired, not paying attention whatever. Yes ideally deployments are only done in an happy calm place with lots of time. In practise they can be rushed and stressed when trying to roll out urgent fixes. This is the mostly likely time to make a mistake, and also the most costly. If the deployment is a single script then the potential for mistakes is limited.

3. Time. As deployments get more complicated the amount that needs doing increases. Scripts just require kicking off, a manual check, and then a manual switch-over (you could automate this as well, but I share some of the paranoia :).

Answer 3

Do you also run your production machines with remote debugging, and you manually stepping through them? Building a proper script is identical to writing a program. All of the issues you have indicate things that it will need to watch for and check against.

If something goes wrong, it should go through proper rollback procedures, and send you a message. Everything that happens can get logged for later. You can version control the scripts, and set up test cases.

But if you are manually running commands, you don't have any of these advantages. You instead have a list of disadvantages.

• You don't have a good log, shell history doesn't count
• No one else knows how to do it
• Steps get missed
• Checks are only sometimes done
• Some items to deploy may get missed, I've done that before
• It takes a lot longer
• You can get interrupted during the process

A proper script should be almost identical to if you typed everything out on the shell. This is one of the reasons that we have bash scripts. If you trust the things you do, why can't you record everything and tighten it up? Better checking, faster checking, more checking can happen because the computer does it.

Answer 4

I think the key here is: why do you think that you can't script the verification process?

My deploy scripts don't just push archives and restart services. They print out lots of color-coded information during each step of the deploy, and provide me a summary of events at the end. It lets me know that processes are up and running, that the homepage is serving a 200 status code, and that all the machines and services can see one another fine. I then have a separate service that's not part of the script which monitors log files, 4xx and 5xx-level errors, and key site metrics. It then proceeds to yell at me through every medium possible (email, txt msg, and alarms) if there are drastic negative-effect spikes.

Between that and CI servers running tests, I literally deploy and forget at this level of automation. I don't even browse a single page on the site after a push because of how reliable the process now is, which not only lets me deploy as often as I want, but lets a new developer on the project make an update to the live site within minutes of coming on board. In the past I've even made the CI servers auto-deploy to production after a commit to a master/trunk branch that passes everything. That's how confident I am in my tools.

You should be too.

Answer 5

I can understand being a bit nervous trying something new on the prod environment. Being wary of potential disaster is a Good Thing^TM.

Automated scripting is also a Good Thing^TM and so long as you approach it carefully, you should be able to minimise the danger and lower your fear. So my advice is this;

• Prepare (and practice on the integration env) a checklist/set of tests so you can quickly find out whether it worked and what if anything went wrong. Verbose logging may help with this.
• Back up everything. Prepare and practice a manual rollback so that you can recover if it goes badly wrong.
• Test as much as you can before you do it for real on prod. Sounds like you're a good way along with this with your integration env.
• First time you try it, do it on a low profile, low impact change. Something like a minor upgrade or patch. The idea is to minimise the fallout if it goes wrong. Do not choose a high profile major upgrade (where the CEO and all your competitors are watching) for your first run.

Once you've got a few successful runs under your belt your confidence will grow and soon you'll wonder how you ever managed doing manual deployments.

Answer 6

Run the scripts on the live server. It will work, and after you've seen it work fine a few times, you will be perfectly confident in it.

Seriously though, you are more likely to make mistakes than the deployment script.

Answer 7

WHat I like is you can test the deployment on staging or qa and know that when you run it on prod the exact same steps will happen. When you do it manaully it is easier to forget a step or do them out of order.

Answer 8

...due to hardware and resource constraints our integration environment is not a 2 server load balanced environment like our production environment. While everything else is equal that would be the only difference between our integration and production environments (although a big one!)

Given above, I would probably be as anxious as you.

I once did review and testing of automated script that deploys to SLB and my feeling is that without pre-testing at load balanced setup I'd prefer to manually do things.

---

Besides prod-like testing setup, another thing that had significant impact on my peace of mind is that prod deployment was done by other team that developers - by guys whose only job was to maintain production environment.

• In one of the projects I was assisting them in deployment as dev team representative. Prior to deployment, they were reviewing my instructions and during deployment I was just sitting online ready to consult if things go wrong. Back then, I learned to appreciate that separation.
   
  Not that they were faster (why would they? I did test deployments 5x-10x more frequently than them). The big difference was in focus. I mean, my head is always loaded by "main" stuff - coding, debugging, new features - there's just too much distractions to properly concentrate on deployment. As opposed to that, their main stuff was just production maintenance and they were focused on that.
   
  It's amazing how much better brain works when focused. These guys, they were just so much more attentive, they made so much less mistakes than me. They just knew that stuff better than me. They even taught me a thing or two that made my own test deployments easier.

Answer 9

Build a deployment script which you use to move your code into any environment. We use the exact same deployment process to move code to dev, qa, staging, and finally production. Since we're deploying multiple times per day to dev, and daily to QA, we've gained confidence that the deployment scripts are correct. Basically, test the hell out of it by using it often.

Answer 10

Computers don't make mistakes, people do.

Write your script once and thoroughly check it, go through it line by line. From then on you can be sure that each time you deploy, it will work.

Do it by hand and you're bound to make mistakes. Maybe you wrote, everything you have to do, down but it's oh so easy to make a mistake. You have to copy all files except the web.config file? You can bet that someday you will overwrite it. A script will never make this mistake.