What are some techniques for updating a production server's code base / database schema without causing any downtime?
For typical databases (Oracle for example) it's possible to alter the database schema while still running queries in parallel. It requires some forward planning though.
Their are some constraints for the change to be applied:
In order for the schema to be backward compatible, you usually can ADD or MODIFY a column, you can only DROP something if the existing code does not use it any longer.
If your code cannot handle the change transparently, then change the code before changing the database.
Simple advice on forward planning: always explicit the column names in your DB requests (don't use
Generally, the websites I've worked on that had this sort of requirement were all behind load-balancers, or had separate failover locations. In this sample, I'll presume that you've got a single load balancer, 2 web servers (A & B) and 2 database servers (M & N - usually DB servers are linked via logshipping - at least in the SQL Server world).
In a very complicated web applications, what is described as steps 1-5 might take all night and be a 50 page Excel spreadsheet with times and emergency contact numbers. In such situations, updating half the system is scheduled for 6pm to 6am while leaving the system available to users. Handling the update for the DR site is usually scheduled for the following night - just hope nothing breaks the first day.
Where uptime is a requirement, patches are tested first on the QA environment, which ideally is the same hardware as production. If they show no disruption, they can then be applied on the regular schedule, which is usually on the weekend.
Not all systems can, it has to be set up in a manner that supports it.
For example, one of our major systems I helped upgrade a few years ago should be available 24/7. It consisted of multiple tiers, including a pure communication tier between the off site User Interface Layer and Business Layer. Due to the way the communication layer was coded, any future changes to the Business Layer or DB schema could be implemented with out a real outage. In the worst case scenario, a user would experience a 10-30 second pause as the changes took affect.
In fact, if the changes were purely code changes to the business layer, they could be queued up and 'cycled in' with only milliseconds delay.
It could do this because:
Other techniques I've heard about involve replication of transactions to another mirror of the existing system. By applying the update to one, switching over and replaying all transactions done between the update and switch. YMMV depending on your systems though.
Here's a different perspective, from the world of embedded database systems and embedded systems. Embedded systems include various network/telecommunications infrastructure equipment, and in this realm they often talk about 99.999% (five 9s) uptime.
We (McObject) are the vendor of the eXtremeDB family of embedded database system products, including eXtremeDB High Availability.
First, understand that "embedded database" means that the database system is a library that is compiled and linked in with your application code; in that sense, it is "embedded" in your application.
With eXtremeDB High Availability, there is a MASTER instance of your application (which might be one or several processes) and one or more REPLICA instances of your application. When a replica establishes a connection to the master, it receives a copy of the master's database through a process called "initial synchronization". This can be done while the master application continues its work. Once syncrhonized, it receives the master's transactions through replication. Therefore, a replica always has current data and can take over (through a process called failover) in the event the master fails.
One feature of the initial synchronization is called "binary schema evolution." In plain English, this means that the process of populating the replica's database will accommodate differences between the replica's database schema and the master's database schema.
In practice, this means that you can build a newer version of your application (with new/dropped tables, new/dropped/changed fields, new/dropped indexes), attach that new version of your application to a master, and then cause that newer replica to become the new master (i.e. force a failover to the new replica so it becomes the master and the old master shuts itself down). Voila, you have migrated your application from version N to N+1, without interrupting the availability of your system. Now you can go about upgrading the old master and any other replicas to version N+1.