The main reason for 20% time is to keep capacity utilization at 80% rather than at 100%.
You can think of a software development organization as a system that turns feature requests into developed features. You can model its behaviour using queueing theory.
If requests arrive faster than the system can service them, they queue up. When arrivals are slower, the queue size decreases. Because the arrival and service processes are random, the queue size changes randomly with time.
The mathematically inclined can ask about this "randomness": there must be some probability distribution, so what will the queue size be on average? Math (queueing theory) has an answer to that: if both arrival and service processes are Markov, then N = rho^2 / (1-rho).
(Where rho is the utilization coefficient equal to the ratio of service and arrival rates. If the processes are non-Markov, the math is more complicated, but doesn't change the conclusions.)
If you plot this function, you can see that the average queue length remains low while utilization is up to 0.8, then rises sharply and goes to infinity. You can understand this intuitively by thinking about your computer's CPU: when its utilization approaches 100%, the computer becomes unresponsive.
The economics of software development is such that software companies incur big costs when their queues are in high-queue states. This includes missed market opportunities, obsolete products, late projects, and waste caused by building features in anticipation of demand.
The 20% time is thus the scientific answer to the problem of optimizing economic outcomes: avoid high-queue states by avoiding utilization ratios causing them. It is essentially the slack that keeps the system responsive.
Several practical conclusions follow immediately:
- if you're considering 20% time and doing cost accounting (developers' time costs X, but/and the company can/cannot afford it), you're doing it wrong.
- if you're allocating 20% to a Friday every week, you're doing it wrong
- if you're setting up a 20% time project proposal submission/review/approval system, you're doing it wrong
- if you're filling out timesheets, you're doing it wrong
- if you're using innovation as a motivator for 20% time, you're doing it wrong. While new products have come out of 20% projects, they were not the point. If your company cannot innovate during its core hours, that's a problem!
- 20% time is not about creativity. Don't say you'll unleash your creativity with 20% time, ask why you're not creative enough already during your core hours.
ANSWERS TO QUESTIONS IN THE COMMENTS
Dan, you got that right and accurately described the mistake made by many. You cannot choose your utilization percentage, because it's an output variable. It is a ratio of characteristics of two processes, so it is what it is because the processes are the way they are. An organization does have influence over both processes; matching capability and demand is one of the hard problems addressed by the lean software development body of knowledge. Utilization is one of the indicators how well this problem solved in an organization. Slack emerges as your lean initiative progresses and you remove waste from the value stream. But if you mandate 20% time, you end up in the same utilization trap with less available capacity.
Kim, it is still partially a culture thing. The closest cultural reference I can think of is the synergistic level of the so-called Marshall model of organizational change. It emerges at the end of successful lean transformations or is present in organizations built lean from the start. (Here's a link to Bob Marshall's white paper (PDF).)
The above logic is well supported in the software engineering literature. Mary and Tom Poppendieck hinted at it in their 2006 book Implementing Lean Software Development. Donald Reinertsen in his 2009 book Principles of Product Development Flow (Chapter 3) gives thourough treatment of this subject, with formulas and graphs.