What makes CPU cache memory so much faster than main memory? I can see some benefit in a tiered cache system. It makes sense that a smaller cache is faster to search. But there must be more to it.
We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.
In the case of a CPU cache, it is faster because it's on the same die as the processor. In other words, the requested data doesn't have to be bussed over to the processor; it's already there.
In the case of the cache on a hard drive, it's faster because it's in solid state memory, and not still on the rotating platters.
In the case of the cache on a web site, it's faster because the data has already been retrieved from the database (which, in some cases, could be located anywhere in the world).
So it's about locality, mostly. Cache eliminates the data transfer step.
Locality is a fancy way of saying data that is "close together," either in time or space. Caching with a smaller, faster (but generally more expensive) memory works because typically a relatively small amount of the overall data is the data that is being accessed the most often.
It is faster because both it is closer and because it is SRAM not DRAM.
SRAM is and can be considerably faster than DRAM the values are kept statically (the S in SRAM) so they don't have to be refreshed which takes away cycles. DRAM is dynamic, like tiny rechargeable batteries, you have to regularly recharge the ones so they don't drain away and become zeros. This steals cycle time in addition to how you have to access the bits, etc.
Being on the same die as or nearer the processor reduces the round trip, both L1 and L2 are faster than DRAM from an access perspective.
SRAM is faster to access than DRAM taken apples to apples, and the caches are usually on chip or closer or on faster busses than the DRAM making the access time faster as well.
One thing that should be mentioned explicitly is the impact of the speed of light. In this video Grace Hopper shows a piece of wire about a foot long, which is how far an electrical signal can travel in one nanosecond*. If a CPU is operating at 3GHz, then that implies a distance of 4" per clock cycle. This is a hard physical limit on memory access speeds. This is a large part of why being close to CPU (as L1 cache is), allows memory to be faster.
EDIT *actually how far light can travel in a vacuum, the distance through copper/silicon is less.
One of the philosophies I studied was the obtain-maximum-throughput-in-minimum- hardware movement when we talk about any cache based memory, be it CPU cache, buffer cache or memory cache for that purpose. The basic motive is achieved when there is the least or no hardware movement for retrieving/reading/writing data and the operation is completed faster.
The data transfers from disk -> main memory (RAM)(temporary storage) -> CPU cache (smaller temporary storage near the CPU for frequently accessed data) -> CPU (processing).
The CPU cache is a smaller, faster memory which stores copies of the data from the most frequently used main memory locations.
The buffer cache is a main memory which stores copies of the data from the most frequently used disk locations.
The memory cache is a memory which stores copies of the data from the most frequently visited websites by users.
Reference: How Computer Memory Works
Other answers already covered all the relevant bits: locality (and the associated data transfer cost, bus width and clock, and so on); speed of light (again, associated to transfer costs and bus width and throughput); different memory technology (SRAM vs.DRAM). All of this seen in the light of cost/performance balance.
One bit that was left out and it's just mentioned in Darkhogg comment: larger caches have better hit rates but longer latency. Multiple levels of cache where introduced also to address this tradeoff.
There is an excellent question and answer on this point on electronics SE
From the answers, it seems to me that a point to be highlighted is: the logic which performs all the required operations for a cache read is not that simple (especially if the cache is set-associative, like most caches today). It requires gates, and logic. So, even if we rule out cost and die space
protected by maple_shaft♦ Mar 31 '14 at 11:03
Thank you for your interest in this question.
Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.
Would you like to answer one of these unanswered questions instead?