Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

What makes CPU cache memory so much faster than main memory? I can see some benefit in a tiered cache system. It makes sense that a smaller cache is faster to search. But there must be more to it.

share|improve this question

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

What do you mean by main memory? – marcocs Mar 31 '14 at 0:12
@marcocs edited for clarity. I'm referring to CPU cache memory and main system memory, i.e. the removable dimms on most motherboards. – ConditionRacer Mar 31 '14 at 0:25
because if cache wasn't faster it wouldn't be used in the first place – ratchet freak Mar 31 '14 at 11:28
@ratchetfreak Uh...OP is pretty clearly asking for a proximate cause, not a final cause. – Kyle Strand Mar 31 '14 at 22:30
up vote 100 down vote accepted

In the case of a CPU cache, it is faster because it's on the same die as the processor. In other words, the requested data doesn't have to be bussed over to the processor; it's already there.

In the case of the cache on a hard drive, it's faster because it's in solid state memory, and not still on the rotating platters.

In the case of the cache on a web site, it's faster because the data has already been retrieved from the database (which, in some cases, could be located anywhere in the world).

So it's about locality, mostly. Cache eliminates the data transfer step.

Locality is a fancy way of saying data that is "close together," either in time or space. Caching with a smaller, faster (but generally more expensive) memory works because typically a relatively small amount of the overall data is the data that is being accessed the most often.

Further Reading
Cache (Computing) on Wikipedia

share|improve this answer
I think this is a good answer. However, it may be worth adding that there is also a cost hierarchy to cache: network storage is cheaper than a local hard drive, which is cheaper than RAM, which is much cheaper than on-CPU cache. If it wasn't for this we'd just make CPUs with 120GB of memory on-chip, and be done with it. – Charles E. Grant Mar 31 '14 at 0:26
@ConditionRacer: Same reasons. L1 is faster, but more expensive, so they give you a little bit of very expensive L1, and more of the less expensive, but slower L2. See also… – Robert Harvey Mar 31 '14 at 0:29
"Why is L1 faster than L2" - the question becomes off topic for Programmers.SE, and should be asked on Electronics.SE – mattnz Mar 31 '14 at 0:42
@ConditionRacer IIRC, it's to do with the technology used to implement the memory. I think L2 is on-chip DRAM, whereas L1 is SRAM or something like that; much more expensive, but much faster. It's over a decade since I worked in CPU design… – Donal Fellows Mar 31 '14 at 1:24
@CharlesE.Grant : while the cost hierarchy is a good example, there is another reason we don't have 120 GB of on-chip cache: addressing larger memory needs larger addresses, so either larger instructions or more CPU cycles. So to be the fastest, the number of registers needs to remain relatively small. Of course, one could have on-chip memory besides the registers, but the fastest tier has to remain small, otherwise it would not be that fast anymore. – vsz Mar 31 '14 at 6:11

It is faster because both it is closer and because it is SRAM not DRAM.

SRAM is and can be considerably faster than DRAM the values are kept statically (the S in SRAM) so they don't have to be refreshed which takes away cycles. DRAM is dynamic, like tiny rechargeable batteries, you have to regularly recharge the ones so they don't drain away and become zeros. This steals cycle time in addition to how you have to access the bits, etc.

Being on the same die as or nearer the processor reduces the round trip, both L1 and L2 are faster than DRAM from an access perspective.

SRAM is faster to access than DRAM taken apples to apples, and the caches are usually on chip or closer or on faster busses than the DRAM making the access time faster as well.

share|improve this answer
SRAM is also more power-hungry per bit, so you'd have to add radiators to it, and anything battery-powered would suffer. – 9000 Mar 31 '14 at 7:28
Not to mention the fact that making SRAM bigger makes it WAY slower. – Darkhogg Mar 31 '14 at 11:30
agreed, more power, and slower relative to other sram. I think it is 4 transistors per bit for sram and 1 per bit for dram. dram relies on the capacitance in the fet, where sram relies on good old fashioned logic. so you have to power all of those transistors all the time and not just pump the charge periodically. – dwelch Mar 31 '14 at 13:46
SRAM in caches generally is 6T, not 4T. Also, SRAM (certainly the 6T variant) is more energy efficient. The real power draw in DRAM is the refresh cycle, and SRAM just doesn't need that. SRAM transistors draw power when switching, the DRAM capacitor leaks all the time. If you replace the leaky cap of DRAM, you end up with EEPROM but that's not fast: if it's hard for the charge to leak out, it's also hard to get it out the normal way. – MSalters Mar 31 '14 at 23:43
@dwelch CMOS FETs don't drain power if they are in open or closed state, so the argument about power consumption is invalid. Moreover, it's the CMOS logic switching, which makes CPUs drain most of their power - the current peaks in the intermediate state (non-1 and non-0), so the situation is inverse to what you say :) – Ruslan Apr 1 '14 at 10:34

One thing that should be mentioned explicitly is the impact of the speed of light. In this video Grace Hopper shows a piece of wire about a foot long, which is how far an electrical signal can travel in one nanosecond*. If a CPU is operating at 3GHz, then that implies a distance of 4" per clock cycle. This is a hard physical limit on memory access speeds. This is a large part of why being close to CPU (as L1 cache is), allows memory to be faster.

EDIT *actually how far light can travel in a vacuum, the distance through copper/silicon is less.

share|improve this answer
Hard limit for copper. This is a good point though, I didn't consider how ridiculously fast modern processors are. – ConditionRacer Mar 31 '14 at 5:00
@ConditionRacer I just added a correction, 11.8 inches is light in a vacuum. – Matthew Finlay Mar 31 '14 at 5:04
You need to halve that number since you need a round-trip to retrieve memory. So it's just 5cm not 10cm per cycle of latency in vacuum. – CodesInChaos Mar 31 '14 at 9:53
The speed of light is why the main memory bus clock stopped getting faster some years ago (2000ish?) It's not a direct effect -- it has to do with CMOS logic signals not being perfect square waves anymore at bus frequencies -- but c shows up prominently in the math. – zwol Mar 31 '14 at 17:10

Other answers already covered all the relevant bits: locality (and the associated data transfer cost, bus width and clock, and so on); speed of light (again, associated to transfer costs and bus width and throughput); different memory technology (SRAM vs.DRAM). All of this seen in the light of cost/performance balance.

One bit that was left out and it's just mentioned in Darkhogg comment: larger caches have better hit rates but longer latency. Multiple levels of cache where introduced also to address this tradeoff.

There is an excellent question and answer on this point on electronics SE

From the answers, it seems to me that a point to be highlighted is: the logic which performs all the required operations for a cache read is not that simple (especially if the cache is set-associative, like most caches today). It requires gates, and logic. So, even if we rule out cost and die space

If someone would try to implement a ridiculously large L1 cache, the logic which performs all the required operations for a cache read would also become large. At some point, the propagation delay through all this logic would be too long and the operations which had taken just a single clock cycle beforehand would have to be split into several clock cycles. This will rise the latency.

share|improve this answer

There are a lot of good points raised in the other answers, but one factor appears to be missing: address decoding latency.

The following is a vast oversimplification of how memory address decoding works, but it gives a good idea of why large DRAM chips ar generally quite slow.

When the processor needs to access memory, it sends a command to the memory chip to select the specific word it wants to use. This command is called a Column Address Select (we'll ignore row addresses for now). The memory chip now has to activate the column requested, which it does by sending the address down a cascade of logic gates to make a single write that connects to all the cells in the column. Depending on how it's implemented, there will be a certain amount of delay for each bit of address until the result comes out the other end. This is called the CAS latency of the memory. Because those bits have to be examined sequentially, this process takes a lot longer than a processor cycle (which usually has only a few transistors in sequence to wait for). It also takes a lot longer than a bus cycle (which is usually a few times slower than a processor cycle). A CAS command on a typical memory chip is likely to take on the order of 5ns (IIRC - it's been a while since I looked at timings), which is more than an order of magnitude slower than a, processor cycle.

Fortunately, we break addresses into three parts (column, row, and bank) which allows each part to be smaller and process those parts concurrently, otherwise the latency would be even longer.

Processor cache, however, does not have this problem. Not only is it much smaller, so address translation is an easier job, it actually doesn't need to translate more than a small fragment of the address (in some variants, none of it at all) because it is associative. That means that along side each cached line of memory, there are extra memory cells that store part (or all) of the address. Obviously this makes the cache even more expensive, but it means that all of the cells can be queried to see whether they have the particular line of memory we want simultaneously, and then the only one (hopefully) that has the right data will dump it onto a bus that connects the entire memory to the main processor core. This happens in less than a cycle, because it is much simpler.

share|improve this answer

One of the philosophies I studied was the obtain-maximum-throughput-in-minimum- hardware movement when we talk about any cache based memory, be it CPU cache, buffer cache or memory cache for that purpose. The basic motive is achieved when there is the least or no hardware movement for retrieving/reading/writing data and the operation is completed faster.

The data transfers from disk -> main memory (RAM)(temporary storage) -> CPU cache (smaller temporary storage near the CPU for frequently accessed data) -> CPU (processing).

The CPU cache is a smaller, faster memory space which stores copies of the data from the most recently used main memory locations.

The buffer cache is a main memory area which stores copies of the data from the most recently used disk locations.

The browser cache is directory or similar space which stores copies of the data from the most recently visited websites by users.

Reference: How Computer Memory Works

share|improve this answer
"operating systems philosophy is..." -- is this only your opinion or you can back it up somehow? – gnat Mar 31 '14 at 10:16
Was taught about this while studying Unix by Maurice Bach. Unfortunately no supportive documentation. Edited.. – roshan4074 Mar 31 '14 at 10:47
howstuffworks is notoriously unreliable for getting reliable technical information. They're nice for gross simplifications that the average Joe and Jane can understand, but that's about it. – jwenting Mar 31 '14 at 11:27

protected by maple_shaft Mar 31 '14 at 11:03

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.