There are some fine answers here but I wanted to echo the same basic idea, yet with a twist.
GPU rasterization is actually pretty amazing at rendering epic amounts of "homogeneous" primitives. For example, I don't even have a very good GPU (GTX 560 Ti) and can rasterize 4 million wireframe lines in 3D through a vertex and frag shader at over 30 frames per second through the GPU.
It would be difficult (if even possible) for a CPU-based rasterizer to come even close despite the amount of conceptual work wasted by the GPU. The GPU just chews through these primitives like nothing, even though it's not even designed to really do this (ex: shader and matrix transforms are wasted with just 2D rasterization).
But the keyword is "homogeneous". Once you start mixing a variety of primitives that all require different shaders to rasterize, possibly even some textures here and there, it's suddenly exponentially harder to get decent performance. Texture images might need to be coalesced into texture atlases, like sprite sheets, with the primitives also sorted to minimize context switches for shaders and textures. Just getting nice subpixel antialiasing on rendered lines required a cutting-edge research paper previously. It might be possible to get something amazing out of that after a whole lot of proprietary work, it's just really hard, and for some desktop application like a web browser or a word processor's GUI, that's a whole lot of work for something that was never the main bottleneck in the the first place. Naive attempts like converting a bezier curve to line strips is also likely to cost more than it saves than rasterizing it directly on the fly with De Casteljau, and it definitely hogs up more memory to have to store all this state.
Vector graphics are awesome and featherweight, I wished we used them more (I can imagine all kinds of applications requiring considerably less disk space and install time, for example, if they used vector graphics). But we don't necessarily need a GPU pipeline to render them quickly.
Another thing is that people tend to underestimate the power of today's CPUs. They are freaking beasts of their own. I actually beat my colleague's GPU code rendering epic bloom over NVidia titans by implementing the bloom filter on the CPU.
His GPU bloom was faster than mine for a really small blur radius, like 3 or 4 pixels. Where mine outperformed it was for bigger amounts of blur, like 20+ pixels, and in that screenshot, ridiculous 400-pixel radius blurs at over 60 frames per second on the CPU.
A lot of people seem to think the CPU can't do this stuff when it can. It's not even difficult, it took me less time to write that blur filter than my peer, who is far better at GPU shaders than me, took to write his deferred frag shader. All I did was multithread the image filter and use SIMD and a tiled memory access pattern for cache efficiency.
So I don't know if that's at all a fair comparison of CPU vs. GPU, maybe my colleague's blur shader wasn't so great. But judging by the surprise people had here, I think a lot of people underestimate the power of the CPU while overestimating the power of the GPU. It's easy to read numbers like 2000+ cuda cores and think that's 500+ times better than an i7 with a measly 4 cores, but it's apples and oranges.
Another place where CPUs can actually start to compete with GPUs, counter-intuitively, is particle rasterization, where I've found it easy to render tens of millions of particles on my i7 with just CPU processing. The fill rate of all kinds of little transparent particles on the GPU doesn't seem to be that great.
I think one of the reasons GPU programming is getting so popular is that there it's more excusable to think micro -- after all, we have to for efficient GPU code. Papers published on GPU techniques focus on things like reducing the number of arithmetical ops performed per fragment -- like uber micro assembly-level thinking. CPU coding is often accompanied by a strong disapproval against micro-optimization (probably largely in part by the vast number of libraries available which have already been micro-optimized for us combined with the competence of optimizing compilers at tasks like register allocation) which has more people reluctant to try things like handwritten SIMD intrinsics, when they can be just as rewarding (if done well) from a performance standpoint. So GPU is, to me, somewhat of an excuse for people even using much higher-level languages to get all micro-level again with their efficiency focus, where they might experience some of the same benefits they would have if they applied that kind of focus towards CPU code.
That's not a slant on GPUs though, since I'm also trying to use them more and more since there are some things they do undeniably better than CPU. But they're not necessarily better at everything, and not even necessarily at the things people often think they're better at (complex image processing with non-sequential access patterns, e.g.), and complex vector rasterization with antialiasing, gradients, drop shadows, bezier paths, things like that might not actually have the GPU providing such an edge, or at least not as much of an edge as some people might think.