The simple answer is that a GPU works best when you need to do a fairly small, fairly simple computation on each of a very large number of items. To accomplish much this way, the computation for each item must be independent of the computations for the other items. If there's (normally) some dependency between one item and another, you generally need to figure out some way to break it before you're going to get much out of executing that code on the GPU. If the dependency can't be broken at all, or requires too much work to break, the code might execute faster on the CPU.
Most current CPUs also support quite a few types of operations that a current GPUs simply don't attempt to support at all (e.g., memory protection for multitasking).
Looking at it from a slightly different direction, CPUs have been (largely) designed to be reasonably convenient for programmers, and the hardware people have done their best (and a darned good best it is!) to create hardware that maintains that convenient model for the programmer, but still executes as quickly as possible.
GPUs come at things from rather the opposite direction: they're designed largely to be convenient for the hardware designer, and things like OpenCL have attempted to provide as reasonable of a programming model as possible given the constraints of the hardware.
Writing code to run on a GPU will typically take more time and effort (so it will cost more) than doing the same on the CPU. As such, doing so primarily makes sense when/if either:
- The problem is so parallel that you can expect a large gain from minimal effort, or
- The speed gain is so important that it justifies a lot of extra work.
There are some obvious possibilities for each -- but a huge number of applications clearly aren't even close to either one. I'd be quite surprised to see (for example) a CRUD application running on a GPU any time soon (and if it does, it'll probably happen because somebody set out with that exact goal in mind, not necessarily anything approaching an optimal cost/benefit ratio).
The reality is that for a lot of (I'm tempted to say "most") applications, a typical CPU is far more than fast enough, and programming convenience (leading to things like easier development of new features) is much more important than execution speed.