The rule of thumb for threads is, you want at least one "active" (able to have its commands executed immediately given CPU time) worker thread for each "execution unit" available on the computer. An "execution unit" is one logical instruction processor, so a quad-chip, quad-core Xeon hyperthreaded server would have 32 EUs (4 chips, 4 cores per chip, each hyperthreaded). Your average Core i7 would have 8.
One thread per EU is the fullest use of the CPU's power, provided that the threads will always be in a running state; this is almost never the case, as threads need access to non-cached memory, the hard disk, network ports, etc. that they must wait for, and that don't require active CPU attention to perform. You can thus further increase overall efficiency with more threads queued up and raring to go. This does come at a cost; when a CPU switches a thread, it must cache the thread's registers, execution pointer and other state info normally kept in the innermost workings of an EU and very quickly accessed, allowing other EUs in that CPU chip to pick it up. It also requires threads in the OS to decide which thread should be switched to. Lastly, when an EU switches threads, it loses the performance gains of the pipelining that most processor architectures use; it has to flush the pipeline before switching threads. But, as all this still takes far less time on average than simply waiting for the hard drive or even RAM to come back with information, it's worth the cost.
However, in general, once you get beyond twice the number of "active" threads as EUs, the OS starts spending more of the EUs' time scheduling threads, and the EUs spend more time switching between them, than are actually spent running active threads of programs. This is the point of diseconomies of scale; it will actually take longer for a multithreaded algorithm to run if you were to add an extra thread at this point.
So, overall, you want to maintain at least as many threads in your program as you have EUs on the computer, but you want to avoid having more than double that number that aren't waiting or sleeping.