Your problem is setting the margin of error too strictly for classifying an image as black and white. The trick is to not go too far the other way and falsely classify color images as black and white.
Do some research on Bayes classification. Basically, you do the classification manually for a sample of your images. That will give you the total probability of an image being black and white in your collection, as well as the probabilities of an average saturation in any given black and white image. Then, you use a formula to compare the probability of a new unknown image being a black and white image versus the probability of it being color, given its average saturation.
For example, say 10% of the images in your collection are black and white, 20% of black and white images have an average saturation of 5, but only 1% of color images have an average saturation of 5.
black and white posterior for saturation of 5 = 0.1 * 0.2 = 0.02
color posterior for saturation of 5 = 0.9 * 0.01 = 0.009
So you would classify an image with saturation of 5 as black and white.
The nice thing about Bayes classification is that you can use more than one factor in your determination, if you find another feature that you think is useful in addition to saturation. Maybe maximum saturation works better than average, or you can combine the two. Also, you don't have to guess at the right cutoff point, because it's based on your training set.