Is it viable to charge for an open-source product? Both legally and financially, it seems like a very strange thing to do. However, I keep hearing people say that it's possible. Are there even any real-world examples for this sort of thing?
There's no legal obstacles. For any license the Open Source Initiative would call open source, or the Free Software Foundation free, you can sell your software for as much as you can get.
It does limit the business models. Selling shrinkwrap software is pointless when anybody can turn around and resell it for any price, or give it away, without you getting anything. Typically, a company will sell the software with something else.
For example, Red Hat sells Red Hat Enterprise Linux, and does reasonably well, even though some people take every release, remove everything proprietary, and distribute it as CentOS. Having real RHEL means getting certain kinds of support from Red Hat, and this is very valuable in an enterprise environment.
It used to be that Linux distros could be bought in a store for a certain sum of money, back when we couldn't download install DVDs conveniently. People will pay for convenience.
I've heard of cases where a software house sold a customer a GPLed system customized for them. The customer generally has no interest in seeing a release of their private software. (The GPL doesn't mandate that you have to redistribute anything, just that you can't redistribute without providing source or while changing the license.)
A lot of open source software is embedded into products, and forms a vital part. Lots of specialized devices run Linux, for example. I don't know whether you'd consider this selling Linux.
A few examples:
Although I can't think of any gcc distribution that's sold, Apple uses it as the development framework for their quite proprietary Unix OS. A lot of free software serves this kind of purpose.
Sure, you can sell an Ubuntu DVD and ship it places. You're not selling the open and free software, so much as the physical DVD itself.
Or there is RedHat's angle, which was to sell phone support to people using their distribution.
I've heard of projects that have charged people for extensions. Say for example, you have a program that deals with png files, they would pay for the program to handle jpeg files as well. But I don't know of any examples off hand.
But there's Maptool's donation wall, which is close. They have a list of features they're going to add, but donators can vote with their wallet which feature is developed first.
And it's entirely legal to sell BSD or any derivative thereof. Which explains OSX.
There's money to be made with the open model, but it has to be made at oblique angles.