If by "open source" you mean "Free Software", then you have no other choice but to allow every possible kind of commercial exploitation, except those that would prevent people from accessing the sources. If you put restrictions on the source, such as mandating the inclusion of certain advertisements, then the software is no longer "free" (as in speech).
So you want to make money off of the software: that's fine, nothing wrong with that. But I believe making in "open source, but not really" isn't the right way to do it: you will miss out on many of the benefits of Free Software (for example, your advertisement clause will make it impossible to include your software in, say, debian), but you won't see much of the benefits of proprietary software either.
Going proprietary is one way to go, and unless you are a full-scale bearded hippie, I don't see any problem with that - you will miss out on free contributions, you will have a harder time gaining traction in the market, and you will have to actively persuade people to buy your product rather than just throwing it out there and waiting for people to bite; but you will also be able to put whatever price tag you like on the software, and pepper it with as many ads as you like.
The other alternative is to go completely Open Source: pick a suitable pre-written Free Software license (there is no point writing your own: the existing ones already cover pretty much all needs), and release it. You can't make any money from selling licenses, and you can't include ads in the software either (that would be pointless, as any fork could just remove them), but here are a few things you can exploit for money:
- Support. As the author, you are the prime source of inside information on the software; it's OK to make people pay for that information.
- Custom extensions and modifications. More often than you think, people are more than willing to pay to see your software incorporate feature X, even if that means everyone else will also get the feature. They don't pay your for exclusive rights to your development effort, but for prioritizing features they need the most.
- Resume building. If it becomes popular, you'll be famous. Not rock star famous, but enough to give your professional networking a significant boost and increase your employability.
And then, there's dual licensing - in short, you can offer the product gratis under an Open Source license (typically one from the GPL family), and offer the option to "upgrade" to a commercial license that allows things your Open Source license doesn't (such as re-selling with modifications without providing sources). This is a thin line though, and it won't help your 'street credit' in Free Software land.