Consider the model used by SQLite. Their product is open source, but I believe that some of their unit test framework, and their SQL integration-testing-engine are in-house only, and you have to buy or license them.
That means, anybody can rebuild their product and embed it anywhere, but not just anyone can rearchitect and rebuild it to suit their fancy. This gives a competitive advantage to the original author while making the most "important" piece open source.
Remember that some people will dislike any such move you make.
Secondly, as a do-not-imitate example; consider the model that was used (and I think not very well) by PyQT. If you put commercial restrictions on the open source "community edition", eventually the community will leave and build something else.
Other bad examples include oracle's recent unpopular behaviour with respect to Hudson and Open Office, two of its Sun/Java related acquisitions. These lead to forks, and a lot of lost goodwill for Oracle.