tl;dr - Sounds like it's time to step up to the big leagues. Putting lipstick on a pig doesn't make it any prettier, unless you're into that sort of thing...
The people problem
The first issue is commit synchronization. IF you have multiple people working on the same code at the same time you only need one rule to prevent problems:
Rule 1: Always pull before you merge/rebase
When it comes to DVCS, it's hard to make changes to a remote branch (ie the main repository) and very easy to make changes to the local. Every person is responsible for making their own code additions fit into the greater whole without issues. Unless 2 people commit at the exact same time, you shouldn't experience. Commit access to the origin/remote master should be limited to only a few developers and they should pull changes from the other developers via remote tracking branches.
The code problem
How do you know that the changes you make don't break code?
Simple answer... Write tests to prove they don't. If you ignore the TDD (Test Driven Design) school of thought, the whole point of tests is to add a level of verification that enables you to change code without breaking it.
Rule 2: Don't make assumptions, write proofs (ie tests).
In addition to this, the full gamut of tests should be run before you push to the origin/remote master.
Keep your commits as small and concise as possible. That way, if you need to back out a change that broke something later on, you'll save from having to re-implement the parts that didn't break the code.
You may need some organizational re-structuring first
If the above solutions can't easily be applied, there are probably some issues with the development structure that need to be addressed first.
The owner of the project should be the gatekeeper. If there are commit syncing issues, there are probably too many people with commit access. Even on massive projects like the Linux kernel, only a handful of developers have commit access to the origin/remote master repository. There are actually multiple levels of repositories to manage commits. Instead of a single layer commit model where everybody is pushing their changes to the origin, the hierarchical model has gatekeepers that pull changes and verify their quality before inclusion into the project. The hierarchical commit model can scale a lot bigger and more effective than the single layer model without sacrificing quality.
For the devs that don't get commit access, they should learn to create their own remote tracking branches (git and gitorious are good for this) so the devs who do have commit access can easily pull/integrate branches into the origin. If the changes are small, patches will work just as well.
The ability to pull changes before doing a merge/rebase assumes that you're not developing on your local master branch. The easy way to handle this is to make an initial pull before you start to code, then do all your work on that branch. The hard way is to branch it just before merging and roll back the master.
Define the coding style for the project overall and make the devs follow it. Contributing devs should be writing code that conforms to the project's standards/norms to minimize cleanup. Coding style can be a big ego barrier in an open project. If no standard is set, everybody will code in their own preferred style and the codebase will get very ugly very fast.
The myth of "The Mythical Man Month"
Believe it or not, the larger/more successful open source projects aren't run like a democracy. They're run as a hierarchy. Stating that a project can't effectively grow beyond 8-10 developers is naive. If that were true then mega-projects like the Linux Kernel wouldn't exist. The deeper issue is that giving everybody commit access just makes effecive communication too hard to handle.
The problem of the mythical man month can actually be overcome. You just need to run your project like the military. There are many levels within the hierarchy because it's common knowledge that individual people are really only effective at managing communications with a handful of people. As long as no single individual is responsible for managing the work of more than 5-7 people, the system can scale indefinitely.
It may limit the best/experienced developers to doing more integration and higher-level design/planning but that's not a bad thing. Part of scaling up is making the move to decide that the project needs a long-term plan. The people at the highest levels who have the greatest investment (time is also a resource) in the projects future should be charged with making the big decisions.
It's nice to hear about an open source project going through growing pains. Congrats and good luck.