Most eCommerce systems do everything but make mounds and mounds of julienne french fries.
The first factor really is available talent pool, i.e. if you are .Net shop, then php or Ruby solutions are probably not the best way to go as your developers will face a pretty steep learning curve (that you will be paying for over and over).
Pick the top 3-5 rated systems and put together a list of the features each provides, then compare that list to what is needed for your project. Not only will that help eliminate some of the candidates, but it will give you a better idea of the actual overall scope.
Of the systems remaining, there are probably only 1 or 2 truly viable alternatives. Look at those and identify the areas that would need customization and put together rough estimates for that. Be sure to include extra time for fighting with their API/Controls/styling.
By now you should have a good feel for the effort needed to implement the desired features so put together a rough estimate for writing it in-house - be sure to include extra time for debugging. Don't forget the Admin pages - someone has to maintain that product catalog!
That should give you a pretty good high-level view of your situation and be enough to make a reasonably informed decision. My bet is that off-the-shelf/Open-Source will win out unless you have pretty simple needs.