3

I'm looking into the feasibility of using off-the-shelf distributed SCMs such as Git or Mercurial to manage millions of XML files. Each file would be a commercial transaction, such as a purchase order, that would be updated perhaps 10 times during the lifecycle of the transaction until it is "done" and changes no more.

And by "manage", I mean that the SCM would be used to not just version the files, but also to replicate them to other machines for redundancy and transfer of IP.

Lets suppose, for the sake of example, that a goal is to provide good performance if it was handling the volume of orders that Amazon.com claimed to have at its peak in December 2010: about 150,000 orders per minute.

We're expecting the system to be distributed over many servers in order to get reasonable performance. We're also planning to use solid-state drives exclusively.

There is a reason why we don't want to use an RDBMS for primary storage, but it's a bit beyond the scope of this question.

Does anyone have first-hand experience with the performance of distributed SCMs under such a load, and what strategies were used?

Open-source preferred, since the final product is to be FOSS, too.

Improve this question
11
  • What exactly does "manage" imply, and is these files spread over multiple servers like Amazon do?
    – user1249
    Feb 27, 2011 at 19:58
  • Yes, they would likely be spread over multiple disks and servers. We're planning to be using SSD drives exclusively as well.
    – Chris Wenham
    Feb 27, 2011 at 20:01
  • 1
    Is there a reason then this is stored in XML files? Surely a transactional RDBMS would be more favourable?
    – James Love
    Feb 27, 2011 at 20:08
  • 1
    Nope, I still think it's a design problem. An RDBMS is far more suited to this than a bunch of XML files. And git or mercurial isn't designed to version this nonsense. You use it for hand-maintained things, not auto-generated content, just like any VCS.
    – Alex Budovski
    Feb 27, 2011 at 23:24
  • 1
    if you do this, i expect to read about it on the Daily WTF later
    – Steven A. Lowe
    Feb 28, 2011 at 2:21

4 Answers 4

Reset to default
4

I believe that aside from performance itself, which is actually quite impressive on both git and mercurial I would be more concerned with the long term issue of merges.

You can automate a number of tasks on DVCS (commits, pushes/pulls, updates, etc.) which include merges but there are always a few collisions that can't be "automagically" solved.

And with 150k/min transactions I would assume that even if only a very small percentage of merges require human intervention, it would become an issue over time.

With your exclusion of RDBMS; there are other more proper and scalable storage methods such as document-oriented databases which would suit your case. I would consider those first.

Improve this answer
2
  • So your concern is basically if there is any chance that two different servers update the same XML-file in an incompatible way causing a collision?
    – user1249
    Feb 27, 2011 at 22:24
  • I think this is the best reason to rule-out SCM for this task. Thanks.
    – Chris Wenham
    Feb 28, 2011 at 16:45
1

You should use a distributed database like mongodb in this case, DVCS won't give you that performance, and what is the point of manual replication? This mechanism is already implemented in major DBMS, why would you bother reimplementing it upon DVCS?

If you still want to stick to file systems, you can use versioned files systems, for example, wayback.

Improve this answer
0

How about Perforce? It is used in gaming industry for keeping both code and game assets, that can be pretty large.

Here you can find some guy comparing Perforce and Git.

Improve this answer
0

So - if I read correctly - you have transactions, and you need to version the operations on transactions.

Daniel makes a great point about the merge problems that could arise. You'd have to build your workflow to ensure that can't happen.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.