Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

A government agency publishes a text file with thousands of records. The entire file is about 60MB. Every day the file has about 60 new or changed records.

We need to validate some info against that file, and for auditing purposes, we may need to recover the file as it was in certain date.

I thought I could use a VCS to store the file using git, but I understand git stores the entire text file gzipped, so the 60BM file will still need about 30MB to store everyday version.

I wonder if one of the free VCS stores only diff/patch files.

share|improve this question
    
Are you sure git doesn't store diffs? That seems inefficient. I know subversion does, in any case. –  Blorgbeard Sep 19 '12 at 22:26
    
@Blorgbeard I re-read docs. Looks like it initially doesn't, but later it uses delta compresion. git-scm.com/book/en/Git-Internals-Packfiles –  MiguelM Sep 19 '12 at 22:47
    
I don't think you'll have performance or space problems with git, but the best way to find out is to try it. –  grossvogel Sep 19 '12 at 22:47
add comment

2 Answers 2

up vote 14 down vote accepted

It is true that Git stores only full revisions of entire files. However, it does not simply store them in the file system as individual files. Instead, it stores them in a pack file.

A pack file is, broadly speaking, every revision of every file of the entire history of the entire repository concatenated into a single file and then compressed. Because of the enormous amount of redundancy this compresses incredibly well, since compressors love redundancy.

This will not only compress different revisions of individual files (since revision n and n+1 will usually be mostly identical), this will even compress identical blocks of content in unrelated files, such as GPL license headers (which is something patch-based storage formats cannot do with the same ease).

In general, a Git repository will end up smaller than a Subversion repository, even though the latter only stores patches.

share|improve this answer
    
Great answer, thanks a lot. –  MiguelM Sep 20 '12 at 1:09
    
@MiguelM: this is misleading. Pack files do NOT contain complete redundant copies of every revision - revisions are saved as deltas. –  Michael Borgwardt Sep 20 '12 at 7:52
1  
And note that initially, each object is stored as separate files on the file system. It requires something like git gc (which gets executed automatically after a while) for git to pack things up. –  Ikke Sep 20 '12 at 11:21
add comment

I understand git stores the entire text file gzipped,

That's what happens immediately. But it frequently reorganizes this into sets of changes that are compressed.

I wonder if one of the free VCS stores only diff/patch files.

All of them do. Every single one. Even SCCS from 1972 did, and probably others before it.

share|improve this answer
    
Thanks. It looks like I misunderstood a description of how git stores objects. It does store the whole file zlib compressed (loose object), but a git gc and other process trigger a process to use delta compression and save space git-scm.com/book/en/Git-Internals-Packfiles –  MiguelM Sep 19 '12 at 22:40
    
It's sets of changes that are compressed, not individual file revisions I thinks this edit needs clarification when applied to git, check link in comment. –  MiguelM Sep 19 '12 at 22:54
    
@MiguelM: you're right, I was only describing the end result, clarified that. –  Michael Borgwardt Sep 20 '12 at 7:49
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.