Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

The log files in question are metrics log files. One of them is a project file produced by SourceMonitor, to which I regularly add checkpoints to chart the progress of projects.

share|improve this question
2  
The answer to such question depends on the goal and the usage on the long run of such information. I think having too much information stored is like having no information at all, it is not useful at all. –  Amine Aug 28 '12 at 14:33
    
@Amine That's a totally different question. Developing a good way to store and manage the data is different than developing a retention strategy. Yes, you need both, but once you have a good way of organizing and accessing your data, you can determine if and when you have too little or too much to deal with. –  Thomas Owens Aug 28 '12 at 14:37
add comment

8 Answers

I wouldn't store them in version control, at least in the same project as the source code for the deliverable product. To me, it just doesn't make much sense to combine what's effectively logging data in a directory structure for a deliverable product. The repository used for the project shouldn't contain anything outside of what is needed to build and deploy/deliver the system at hand (including instructions for building and deploying).

I would recommend storing them in the same place or using the same methodology as you do your other documents, such as test result history, design documents, requirements specifications, and so on. I don't know what technology you use, but uploading them to a SharePoint server would work, as would uploading them to a shared data server and putting the path to them (and perhaps summaries of them) on a wiki page.

share|improve this answer
    
same place as other documents... yup, in the SCM :) –  gbjbaanb Aug 28 '12 at 14:37
2  
@gbjbaanb Why are you storing documents in the SCM? There are often much better tools, especially when you're dealing with binary files (Word docs, PDFs, images, the output of modeling tools). Often, a file server and a wiki are far more useful for documents than SCM. –  Thomas Owens Aug 28 '12 at 14:39
add comment

Nowadays storage is cheap. If these are metrics that you can use to show performance increases that coincide with versions, then I'd say keeping them in source control is just fine. Just make sure that you're storing them for a reason. If you can regenerate these logs/performance files from the source (meaning I can check out the code, run it, and produce the exact file you're about to store), then you probably don't need to.

Lots of things are stored in version control these days, including any and all documentation and other files that change along with the version of the project/code.

Note: You would store these in a branch outside your code branch. Like Thomas' answer states, the source code branch that is used to build/deploy the app should be be separate so that your build server (or you, if you don't have time/space) can download/checkout the code independently of the documentation.

share|improve this answer
add comment

I should not add log file in version control, they are temporary and static so you don't have to maintain the version of it. You also collect the results of the log files into a chart, so I don't think it's necessary to add the log files into version control.

share|improve this answer
2  
I disagree that they are "temporary". They might not be required for the life of the project (perhaps keeping metrics through a specific window in time is sufficient), but charting or graphing or anything similar is often a distillation of data - depending on what is indicated, it might be necessary to actually delve into the raw data to gain useful insights into the behavior of the system. –  Thomas Owens Aug 28 '12 at 14:36
    
Good point Thomas, indeed if you want to use the data to create some other overviews or analyse it at a later point, then it's useful to store the log files. –  Erwin Aug 28 '12 at 14:40
add comment

IMO, no, it's place in archived backups, if you wish to have that data later. General rule I use, store only things that you make youself or some others, if you say, include source code of other tool. Any files produced by codegen tools, logs, any other output data should be left out of repository, or stored separetly. The point is that, usually, you can checkout required version and recreate that data at any moment. The are exceptions, I suppose, but I have not encountered them so far.

share|improve this answer
add comment

Get your build server to gather your metrics and store them there against each build.

There should be some stuff that you only need the latest version of and some stuff that you need permanently to graph improvements. For example, if you build a whole code coverage report every build, you only ever need to look at the latest report to decide where to focus your test-writing efforts. But it is useful to keep code coverage %, branch coverage %, etc for every build and graph it out.

Jenkins is very good for managing this kind of thing. The HTML publisher plugin can be made to publish temporary, current reports (or you can keep them for n days, if you have the space and need), while the plot plugin can be made to track long-term progress.

share|improve this answer
add comment

You can also store them in a NoSQL database, then you could write an interface to get data reporting. I think NoSQL should be suitable for this.

share|improve this answer
add comment
  • Are they files that change over time?

  • Is it helpful to be able to retrieve older versions?

  • Would losing these files be a problem?

If you answered 'yes' to one or more of the above, then sure, go ahead and store them in a version control system. Since you answered 'yes', there's obviously some benefit to be gained by doing so, and little reason not to.

The best argument I can think of against using version control for these files is that log files tend to be linear in nature -- the only way you normally modify them is to append new data to the end of the file. Given that, they have a sort of primitive change tracking mechanism built in, so the added complexity of using version control may not be necessary.

share|improve this answer
add comment

Overview

It's a good idea to back up log files, but they do NOT belong in source control!

Good Places For Backups

  • Burn them encrypted to DVD once a month and pop the DVD in a safe deposit box off-site.
  • In zip files on a secure drive in a separate secure server room.
  • Maybe in a secure NoSQL database?

Security

Log files often contain confidential information. Users can paste their password into the wrong field and have it end up in the logs. All kinds of information ends up there that's probably covered by Safe Harbor and other international privacy regulations even if it's not otherwise considered private by your customers. You want to limit the number of people who have access to this stuff and generally all programmers have access to source control.

Size/Speed

Source control is designed for little 100K-5M source files. Not Gigabytes of logging junk. Depending on the source control tool you use, you may find yourself waiting for weeks just to get the latest version of your code if you store huge logs in there.

Features

Source control is designed to compare versions. There is no version comparison like this with log files. If you need to reference these files quickly and constantly, then log table in a database (either SQL or NoSQL would work) is the way to go. Version control is just not a good feature-fit for this purpose.

Tools

  • Gnupg is great for compressing and encrypting huge things. Use this on the files on the server where the files are generated. They will transfer faster and be secure when they travel over the wire.
  • Linux/bash has a logrotate tool, but most logging tools have a logrotate feature that can keep sixty 100Meg files in your log folder just by renaming them .1, .2, .3, etc. Check how big a file your favorite editor can handle and keep each file a little smaller than that. Presumably your server drive is backed up and these files with it, but when you upgrade servers (e.g. for an OS upgrade) remember to put a copy of these files somewhere else and restore them to the new server when you are done!
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.