Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

As per the title of this question, for extremely performance critical situations, is storing a file's metadata (e.g. location, size, download on, etc) in a database going to allow for better performance than attempting to get it from the file system itself? Have there been any case studies into this problem?

To provide a bit more detail on a specific situation, the application needs to mirror terabytes of data (hundreds of files) between a remote site on a continual basis and the current program architecture uses Unix commands (i.e. ls) to determine which files needed to be updated. The file themselves are split between Isilon IQ clusters and Sun Thumper clusters which I have been told good throughput but poor metadata performance. As the application will be the only process to have write permissions to the files we aren't concerned with things getting out of sync, but we are concerned with performance as it currently takes between six and ten hours to transfer the data.

share|improve this question
add comment

3 Answers

up vote 3 down vote accepted

For actually getting an individual file's meta data I would not expect much difference, and, it would very much depend on which database went head to head with which file system and how well either was configured.

However if you say wanted to search for files with a ".mp4" suffix or all movies > 1GB then the database will win hands down. Even if the file systems index was organized to be efficiently searchable the normally available POSIX APIs would limit you to searching sequentially through a directory. If you have distributed your data over several file systems and needed a separate search on each "leaf" directory.

However this may not be the case for much longer as there are several projects (including one from Google) which are actively working on file systems with searchable meta-data

share|improve this answer
    
The metadata is going to be more relevant for aggregate searches over the files as opposed to doing a stat operation on a given file. –  rob Oct 13 '11 at 13:06
add comment

Works great. Right up until something gets out of sync and then you're left with having to do a full scan anyways, and in the meantime with any little quirk in your systems you'll do yet another full scan just to rule out sync issues.

If this is one master system sourcing the data can you monitor file system activity? If you can detect changes instead of scanning the entire system you could create a baseline (using your DB, or even a flat file manager) and then update based on recent activity.

Can you get reports from the data source(s) that modify the files? Instead of having to detect changes after the fact you could queue up lists of files to watch for changes to complete and only update those.

share|improve this answer
    
The files tend to run several megabytes a piece and may or may not be compressed so diffs are not a viable option. Likewise, locally file updates are a non-issue as access is controlled and updates are performed progamatically. –  rob Oct 12 '11 at 16:27
add comment

Have you considered having a daemon or cron job run on the file server and collect the relevant information and put it in a separate file (or pipe)? That gives you your list of files to process, without having to waste a lot of time assembling and groveling over directory listings before you can start the real work.

Regardless, I don't think the speed difference between reading a file and querying a database will be significant. Keeping the information in a database might be useful for other reasons (e.g., process auditing), but I don't think it'll help your performance.

share|improve this answer
    
There is a little bit more involved than just doing a basic ls on the directories involved plus once you start saving information in a flat file you might as well go the database route if you need to process the data in any way. In terms of performance, changing things will have an impact it's more of question of what that impact will be though. This isn't exactly a standard operating environment and file servers are optimized for throughput as opposed to interactivity, happens once you start dealing with petabytes of data. –  rob Oct 13 '11 at 13:50
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.