Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm self-taught and I don't have a CS degree. The more I've been learning about data structure, the more I wonder, in this day and age, how are we still saddled with the filesystem, with directories and files, as the basic data storage structure on the OS?

I understand the simplicity of it, but it seems nowadays that there could be more options available natively. As far as I'm aware, the only project to improve the basic functionality of the filesystem was ReiserFS, where you could tell what line of a file was changed by whom, and when.

For instance, if I could have native tagging for files, where I could tag images, diagrams, word-processing documents, an entire code repository, all as belonging to a single project, that would really be helpful to me. Since I'm stuck in the filesystem paradigm, I know that I could put all those into a single folder/directory, but what if they already exist in disparate directories, and they need to stay there? I know there are programs out there that can do this, but why aren't they on the filesystem?

Something that would be nice to have is some kind of relational feature in the filesystem, like you get with RDBMSes. I understand that that was supposed to be part of Vista/7, but that fell off the feature list too.

Sure, any program can store a binary file and have any data structure it wants in it, by why couldn't the OS offer more complex ways of storing data, beyond the simple heirarchy of the filesystem?

share|improve this question
    
This should probably be on SuperUser –  Malfist Mar 16 '11 at 20:36
2  
The core of it should be simple. The optional bloat you mention should go on top of a simple core. Alternatively, wait two decades and someone will reinvent the notion of a file system. –  Job Mar 16 '11 at 20:42
3  
"what if they already exist in disparate directories, and they need to stay there?" Sometimes you can use hard-links to solve this problem... –  FrustratedWithFormsDesigner Mar 16 '11 at 20:43
1  
Also, some interesting reading on the topic: c2.com/cgi/wiki?FileSystemAlternatives –  FrustratedWithFormsDesigner Mar 16 '11 at 20:48
3  
Not really a solution in Windows 7 but the new Libraries can give you some of the functionality you seem interested in: lifehacker.com/#!5464350/… –  DKnight Mar 16 '11 at 21:19
show 3 more comments

6 Answers 6

up vote 17 down vote accepted

Start with this: http://en.wikipedia.org/wiki/Unix_File_System

Read this: http://www.unix.org/what_is_unix/history_timeline.html

Then read this: http://www.amazon.com/UNIX-Filesystems-Evolution-Design-Implementation/dp/0471164836

There's a simple answer to "why couldn't the OS offer more complex ways of storing data, beyond the simple heirarchy of the filesystem?"

Because it's too much for the OS to do.

That's what libraries and application packages are for.

Oracle, for example, will sell you a file-system-like set of features that you manage with the Oracle toolset.

Python uses the DBM library to create very sophisticated on-disk storage structures.

CouchDB and Mongo (and others) are very sophisticated storage structures that offer some database-like features.

The point is that the OS should do the minimum and everything is an add-on.

share|improve this answer
3  
Quite agree. Actually, a lot of what OP was asking for is present in the eaither dead or dying WinFS project: en.wikipedia.org/wiki/WinFS. As much as the geek in says, 'Neat!' the experienced user and software engineer in me says, "Trying way too hard!" –  Adam Crossland Mar 16 '11 at 21:03
4  
"The point is that the OS should do the minimum and everything is an add-on." Quite a bold statement in an age where some operating systems contain a built-in windowing system, file indexing service, media player, remote desktop, firewall or Netris. –  biziclop Mar 16 '11 at 21:23
    
@biziclop: Agreed. Windows has diverged from the Linux viewpoint. Nothing surprising there. –  S.Lott Mar 16 '11 at 21:26
1  
@S.Lott Don't get me wrong, I agree with your approach, but Windows is saddled with so much useless rubbish anyway, one extra feature won't make a difference. :) –  biziclop Mar 16 '11 at 21:39
3  
That is the Unix philosophy. It is not necessarily right. It does (and a C complier) make Unix easy to port to hardware. It also makes it simple enough for people to clone Unix into the flavours of -ix likes we find today. If a feature is useful, and all programs need it, like say, spell checked input fields, then there is value in having the runtime environment provide it. We don't need 400 independent versions of a ribbon bar. –  Tim Williscroft Mar 16 '11 at 23:25
show 16 more comments

I don't see where (at least some) current file systems really need to do much [Edit: anything, to be honest] to support tags. When you get down to it, supporting tags means little more than some extra data associated with a file, but isn't written into the stream of bytes for that file.

NTFS (to choose one example that's in wide use) can do that just fine: as far as NTFS cares, a file isn't necessarily a single stream of bytes. On NTFS you can associate an arbitrary number of streams of data with a single file name. Each file has a (possibly empty) "primary stream" that doesn't have a name. It can also, however, have an arbitrary number of other streams, each of which must have a name. Using this, it would be truly trivial to add a stream named (just for example) "tags" to an existing file, and (obviously enough) write your tags to that stream.

After that comes the somewhat more difficult part: getting your tools to make use of the tags you put there. Ideally, you'd probably want to index them for fast searching, so you'd be able to do things like create a "virtual directory" of all the files with a specific tag.

At least from my perspective, the file system already has what's needed though -- it's supposed to store and retrieve the data, and it can do that perfectly well right now. Making use of that data is the job of other tools. Those tools don't currently exist, but the file system infrastructure to support them does.

If I'm allowed to be cynical for a moment, I'd say it was inevitable that this feature of NTFS would remain almost completely ignored and unknown. After all, it's simple to use and doesn't require any special API or anything else. You can use it quite nicely in completely portable C, C++, or anything else that will let you specify an arbitrary file name. Here's a quick bit of code to demonstrate creating a file with an AFS:

#include <fstream>

int main() {
    std::ofstream out("test.txt");
    std::ofstream tag("test.txt:tags");

    out << "This is the output file";
    tag << "tag1 tag2";

    return 0;
}

And, here's some code to read and display the tags:

#include <fstream>
#include <iterator>
#include <iostream>
#include <string>

int main() { 
    std::ifstream tags("test.txt:tags");

    std::copy(std::istream_iterator<std::string>(tags),
          std::istream_iterator<std::string>(),
          std::ostream_iterator<std::string>(std::cout, " "));
    return 0;
}

All very simple and easy. Note that although I've only written a trivial bit of data there, you can treat an AFS just like any other file -- all the usual "stuff" works just like with anything else. In a normal directory display, all that will show up is the primary stream (e.g., the size shown for the file will be the size of the primary stream), but if you want to see it, dir can show information about alternate streams as well with the /R flag. For example, a listing for the file created above looks like this:

03/16/2011  08:22 PM                23 test.txt
                                     9 test.txt:tags:$DATA
               1 File(s)             23 bytes
share|improve this answer
1  
DIR might be able to show it, but backup of a file with alternate streams is horribly difficult, especially to some other system. For example most NAS drives today use Linux, and the file systems there don't handle alternate streams at all. Copy the file over... and all the alt stuff just disappears. –  quickly_now Mar 17 '11 at 3:04
    
Yes, I have noticed that most NAS systems are rather...challenged (and this isn't the only way either). For actual backup and restore kinds of things, it doesn't cause a problem though (at least if the software in question is competently written at all): BackupRead will serialize all the streams, and BackupWrite will reconstitute the file (with alternate streams) from the serialized format. –  Jerry Coffin Mar 17 '11 at 4:14
    
Depends if you want the backed-up files to be directly readable on the NAS. If you do (and avoid the need for special restore programs) then you are stuck with plain-ole files. –  quickly_now Mar 17 '11 at 8:08
add comment

There's a little truth in every answer here but I don't think it's the whole truth.

What you list are mostly features that are sorely missed every day by users and developers alike.

People don't understand the tree-based file system any more than they would understand a DAG-based one.

And there's absolutely no excuse for the pathetic appendages of file names called extensions. They are not only completely unsuitable for their purpose (identifying the file type) but also an endless source of nuisance for users.

The reason we're still using them is a mixture of a "that'll do" attitude and the real need to maintain compatibility with older code. A new approach to storing files would mean a radical change in basic file I/O API, rendering most existing code useless. Either that or you have to tiptoe around them, maintaining the legacy API. Remember PROGRA~1.

I think for the reasons above, although the future could hold more specialised file systems for special applications, but while the desktop and laptop PC architectures of present survive, we're stuck with the largely tree-based file system with its lack of metadata and its horrible little extensions.


Now I'm going to switch sides.

Because it's all around us, we never really appreciate how mind-bogglingly powerful the tree metaphor is. On my hard drive I've got several hundred thousand files. If I have to find one, it rarely takes more than a minute, even if I know very little about the file. Now imagine the same task without any structure, just a flat list of names, scrolling endlessly.

Yet all the operations are straightforward, there's no spooky action at a distance, nothing that would make me go wtf.

Actually, I did implement a document store with rich metadata and a DAG-based hierarchy once. (It wasn't even a free-form DAG, it was strictly a two-level metastructure and the documents, which could be children of either a level 1 or a level 2 collection. So it's really simple.)

Obviously, the requirement that document names should be unique within a collection had to stay.

And then the problems started to flow. What if you open a collection and change the name of the document to something that clashes in a different collection the document also belongs to? We displayed an error message but the users were completely baffled. (These are the very same users who had asked for this requirement.)

They tried to delete a document, but all that did was remove it from the collection. So it still showed up in search results. We tried it the other way around too, but then they were complaining that they deleted a document from collection A and it magically disappeared from collection B. So we needed both an "unlink" and a hard delete operation.

Eventually we conceded defeat, fortunately still in time.

The extra search facets the metadata made possible worked an absolute treat though.

share|improve this answer
    
Rememebr CP/M on a 5 MB hard drive? Hundreds and hundreds of files scrolling past. AWFUL! –  quickly_now Mar 17 '11 at 3:01
    
@quickly_now Ah, the good old CP/M. :) –  biziclop Mar 17 '11 at 10:41
add comment

I've worked with non-hierarchical file systems on older minis like HP3000 and Encore/Gould. You didn't have directories; you had a group and an account, and files were named as "group . account . file", like "users.jbode.myfile1", "dev.jbode.main", etc.

Now, these are old systems, where individual disk space quotas were in the single megabytes, so it's not like you needed too many levels to organize your stuff, but from a user's and programmer's perspective hierarchical systems are much nicer.

share|improve this answer
add comment

To be honest, I barely touch metadata on my files on the Mac. I think in the last 5 years of using OSX (which supports comments and so forth), I've used metadata on maybe 2 files. Not saying it's a bad idea.

I'm just not sure how the overhead of tagging is pragmatic for me.

I think the nicest filesystem feature that I know of would be a filesystem level versioning system... that works cross partitions. It was done on VAXen in the 70s and early 80s, not sure why it didn't catch on with Unix and NTFS/Windows.

share|improve this answer
    
Modern versions of NTFS/Windows do offer versioning. It's not exactly in-your-face, but it does exist. Can't say how it compares to VMS though. –  Shog9 Mar 17 '11 at 4:45
add comment

The short answer is : Everyday people understand the file system. It reminds them of a file Cabinet. Think about web pages and even Fat apps, why do you think Tabs are so popular? People can identify with them, and understand them quickly.

Imaging trying to teach Grandma to search a DB for File based on property tags.. With the file system, Grandma knows the file is simply where she Put it.

Even with WinFS I don't think MS was going to get rid of the file system look and feel.

share|improve this answer
8  
I have to disagree with this. Most people who aren't forced to navigate the filesystem don't do it. They open a word processor and click their recent document, or search in Windows 7's start menu, etc. And lots of people lose track of where they put their files. It would be a lot easier for Grandma to search "cookie recipes" or "grandson photos" or whatever than to maintain a folder hierarchy. –  Matthew Read Mar 16 '11 at 20:55
1  
This assumes she was properly tagged her files.. What if she just wants to "Look for them"? The File system acts and menu sub-menu system as well. –  Morons Mar 16 '11 at 21:01
13  
This might come as a shock for you: everyday people don't understand the file system. They haven't got the faintest of ideas. And I don't mean a Unix style FS with its mount points, symlinks and hardlinks, but a bog standard directory structure with files in it. –  biziclop Mar 16 '11 at 21:26
1  
@Morons, my grandmother never knows where she puts things. Gmail already has shifted my desired paradigm to a tagging system, especially with filters to automatically tag things. I think the filesystem paradigm was implemented largely due to the simplicity of programming tree-structures. It also makes addressing easier from a programming perspective. How would you specify a document's location in a tag-based system? Not saying it can't be done, but the details need to be ironed out. –  zzzzBov Mar 16 '11 at 21:26
2  
I have a CS degree. But I don't know into which folders any Windows does put what files into. Especially Desktop, StartMenu, QuickLaunch, and all the other user/system specific default folders. (That M$-Help system does not help in explaining to me how to press a button.) I need to install CygWin to be able to search for my own files, because newer M$ search features do not find simple existing files anymore like on win2k. Disabling misfeatures like hide-system-files, hide-file-extensions don't solve most problems anymore. I gave up Windows, when I was forced to work on the (brand new) winXP. –  comonad Mar 17 '11 at 3:54
show 5 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.