techniques to organize and store files with their data on a computer.

learn more… | top users | synonyms

44
votes
2answers
170k views

How to detect the encoding of a file?

On my filesystem (Windows 7) I have some text files (These are SQL script files, if that matters). When opened with Notepad++, in the "Encoding" menu some of them are reported to have an encoding of ...
40
votes
2answers
3k views

What is the benefit of writing to a temp location, And then copying it to the intended destination?

I am writing an application that works with satellite images, and my boss asked me to look at some of the commercial applications, and see how they behave. I found a strange behavior and then as I was ...
27
votes
11answers
2k views

Why are there so many competing filesystem designs? [closed]

Just a quick question, but why are there so many file systems still competing and in use today? (ntfs, fat32, ext3(ffs), etc) It seems that file system designers could agree upon the best aspects of ...
21
votes
7answers
3k views

Why is filesystem preferred for logs instead of RDBMS?

Question should be clear from its title. For example Apache saves its access and error logs in files instead of RDBMS no matter on how large or small scale it is being utilized. For RDMS we just have ...
17
votes
4answers
2k views

Why is the Git .git/objects/ folder subdivided in many SHA-prefix folders?

Git internally stores objects (Blobs, trees) in the .git/objects/ folder. Each object can be referenced by a SHA1 hash that is computed from the contents of the object. However, Objects are not ...
8
votes
4answers
609 views

What exactly does it mean that storing “large blobs in the database reduces performance”?

To someone who knows database internals this may be an easy question, but can someone explain in a clear way why storing large blobs (say 400 MB movies) in the database is supposed to decrease ...
8
votes
4answers
608 views

File shredder algorithm

As a part of learning system programming, I am looking to implement a file shredder. The simplest way (and probably seen as naive) would be to replace the data bytes with zeroes (I know OS splits the ...
8
votes
3answers
312 views

Examples of limitations in IT due to different bit length by design [closed]

I am teaching the course "Introduction in Programming" for the first-year students and would like to find interesting examples where the datatype size in bits, chosen by design, led to certain known ...
7
votes
3answers
5k views

Lowercase in Linux file names

As I find that UpperCase are really readable for first letter word separation in long complex names, I tend to give some of my Linux files names with some UpperCase. Mostly executables, some ...
6
votes
2answers
253 views

Is there a fundamental reason for keeping file systems and versioning systems separate?

I've done some work looking at Next3 and ext3cow, and a few other file systems that provide snapshots and/or file versioning. It seems a bit strange to me that file systems only go part-way with ...
6
votes
4answers
7k views

Is it safe to convert Windows file paths to Unix file paths with a simple replace?

So for example say I had it so that all of my files will be transferred from a windows machine to a unix machine as such: C:\test\myFile.txt to {somewhere}/test/myFile.txt (drive letter is irrelevant ...
6
votes
5answers
7k views

How to Model a simple file-system by UML class diagram

I want to model a file system which contains both files and directories, and directories can contain either files or other directories. This is what I have reached so far: In OOSE book, however, a ...
6
votes
3answers
2k views

How many threads should access the file system at the same time?

We have a module in an application which stores data in multiple files and multilevel directories and access them from multiple threads (both reads and writes). The directory structure is based on a ...
5
votes
4answers
2k views

Why is Tortoise SVN case sensitive?

I've recently encountered this using TortoiseSVN, but I assume it will be the same for CVS based programs (correct me?). Out of pure curiosity, is there any reason why the CVS filesystem is ...
5
votes
1answer
361 views

Efficient data structure to implement fake file system

I want to implement a data structure that will hold the paths of directories, sort of fake file system. Input:- I have a text configuration file containing the paths as follows ... C:/temp1 ...
4
votes
4answers
9k views

Is it wise to store a big lump of json on a database row

I have this project which stores product details from amazon into the database. Just to give you an idea on how big it is: [{"title":"Genetic Engineering (Opposing ...
4
votes
7answers
1k views

How do i speedily traverse a file system while extracting/extrapolating various data and provide user feedback?

I'm working on a system file scanner that reveals info about various files (e.g. size, last used, duplicates, etc). Currently I'm traversing the file system once just to get a good measure of the ...
4
votes
1answer
3k views

File system implementation in MongoDB with GridFS

I am working on two projects that will both implement a Webdav server backed by a MongoDB GridFS. In each case, there is the potential for the system to store tens of millions of files spread across ...
4
votes
2answers
1k views

Is it a bad idea to sync file system with remote server using HTTP?

I have started a project which will duplicate Dropbox or Google Drive behavior but using Amazon S3 az a backend. Idea is very simple, a Node.js server that watchs a directory for file changes and PUT ...
3
votes
7answers
712 views

Why can we not insert into files without the additional writes? (I neither mean append, nor over-write)

This occurs as a programming language independent problem to me. I have a file with the content aaabddd When I want to insert C behind b then my code needs to rewrite ddd to get aaabCddd Why ...
3
votes
3answers
196 views

Filesystems that use logging: If you're writing the data in the log (on disk), and in the actual locations themselves (also on disk) then…?

Aren't you essentially writing the same data twice, to the disk? Doesn't this cause a slowdown of a factor of ~2x? What optimizations can be made to minimize the cost of having to write things twice? ...
3
votes
3answers
182 views

What's the point of hidden files?

What is the point of hidden files? In Microsoft Windows they exist, in Mac OS X they exist and in Linux they exist. It seems to me that it just makes detecting malware more difficult. The only upside ...
3
votes
1answer
144 views

Filesystem superblocks and their backup copies

I'd like to understand how (modern) filesystems are implemented and having trouble to fully understand superblocks and their backups. I reference ext4 and btrfs, but the questions may also apply to ...
3
votes
3answers
292 views

Storing lots of large strings with frequent “appends” and few reads

In my current project, I need to store a very long ASCII string to each instance of a given object. This string will receive an 2 appends per minute and will not be retrieved so frequently. The ...
3
votes
1answer
154 views

What is the most sensible design for making files available for download from a URL?

This is what I need to do, in a nutshell: Generate Excel spreadsheet files (programmatically). Store these .xlsx files in a location where they can be accessed by users later. These files need to ...
3
votes
1answer
243 views

How to store the file names, start offset and length while avoiding the issue of self imposed limits (lookup table) or having to scan the entire file?

I am attempting to learn more about C and it's descendants(C++ mainly). I have decided that I would like to create a "file system" of sorts. Not a particularly advanced one mind you but something to ...
2
votes
5answers
474 views

Is it plausible to use a filesystem-based configuration format rather than an INI file?

After having had endless issues with INI-style configuration files and parsing them correctly (let along getting binary data right), a crazy idea crossed my mind recently: What if an INI file like # ...
2
votes
7answers
4k views

Automatically delete files after they expire

I've got this idea for some time and I was wondering if anyone has seen such a feature/app in any operating system and if you haven't, what do you think about it. Where do you think I should begin? ...
2
votes
3answers
226 views

Is there an established convention for separating Windows file names in a string?

I have a function which needs to output a string containing a list of file paths. I can choose the separation character but I cannot change the data type (e.g. I cannot return a List<string> or ...
2
votes
2answers
377 views

Why is it not standard to implement abstraction layers for the file system?

I have been taught to access databases through abstraction layers. I was wondering why it is not also standard practice to access the file system through an abstraction layer? It seems to me unit ...
2
votes
4answers
283 views

Where can I learn about how programs handle file systems like NTFS?

I'm writing a program to handle the xbox 360's STFS files. I'm encountering all sorts of problems about how to keep track of used vs unused blocks, reading non-sequential files, etc. These aren't so ...
2
votes
4answers
688 views

Can memristors change programming paradigms? In what way?

Think about persistent storage that can preserve pointers: among other things there is no need to serialize/deserialize data. For example, you typically build a tree structure and then serialize it ...
2
votes
4answers
1k views

Best practice to sync long paths to filesystems where path length is limited?

I wrote a sync tool to synchronize folders/files from Alfresco to Windows. PROBLEM: In Alfresco, /files/can/have/very/very/very/long/filepaths/like/this.txt The Windows API prevents me from creating ...
2
votes
2answers
405 views

Why must directories be empty before being deleted?

As far as I know, deleting a non empty directory could work the same way as deleting an empty directory: by removing the pointer to the directory's metadata there would be no pointers to the items it ...
2
votes
1answer
1k views

NoSQL as file meta database

I am trying to implement a virtual file system structure in front of an object storage (Openstack). For availability reasons we initially chose Cassandra, however while designing file system data ...
2
votes
1answer
2k views

Directory structure (file system design)

I was looking at how file systems are designed and noticed that most places say that the directory hierarchy can be implemented using a hash table. Could someone please explain me how using a hash ...
2
votes
1answer
364 views

HTML5 file API or Java bridge to acces to local Files?

I've to access to files on the software user and I don't know if it's better to use a full JS app with HTML5 File Api rules or use Java and communicate with it ?
2
votes
1answer
1k views

How to organize my site's file system properly?

Doing some reading on Stack Overflow, I've found a lot of information suggesting that proper organization of a file system is crucial to a well-written web app. One of the key pieces of evidence is ...
2
votes
1answer
99 views

concurrency semantics reading a file being written (ext)

In Ext 3/4 filesystem, if I have a log file open and being appended to, if another process tries to read it (e.g. with cat), what happens? Can the reading process read part of / an incomplete write? ...
2
votes
1answer
271 views

Data structure well suited for duplicate entries

I'm in the process of getting to know (modern) filesystems. As part of it, I came across log structured filesystems that also handle allocations in a log structured way. I wonder how they handle ...
1
vote
2answers
200 views

Why do filesystems read and write in blocks?

I read that file systems usually access data in blocks, whose size is integral multiple of disk block size. Why don't they read individual disk blocks?
1
vote
2answers
150 views

How are non-folder files called? [closed]

Maybe it looks like a weird question, but what term should be attributed inside the code for files that are not folders to differentiate them from folders? If I need write 2 functions isFolder() and ...
1
vote
6answers
196 views

Ways of marking changes being made to a file

I am looking for a lightweight way to notate changes made to a file. The obvious answer to this is a version control system. But this is more robust than necessary for what I would like to do. All I ...
1
vote
2answers
3k views

Finding duplicate files? [duplicate]

I am going to be developing a program that detects duplicate files and I was wondering what the best/fastest method would be to do this? I am more interested in what the best hash algorithm would be ...
1
vote
2answers
54 views

Can version control systems use the filesystem log to capture changes?

I was trying to find a "perfect" syncing program between a network share and a local folder, when I realised that it's probably impossible to do it right unless all the filesystem operations were ...
1
vote
2answers
216 views

Avoid data manipulation by user

I have a C# program (could be any programming language) that saves data to a file on a memory (hard disk, USB drive, etc.). The program uses this data for monitoring its operation time, but it could ...
1
vote
1answer
439 views

Watching file changes/additions/removal, but with an eye on partial transfer

I would like to monitor the filesystem in python, so that my application gets warned of the new file addition, file removal, or file change. Once the file is detected, the application starts ...
1
vote
0answers
291 views

What database structure is suitable for tracking File audits?

I need to track permissions and access requests to a file server in a database. I'm given the full path of the folder and am considering parsing the path (splitting on the "/" character) and creating ...
0
votes
4answers
174 views

Why is there little use of filesharing as compression (outside of libraries)?

Recently I was looking for a program that will run as a daemon and find files that have the same size/type, check if they're the same, then make both a hard link to a single copy if they are. And I ...
0
votes
2answers
119 views

Is renaming an 'alias' for moving?

Is it true to say (on Windows and Unix\Linux\OS X) that renaming a file or directory is just an alias for moving? e.g. Are there any side effects to either which are not present on the other? Does ...