techniques to organize and store files with their data on a computer.

learn more… | top users | synonyms

7
votes
5answers
7k views

Is it safe to convert Windows file paths to Unix file paths with a simple replace?

So for example say I had it so that all of my files will be transferred from a windows machine to a unix machine as such: C:\test\myFile.txt to {somewhere}/test/myFile.txt (drive letter is irrelevant ...
2
votes
3answers
200 views

Why would anyone want to build a file system for windows? [closed]

I saw an ad on StackOverflow today for a project called WinFsp. The site mentions the following features: Allows for easy development of file systems in user mode. There are no restrictions on what ...
2
votes
1answer
46 views

VBA Outlook: quickly find subfolder

I have the following structure in my Outlook Public Folders. -Public Folders --1001_RandomProject --1002_AnotherProject --1003_Yetanotherproject ... and so on, basically there's a couple of ...
-4
votes
1answer
65 views

Where should we put folders on the user's computer? [closed]

This reddit post makes a good point: Why do developers create so many folders directly in "My Documents". My home folder is completely full of this kind of stuff (especially when I show hidden files ...
3
votes
1answer
180 views

What is the most sensible design for making files available for download from a URL?

This is what I need to do, in a nutshell: Generate Excel spreadsheet files (programmatically). Store these .xlsx files in a location where they can be accessed by users later. These files need to ...
5
votes
1answer
395 views

Efficient data structure to implement fake file system

I want to implement a data structure that will hold the paths of directories, sort of fake file system. Input:- I have a text configuration file containing the paths as follows ... C:/temp1 C:/...
3
votes
3answers
189 views

What's the point of hidden files?

What is the point of hidden files? In Microsoft Windows they exist, in Mac OS X they exist and in Linux they exist. It seems to me that it just makes detecting malware more difficult. The only upside ...
1
vote
2answers
153 views

How are non-folder files called? [closed]

Maybe it looks like a weird question, but what term should be attributed inside the code for files that are not folders to differentiate them from folders? If I need write 2 functions isFolder() and ...
0
votes
0answers
102 views

File based communication vs Rest or RPC

My company make automation machines (with a PC inside and a custom built software to control it). So, for the purpose of this question, we can say that these are normal PCs. Sometimes we have the ...
1
vote
2answers
54 views

Can version control systems use the filesystem log to capture changes?

I was trying to find a "perfect" syncing program between a network share and a local folder, when I realised that it's probably impossible to do it right unless all the filesystem operations were ...
17
votes
4answers
2k views

Why is the Git .git/objects/ folder subdivided in many SHA-prefix folders?

Git internally stores objects (Blobs, trees) in the .git/objects/ folder. Each object can be referenced by a SHA1 hash that is computed from the contents of the object. However, Objects are not ...
4
votes
4answers
10k views

Is it wise to store a big lump of json on a database row

I have this project which stores product details from amazon into the database. Just to give you an idea on how big it is: [{"title":"Genetic Engineering (Opposing Viewpoints)","short_title":"...
3
votes
7answers
738 views

Why can we not insert into files without the additional writes? (I neither mean append, nor over-write)

This occurs as a programming language independent problem to me. I have a file with the content aaabddd When I want to insert C behind b then my code needs to rewrite ddd to get aaabCddd Why ...
1
vote
2answers
246 views

Avoid data manipulation by user

I have a C# program (could be any programming language) that saves data to a file on a memory (hard disk, USB drive, etc.). The program uses this data for monitoring its operation time, but it could ...
23
votes
7answers
4k views

Why is filesystem preferred for logs instead of RDBMS?

Question should be clear from its title. For example Apache saves its access and error logs in files instead of RDBMS no matter on how large or small scale it is being utilized. For RDMS we just have ...
0
votes
1answer
65 views

What is a simple way to let a user select a folder from a tree?

I have a Python-Flask app in which users can place files into a folder. As of now the directory structure is something like: /app /storage /templates . . . server.py The ...
0
votes
2answers
120 views

Is renaming an 'alias' for moving?

Is it true to say (on Windows and Unix\Linux\OS X) that renaming a file or directory is just an alias for moving? e.g. Are there any side effects to either which are not present on the other? Does ...
40
votes
2answers
3k views

What is the benefit of writing to a temp location, And then copying it to the intended destination?

I am writing an application that works with satellite images, and my boss asked me to look at some of the commercial applications, and see how they behave. I found a strange behavior and then as I was ...
1
vote
2answers
200 views

Why do filesystems read and write in blocks?

I read that file systems usually access data in blocks, whose size is integral multiple of disk block size. Why don't they read individual disk blocks?
0
votes
1answer
198 views

Best way to centralise a php project (except bitbucket)

The structure is as follows: A: this is the live/production website B: this is the staging, which is a copy of live C: this is the testing environment for designers 2 developer and 2 designers. ...
0
votes
1answer
281 views

Efficient way to map changes in a filesystem hierarchy

I'm currently working on a project that will enable file searching based on metadata found in the file. It'll be comprised of 2 parts: a filesystem crawler that passively scans for changes and ...
0
votes
1answer
35 views

Static analysis for finding capitalisation / case inconsistencies in file names

TLDR; I'm looking for ideas on how to flag code containing file names/paths that have inconsistent capitalisation with the actual file/directory. Situation I am migrating a significant code base ...
0
votes
4answers
175 views

Why is there little use of filesharing as compression (outside of libraries)?

Recently I was looking for a program that will run as a daemon and find files that have the same size/type, check if they're the same, then make both a hard link to a single copy if they are. And I ...
2
votes
1answer
282 views

Data structure well suited for duplicate entries

I'm in the process of getting to know (modern) filesystems. As part of it, I came across log structured filesystems that also handle allocations in a log structured way. I wonder how they handle ...
1
vote
1answer
463 views

Watching file changes/additions/removal, but with an eye on partial transfer

I would like to monitor the filesystem in python, so that my application gets warned of the new file addition, file removal, or file change. Once the file is detected, the application starts ...
3
votes
1answer
148 views

Filesystem superblocks and their backup copies

I'd like to understand how (modern) filesystems are implemented and having trouble to fully understand superblocks and their backups. I reference ext4 and btrfs, but the questions may also apply to ...
-2
votes
2answers
122 views

file quantity limit in a directory on a linux file server and why?

What is a good limit to use on the quantity of files in a directory, and why? EDIT: Why shouldn't someone create a system that puts hundreds of thousands of files in the same directory? Why I ask: ...
2
votes
4answers
1k views

Best practice to sync long paths to filesystems where path length is limited?

I wrote a sync tool to synchronize folders/files from Alfresco to Windows. PROBLEM: In Alfresco, /files/can/have/very/very/very/long/filepaths/like/this.txt The Windows API prevents me from creating ...
3
votes
1answer
248 views

How to store the file names, start offset and length while avoiding the issue of self imposed limits (lookup table) or having to scan the entire file?

I am attempting to learn more about C and it's descendants(C++ mainly). I have decided that I would like to create a "file system" of sorts. Not a particularly advanced one mind you but something to ...
2
votes
2answers
444 views

Why must directories be empty before being deleted?

As far as I know, deleting a non empty directory could work the same way as deleting an empty directory: by removing the pointer to the directory's metadata there would be no pointers to the items it ...
48
votes
2answers
188k views

How to detect the encoding of a file?

On my filesystem (Windows 7) I have some text files (These are SQL script files, if that matters). When opened with Notepad++, in the "Encoding" menu some of them are reported to have an encoding of "...
1
vote
0answers
311 views

What database structure is suitable for tracking File audits?

I need to track permissions and access requests to a file server in a database. I'm given the full path of the folder and am considering parsing the path (splitting on the "/" character) and creating ...
2
votes
2answers
390 views

Why is it not standard to implement abstraction layers for the file system?

I have been taught to access databases through abstraction layers. I was wondering why it is not also standard practice to access the file system through an abstraction layer? It seems to me unit ...
2
votes
7answers
4k views

Automatically delete files after they expire

I've got this idea for some time and I was wondering if anyone has seen such a feature/app in any operating system and if you haven't, what do you think about it. Where do you think I should begin? ...
8
votes
4answers
616 views

What exactly does it mean that storing “large blobs in the database reduces performance”?

To someone who knows database internals this may be an easy question, but can someone explain in a clear way why storing large blobs (say 400 MB movies) in the database is supposed to decrease ...
6
votes
3answers
2k views

How many threads should access the file system at the same time?

We have a module in an application which stores data in multiple files and multilevel directories and access them from multiple threads (both reads and writes). The directory structure is based on a ...
1
vote
2answers
3k views

Finding duplicate files? [duplicate]

I am going to be developing a program that detects duplicate files and I was wondering what the best/fastest method would be to do this? I am more interested in what the best hash algorithm would be ...
4
votes
2answers
2k views

Is it a bad idea to sync file system with remote server using HTTP?

I have started a project which will duplicate Dropbox or Google Drive behavior but using Amazon S3 az a backend. Idea is very simple, a Node.js server that watchs a directory for file changes and PUT ...
2
votes
5answers
475 views

Is it plausible to use a filesystem-based configuration format rather than an INI file?

After having had endless issues with INI-style configuration files and parsing them correctly (let along getting binary data right), a crazy idea crossed my mind recently: What if an INI file like # ...
6
votes
5answers
7k views

How to Model a simple file-system by UML class diagram

I want to model a file system which contains both files and directories, and directories can contain either files or other directories. This is what I have reached so far: In OOSE book, however, a ...
4
votes
7answers
1k views

How do i speedily traverse a file system while extracting/extrapolating various data and provide user feedback?

I'm working on a system file scanner that reveals info about various files (e.g. size, last used, duplicates, etc). Currently I'm traversing the file system once just to get a good measure of the ...
9
votes
3answers
5k views

Lowercase in Linux file names

As I find that UpperCase are really readable for first letter word separation in long complex names, I tend to give some of my Linux files names with some UpperCase. Mostly executables, some ...
0
votes
3answers
661 views

Which language is more suitable heavy file tasks?

I need to write a script (based on basic functions) to process /image/audio/video files. The process is mainly filesystem tasks and converts. The database of files has been stored by mysql. The script ...
3
votes
3answers
298 views

Storing lots of large strings with frequent “appends” and few reads

In my current project, I need to store a very long ASCII string to each instance of a given object. This string will receive an 2 appends per minute and will not be retrieved so frequently. The ...
2
votes
1answer
1k views

NoSQL as file meta database

I am trying to implement a virtual file system structure in front of an object storage (Openstack). For availability reasons we initially chose Cassandra, however while designing file system data ...
2
votes
1answer
1k views

How to organize my site's file system properly?

Doing some reading on Stack Overflow, I've found a lot of information suggesting that proper organization of a file system is crucial to a well-written web app. One of the key pieces of evidence is ...
4
votes
1answer
3k views

File system implementation in MongoDB with GridFS

I am working on two projects that will both implement a Webdav server backed by a MongoDB GridFS. In each case, there is the potential for the system to store tens of millions of files spread across ...
2
votes
3answers
229 views

Is there an established convention for separating Windows file names in a string?

I have a function which needs to output a string containing a list of file paths. I can choose the separation character but I cannot change the data type (e.g. I cannot return a List<string> or ...
2
votes
1answer
2k views

Directory structure (file system design)

I was looking at how file systems are designed and noticed that most places say that the directory hierarchy can be implemented using a hash table. Could someone please explain me how using a hash ...
2
votes
1answer
100 views

concurrency semantics reading a file being written (ext)

In Ext 3/4 filesystem, if I have a log file open and being appended to, if another process tries to read it (e.g. with cat), what happens? Can the reading process read part of / an incomplete write? ...