techniques to organize and store files with their data on a computer.

learn more… | top users | synonyms

3
votes
3answers
116 views

What's the point of hidden files?

What is the point of hidden files? In Microsoft Windows they exist, in Mac OS X they exist and in Linux they exist. It seems to me that it just makes detecting malware more difficult. The only upside ...
0
votes
0answers
62 views

File based communication vs Rest or RPC

My company make automation machines (with a PC inside and a custom built software to control it). So, for the purpose of this question, we can say that these are normal PCs. Sometimes we have the ...
1
vote
2answers
145 views

How are non-folder files called? [closed]

Maybe it looks like a weird question, but what term should be attributed inside the code for files that are not folders to differentiate them from folders? If I need write 2 functions isFolder() and ...
1
vote
2answers
48 views

Can version control systems use the filesystem log to capture changes?

I was trying to find a "perfect" syncing program between a network share and a local folder, when I realised that it's probably impossible to do it right unless all the filesystem operations were ...
16
votes
4answers
1k views

Why is the Git .git/objects/ folder subdivided in many SHA-prefix folders?

Git internally stores objects (Blobs, trees) in the .git/objects/ folder. Each object can be referenced by a SHA1 hash that is computed from the contents of the object. However, Objects are not ...
1
vote
2answers
159 views

Avoid data manipulation by user

I have a C# program (could be any programming language) that saves data to a file on a memory (hard disk, USB drive, etc.). The program uses this data for monitoring its operation time, but it could ...
0
votes
1answer
64 views

What is a simple way to let a user select a folder from a tree?

I have a Python-Flask app in which users can place files into a folder. As of now the directory structure is something like: /app /storage /templates . . . server.py The ...
4
votes
0answers
242 views

Efficient data structure to implement fake file system

I want to implement a data structure that will hold the paths of directories, sort of fake file system. Input:- I have a text configuration file containing the paths as follows ... C:/temp1 ...
0
votes
2answers
113 views

Is renaming an 'alias' for moving?

Is it true to say (on Windows and Unix\Linux\OS X) that renaming a file or directory is just an alias for moving? e.g. Are there any side effects to either which are not present on the other? Does ...
1
vote
2answers
196 views

Why do filesystems read and write in blocks?

I read that file systems usually access data in blocks, whose size is integral multiple of disk block size. Why don't they read individual disk blocks?
0
votes
1answer
176 views

Best way to centralise a php project (except bitbucket)

The structure is as follows: A: this is the live/production website B: this is the staging, which is a copy of live C: this is the testing environment for designers 2 developer and 2 designers. ...
0
votes
1answer
276 views

Efficient way to map changes in a filesystem hierarchy

I'm currently working on a project that will enable file searching based on metadata found in the file. It'll be comprised of 2 parts: a filesystem crawler that passively scans for changes and ...
0
votes
1answer
30 views

Static analysis for finding capitalisation / case inconsistencies in file names

TLDR; I'm looking for ideas on how to flag code containing file names/paths that have inconsistent capitalisation with the actual file/directory. Situation I am migrating a significant code base ...
0
votes
4answers
174 views

Why is there little use of filesharing as compression (outside of libraries)?

Recently I was looking for a program that will run as a daemon and find files that have the same size/type, check if they're the same, then make both a hard link to a single copy if they are. And I ...
2
votes
1answer
249 views

Data structure well suited for duplicate entries

I'm in the process of getting to know (modern) filesystems. As part of it, I came across log structured filesystems that also handle allocations in a log structured way. I wonder how they handle ...
3
votes
1answer
133 views

Filesystem superblocks and their backup copies

I'd like to understand how (modern) filesystems are implemented and having trouble to fully understand superblocks and their backups. I reference ext4 and btrfs, but the questions may also apply to ...
-2
votes
2answers
112 views

file quantity limit in a directory on a linux file server and why?

What is a good limit to use on the quantity of files in a directory, and why? EDIT: Why shouldn't someone create a system that puts hundreds of thousands of files in the same directory? Why I ask: ...
6
votes
4answers
6k views

Is it safe to convert Windows file paths to Unix file paths with a simple replace?

So for example say I had it so that all of my files will be transferred from a windows machine to a unix machine as such: C:\test\myFile.txt to {somewhere}/test/myFile.txt (drive letter is irrelevant ...
3
votes
1answer
230 views

How to store the file names, start offset and length while avoiding the issue of self imposed limits (lookup table) or having to scan the entire file?

I am attempting to learn more about C and it's descendants(C++ mainly). I have decided that I would like to create a "file system" of sorts. Not a particularly advanced one mind you but something to ...
3
votes
7answers
664 views

Why can we not insert into files without the additional writes? (I neither mean append, nor over-write)

This occurs as a programming language independent problem to me. I have a file with the content aaabddd When I want to insert C behind b then my code needs to rewrite ddd to get aaabCddd Why ...
2
votes
2answers
356 views

Why must directories be empty before being deleted?

As far as I know, deleting a non empty directory could work the same way as deleting an empty directory: by removing the pointer to the directory's metadata there would be no pointers to the items it ...
1
vote
0answers
256 views

What database structure is suitable for tracking File audits?

I need to track permissions and access requests to a file server in a database. I'm given the full path of the folder and am considering parsing the path (splitting on the "/" character) and creating ...
2
votes
2answers
356 views

Why is it not standard to implement abstraction layers for the file system?

I have been taught to access databases through abstraction layers. I was wondering why it is not also standard practice to access the file system through an abstraction layer? It seems to me unit ...
3
votes
4answers
7k views

Is it wise to store a big lump of json on a database row

I have this project which stores product details from amazon into the database. Just to give you an idea on how big it is: [{"title":"Genetic Engineering (Opposing ...
9
votes
4answers
596 views

What exactly does it mean that storing “large blobs in the database reduces performance”?

To someone who knows database internals this may be an easy question, but can someone explain in a clear way why storing large blobs (say 400 MB movies) in the database is supposed to decrease ...
5
votes
3answers
2k views

How many threads should access the file system at the same time?

We have a module in an application which stores data in multiple files and multilevel directories and access them from multiple threads (both reads and writes). The directory structure is based on a ...
1
vote
2answers
2k views

Finding duplicate files? [duplicate]

I am going to be developing a program that detects duplicate files and I was wondering what the best/fastest method would be to do this? I am more interested in what the best hash algorithm would be ...
1
vote
1answer
385 views

Watching file changes/additions/removal, but with an eye on partial transfer

I would like to monitor the filesystem in python, so that my application gets warned of the new file addition, file removal, or file change. Once the file is detected, the application starts ...
4
votes
2answers
1k views

Is it a bad idea to sync file system with remote server using HTTP?

I have started a project which will duplicate Dropbox or Google Drive behavior but using Amazon S3 az a backend. Idea is very simple, a Node.js server that watchs a directory for file changes and PUT ...
40
votes
2answers
136k views

How to detect the encoding of a file?

On my filesystem (Windows 7) I have some text files (These are SQL script files, if that matters). When opened with Notepad++, in the "Encoding" menu some of them are reported to have an encoding of ...
2
votes
5answers
470 views

Is it plausible to use a filesystem-based configuration format rather than an INI file?

After having had endless issues with INI-style configuration files and parsing them correctly (let along getting binary data right), a crazy idea crossed my mind recently: What if an INI file like # ...
2
votes
4answers
1k views

Best practice to sync long paths to filesystems where path length is limited?

I wrote a sync tool to synchronize folders/files from Alfresco to Windows. PROBLEM: In Alfresco, /files/can/have/very/very/very/long/filepaths/like/this.txt The Windows API prevents me from creating ...
7
votes
3answers
4k views

Lowercase in Linux file names

As I find that UpperCase are really readable for first letter word separation in long complex names, I tend to give some of my Linux files names with some UpperCase. Mostly executables, some ...
0
votes
3answers
562 views

Which language is more suitable heavy file tasks?

I need to write a script (based on basic functions) to process /image/audio/video files. The process is mainly filesystem tasks and converts. The database of files has been stored by mysql. The script ...
3
votes
3answers
278 views

Storing lots of large strings with frequent “appends” and few reads

In my current project, I need to store a very long ASCII string to each instance of a given object. This string will receive an 2 appends per minute and will not be retrieved so frequently. The ...
1
vote
1answer
935 views

NoSQL as file meta database

I am trying to implement a virtual file system structure in front of an object storage (Openstack). For availability reasons we initially chose Cassandra, however while designing file system data ...
2
votes
1answer
976 views

How to organize my site's file system properly?

Doing some reading on Stack Overflow, I've found a lot of information suggesting that proper organization of a file system is crucial to a well-written web app. One of the key pieces of evidence is ...
4
votes
1answer
3k views

File system implementation in MongoDB with GridFS

I am working on two projects that will both implement a Webdav server backed by a MongoDB GridFS. In each case, there is the potential for the system to store tens of millions of files spread across ...
2
votes
3answers
225 views

Is there an established convention for separating Windows file names in a string?

I have a function which needs to output a string containing a list of file paths. I can choose the separation character but I cannot change the data type (e.g. I cannot return a List<string> or ...
2
votes
1answer
2k views

Directory structure (file system design)

I was looking at how file systems are designed and noticed that most places say that the directory hierarchy can be implemented using a hash table. Could someone please explain me how using a hash ...
2
votes
1answer
99 views

concurrency semantics reading a file being written (ext)

In Ext 3/4 filesystem, if I have a log file open and being appended to, if another process tries to read it (e.g. with cat), what happens? Can the reading process read part of / an incomplete write? ...
40
votes
2answers
3k views

What is the benefit of writing to a temp location, And then copying it to the intended destination?

I am writing an application that works with satellite images, and my boss asked me to look at some of the commercial applications, and see how they behave. I found a strange behavior and then as I was ...
2
votes
1answer
359 views

HTML5 file API or Java bridge to acces to local Files?

I've to access to files on the software user and I don't know if it's better to use a full JS app with HTML5 File Api rules or use Java and communicate with it ?
2
votes
7answers
3k views

Automatically delete files after they expire

I've got this idea for some time and I was wondering if anyone has seen such a feature/app in any operating system and if you haven't, what do you think about it. Where do you think I should begin? ...
2
votes
4answers
278 views

Where can I learn about how programs handle file systems like NTFS?

I'm writing a program to handle the xbox 360's STFS files. I'm encountering all sorts of problems about how to keep track of used vs unused blocks, reading non-sequential files, etc. These aren't so ...
6
votes
5answers
7k views

How to Model a simple file-system by UML class diagram

I want to model a file system which contains both files and directories, and directories can contain either files or other directories. This is what I have reached so far: In OOSE book, however, a ...
1
vote
6answers
195 views

Ways of marking changes being made to a file

I am looking for a lightweight way to notate changes made to a file. The obvious answer to this is a version control system. But this is more robust than necessary for what I would like to do. All I ...
8
votes
3answers
312 views

Examples of limitations in IT due to different bit length by design [closed]

I am teaching the course "Introduction in Programming" for the first-year students and would like to find interesting examples where the datatype size in bits, chosen by design, led to certain known ...
3
votes
3answers
192 views

Filesystems that use logging: If you're writing the data in the log (on disk), and in the actual locations themselves (also on disk) then…?

Aren't you essentially writing the same data twice, to the disk? Doesn't this cause a slowdown of a factor of ~2x? What optimizations can be made to minimize the cost of having to write things twice? ...
27
votes
11answers
2k views

Why are there so many competing filesystem designs? [closed]

Just a quick question, but why are there so many file systems still competing and in use today? (ntfs, fat32, ext3(ffs), etc) It seems that file system designers could agree upon the best aspects of ...