techniques to organize and store files with their data on a computer.

learn more… | top users | synonyms

0
votes
0answers
47 views

Efficient File Storage [on hold]

I'm keeping a storage of a large number of files which grows indefinitely. Some of these files will be identical, but most will be unique. Over the last 5-6 days, the storage has grown to nearly ...
1
vote
2answers
89 views

Avoid data manipulation by user

I have a C# program (could be any programming language) that saves data to a file on a memory (hard disk, USB drive, etc.). The program uses this data for monitoring its operation time, but it could ...
0
votes
1answer
63 views

What is a simple way to let a user select a folder from a tree?

I have a Python-Flask app in which users can place files into a folder. As of now the directory structure is something like: /app /storage /templates . . . server.py The ...
2
votes
0answers
82 views

Efficient data structure to implement fake file system

I want to implement a data structure that will hold the paths of directories, sort of fake file system. Input:- I have a text configuration file containing the paths as follows ... C:/temp1 ...
-1
votes
1answer
40 views

What technical caveats exist using EXT file system on Windows that programmers should be aware of? [closed]

Every year I search for progress in mounting my Linux EXT (ext2,ext3,ext4) volumes on Windows. This isn't looking for those tools, I already know of things like Ext2Fsd - though if you know if ...
0
votes
2answers
107 views

Is renaming an 'alias' for moving?

Is it true to say (on Windows and Unix\Linux\OS X) that renaming a file or directory is just an alias for moving? e.g. Are there any side effects to either which are not present on the other? Does ...
1
vote
2answers
175 views

Why do filesystems read and write in blocks?

I read that file systems usually access data in blocks, whose size is integral multiple of disk block size. Why don't they read individual disk blocks?
0
votes
1answer
146 views

Best way to centralise a php project (except bitbucket)

The structure is as follows: A: this is the live/production website B: this is the staging, which is a copy of live C: this is the testing environment for designers 2 developer and 2 designers. ...
0
votes
1answer
267 views

Efficient way to map changes in a filesystem hierarchy

I'm currently working on a project that will enable file searching based on metadata found in the file. It'll be comprised of 2 parts: a filesystem crawler that passively scans for changes and ...
0
votes
1answer
27 views

Static analysis for finding capitalisation / case inconsistencies in file names

TLDR; I'm looking for ideas on how to flag code containing file names/paths that have inconsistent capitalisation with the actual file/directory. Situation I am migrating a significant code base ...
0
votes
4answers
171 views

Why is there little use of filesharing as compression (outside of libraries)?

Recently I was looking for a program that will run as a daemon and find files that have the same size/type, check if they're the same, then make both a hard link to a single copy if they are. And I ...
2
votes
1answer
200 views

Data structure well suited for duplicate entries

I'm in the process of getting to know (modern) filesystems. As part of it, I came across log structured filesystems that also handle allocations in a log structured way. I wonder how they handle ...
3
votes
1answer
106 views

Filesystem superblocks and their backup copies

I'd like to understand how (modern) filesystems are implemented and having trouble to fully understand superblocks and their backups. I reference ext4 and btrfs, but the questions may also apply to ...
-2
votes
2answers
101 views

file quantity limit in a directory on a linux file server and why?

What is a good limit to use on the quantity of files in a directory, and why? EDIT: Why shouldn't someone create a system that puts hundreds of thousands of files in the same directory? Why I ask: ...
4
votes
4answers
4k views

Is it safe to convert Windows file paths to Unix file paths with a simple replace?

So for example say I had it so that all of my files will be transferred from a windows machine to a unix machine as such: C:\test\myFile.txt to {somewhere}/test/myFile.txt (drive letter is irrelevant ...
3
votes
1answer
204 views

How to store the file names, start offset and length while avoiding the issue of self imposed limits (lookup table) or having to scan the entire file?

I am attempting to learn more about C and it's descendants(C++ mainly). I have decided that I would like to create a "file system" of sorts. Not a particularly advanced one mind you but something to ...
1
vote
6answers
536 views

Why can we not insert into files without the additional writes? (I neither mean append, nor over-write)

This occurs as a programming language independent problem to me. I have a file with the content aaabddd When I want to insert C behind b then my code needs to rewrite ddd to get aaabCddd Why ...
2
votes
2answers
306 views

Why must directories be empty before being deleted?

As far as I know, deleting a non empty directory could work the same way as deleting an empty directory: by removing the pointer to the directory's metadata there would be no pointers to the items it ...
1
vote
0answers
215 views

What database structure is suitable for tracking File audits?

I need to track permissions and access requests to a file server in a database. I'm given the full path of the folder and am considering parsing the path (splitting on the "/" character) and creating ...
2
votes
2answers
332 views

Why is it not standard to implement abstraction layers for the file system?

I have been taught to access databases through abstraction layers. I was wondering why it is not also standard practice to access the file system through an abstraction layer? It seems to me unit ...
2
votes
2answers
6k views

Is it wise to store a big lump of json on a database row

I have this project which stores product details from amazon into the database. Just to give you an idea on how big it is: [{"title":"Genetic Engineering (Opposing ...
8
votes
4answers
558 views

What exactly does it mean that storing “large blobs in the database reduces performance”?

To someone who knows database internals this may be an easy question, but can someone explain in a clear way why storing large blobs (say 400 MB movies) in the database is supposed to decrease ...
5
votes
3answers
2k views

How many threads should access the file system at the same time?

We have a module in an application which stores data in multiple files and multilevel directories and access them from multiple threads (both reads and writes). The directory structure is based on a ...
1
vote
2answers
2k views

Finding duplicate files? [duplicate]

I am going to be developing a program that detects duplicate files and I was wondering what the best/fastest method would be to do this? I am more interested in what the best hash algorithm would be ...
1
vote
1answer
341 views

Watching file changes/additions/removal, but with an eye on partial transfer

I would like to monitor the filesystem in python, so that my application gets warned of the new file addition, file removal, or file change. Once the file is detected, the application starts ...
4
votes
2answers
964 views

Is it a bad idea to sync file system with remote server using HTTP?

I have started a project which will duplicate Dropbox or Google Drive behavior but using Amazon S3 az a backend. Idea is very simple, a Node.js server that watchs a directory for file changes and PUT ...
27
votes
2answers
95k views

How to detect the encoding of a file?

On my filesystem (Windows 7) I have some text files (These are SQL script files, if that matters). When opened with Notepad++, in the "Encoding" menu some of them are reported to have an encoding of ...
2
votes
5answers
454 views

Is it plausible to use a filesystem-based configuration format rather than an INI file?

After having had endless issues with INI-style configuration files and parsing them correctly (let along getting binary data right), a crazy idea crossed my mind recently: What if an INI file like # ...
2
votes
4answers
1k views

Best practice to sync long paths to filesystems where path length is limited?

I wrote a sync tool to synchronize folders/files from Alfresco to Windows. PROBLEM: In Alfresco, /files/can/have/very/very/very/long/filepaths/like/this.txt The Windows API prevents me from creating ...
6
votes
3answers
3k views

Lowercase in Linux file names

As I find that UpperCase are really readable for first letter word separation in long complex names, I tend to give some of my Linux files names with some UpperCase. Mostly executables, some ...
0
votes
3answers
513 views

Which language is more suitable heavy file tasks?

I need to write a script (based on basic functions) to process /image/audio/video files. The process is mainly filesystem tasks and converts. The database of files has been stored by mysql. The script ...
3
votes
3answers
247 views

Storing lots of large strings with frequent “appends” and few reads

In my current project, I need to store a very long ASCII string to each instance of a given object. This string will receive an 2 appends per minute and will not be retrieved so frequently. The ...
1
vote
1answer
824 views

NoSQL as file meta database

I am trying to implement a virtual file system structure in front of an object storage (Openstack). For availability reasons we initially chose Cassandra, however while designing file system data ...
2
votes
1answer
887 views

How to organize my site's file system properly?

Doing some reading on Stack Overflow, I've found a lot of information suggesting that proper organization of a file system is crucial to a well-written web app. One of the key pieces of evidence is ...
4
votes
1answer
2k views

File system implementation in MongoDB with GridFS

I am working on two projects that will both implement a Webdav server backed by a MongoDB GridFS. In each case, there is the potential for the system to store tens of millions of files spread across ...
2
votes
3answers
223 views

Is there an established convention for separating Windows file names in a string?

I have a function which needs to output a string containing a list of file paths. I can choose the separation character but I cannot change the data type (e.g. I cannot return a List<string> or ...
2
votes
1answer
1k views

Directory structure (file system design)

I was looking at how file systems are designed and noticed that most places say that the directory hierarchy can be implemented using a hash table. Could someone please explain me how using a hash ...
2
votes
1answer
93 views

concurrency semantics reading a file being written (ext)

In Ext 3/4 filesystem, if I have a log file open and being appended to, if another process tries to read it (e.g. with cat), what happens? Can the reading process read part of / an incomplete write? ...
40
votes
2answers
3k views

What is the benefit of writing to a temp location, And then copying it to the intended destination?

I am writing an application that works with satellite images, and my boss asked me to look at some of the commercial applications, and see how they behave. I found a strange behavior and then as I was ...
1
vote
1answer
346 views

HTML5 file API or Java bridge to acces to local Files?

I've to access to files on the software user and I don't know if it's better to use a full JS app with HTML5 File Api rules or use Java and communicate with it ?
2
votes
7answers
3k views

Automatically delete files after they expire

I've got this idea for some time and I was wondering if anyone has seen such a feature/app in any operating system and if you haven't, what do you think about it. Where do you think I should begin? ...
2
votes
4answers
265 views

Where can I learn about how programs handle file systems like NTFS?

I'm writing a program to handle the xbox 360's STFS files. I'm encountering all sorts of problems about how to keep track of used vs unused blocks, reading non-sequential files, etc. These aren't so ...
5
votes
5answers
6k views

How to Model a simple file-system by UML class diagram

I want to model a file system which contains both files and directories, and directories can contain either files or other directories. This is what I have reached so far: In OOSE book, however, a ...
1
vote
6answers
193 views

Ways of marking changes being made to a file

I am looking for a lightweight way to notate changes made to a file. The obvious answer to this is a version control system. But this is more robust than necessary for what I would like to do. All I ...
8
votes
3answers
303 views

Examples of limitations in IT due to different bit length by design [closed]

I am teaching the course "Introduction in Programming" for the first-year students and would like to find interesting examples where the datatype size in bits, chosen by design, led to certain known ...
3
votes
3answers
187 views

Filesystems that use logging: If you're writing the data in the log (on disk), and in the actual locations themselves (also on disk) then…?

Aren't you essentially writing the same data twice, to the disk? Doesn't this cause a slowdown of a factor of ~2x? What optimizations can be made to minimize the cost of having to write things twice? ...
26
votes
11answers
2k views

Why are there so many competing filesystem designs? [closed]

Just a quick question, but why are there so many file systems still competing and in use today? (ntfs, fat32, ext3(ffs), etc) It seems that file system designers could agree upon the best aspects of ...
6
votes
2answers
237 views

Is there a fundamental reason for keeping file systems and versioning systems separate?

I've done some work looking at Next3 and ext3cow, and a few other file systems that provide snapshots and/or file versioning. It seems a bit strange to me that file systems only go part-way with ...
8
votes
4answers
524 views

File shredder algorithm

As a part of learning system programming, I am looking to implement a file shredder. The simplest way (and probably seen as naive) would be to replace the data bytes with zeroes (I know OS splits the ...
19
votes
7answers
2k views

Why is filesystem preferred for logs instead of RDBMS?

Question should be clear from its title. For example Apache saves its access and error logs in files instead of RDBMS no matter on how large or small scale it is being utilized. For RDMS we just have ...