File handling refers to the set of tools, functions, and libraries to work with files and file handles. Creating, writing, appending, moving, and deleting files fall in this domain.

learn more… | top users | synonyms

71
votes
12answers
65k views

Is it a bad practice to store large files (10 MB) in a database? [closed]

I am currently creating a web application that allows users to store and share files, 1 MB - 10 MB in size. It seems to me that storing the files in a database will significantly slow down ...
57
votes
7answers
35k views

Difference between '\n' and '\r\n'

Yes yes, I am aware that '\n' writes a newline in UNIX while for Windows there is the two character sequence: '\r\n'. All this is very nice in theory, but my question is why? Why the carriage return ...
40
votes
2answers
3k views

What is the benefit of writing to a temp location, And then copying it to the intended destination?

I am writing an application that works with satellite images, and my boss asked me to look at some of the commercial applications, and see how they behave. I found a strange behavior and then as I was ...
21
votes
3answers
3k views

Should I pass in filenames to be opened, or open files?

Suppose I have a function that does things with a text file - for example reads from it and removes the word 'a'. I could either pass it a filename and handle the opening/closing in the function, or ...
16
votes
1answer
2k views

How do I test a file reader?

I am working on a project with a few file formats. Some formats are specified by .xsds, others by documentation on their respective websites, and some are custom in-house formats that have no ...
14
votes
7answers
3k views

Best way to handle delimited files

So typically a CSV file uses a comma and the return character as it's field and line delimiters. This runs into obvious problems with text which can contain both these characters. Obviously there ...
12
votes
7answers
2k views

Is it necessary to read every single byte to check if a copied file is identical to the original?

I recently learned of a program called Total Commander. It's a Windows Explorer replacement and has its own stuff to copy files. To check whether the files are identical, instead of calculation a CRC, ...
10
votes
4answers
4k views

What's the best practice for naming uploaded images?

Suppose I have a form in my web application where users can upload a profile picture. I've got few requirements about file size, dimensions etc, but when the user uploads the image, how should I name ...
9
votes
3answers
370 views

What is a good design for allowing backwards compatibility of files between different versions of software?

What is a good design for allowing backwards compatibility of a file type between different versions of software? For instance, how does microsoft get word 2007, 2010 and 2013 etc... to all open docx ...
8
votes
3answers
303 views

Should the creation of an object implicitly or explicitly create a file?

I'm creating an object whose sole purpose is to read in a file of one format and create another of a different format. Is it best to create the output file implicitly during object initialization or ...
7
votes
9answers
4k views

Multithreaded file copy

There is an utility we have which is used to upload files (and perform other operations on the file) to a network shared location. The file size tends to vary from a few mb to 500 mb. A suggestion has ...
7
votes
5answers
1k views

Test Driven Development: A good/accepted way to test file system operations?

I am working on a project at the moment that generates a table (among other things) based on the contents of a file-system, and and in turn does some meta-data modifications on the things it finds. ...
7
votes
1answer
1k views

“Read” a file without using a file pointer

I was asked this question in an interview. I'm somehow supposed to "read" a file into my C program as input without using a file pointer (including "f" functions, e.g. fgets, fscanf etc.). I'm also ...
5
votes
5answers
395 views

How are new file formats constructed?

I've used a software suite that is installed in offices and on remote vessels. The installations communicate back and forth, and they do that by using a simple proprietary file format that looks ...
5
votes
3answers
448 views

Where to load and store settings from a file?

I think this question should apply to most programs that load settings from a file. My question is from a programming point of view, and it is really how to deal with the loading of settings from a ...
5
votes
2answers
365 views

Hard Coded File Paths - Solution?

Say you have an application that saves image(.jpg, .png etc) and text(.txt, .xml) files, and the application has all of the files paths hard coded throughout code, per the example below. Assuming the ...
5
votes
2answers
135 views

How to distinguish whether a MS Word file is being saved or deleted? (“Save” deletes then re-creates)

I maintain a tool that syncs files between a client computer and a document management server (using the CMIS protocol). Versions history (on the CMIS server) is important. When saving a document ...
5
votes
1answer
163 views

Is there a standard way of cleaning data files?

Having spent quite a lot of time working in software companies that deal in medium to large data sets, I've seen that a choke point in their processes is often prepping data files for loading into ...
4
votes
5answers
482 views

Should I encrypt files saved by my program

I'm trying to decide on what format I should save data in, and hopefully this question will help me come to a decision. The program I'm writing uses 'databases' in which word=meaning (not a ...
4
votes
3answers
664 views

What is the best practice for reading a large number of custom settings from a text file?

So I have been looking through some code I wrote a few years ago for an economic simulation program. Each simulation has a large number of settings that can be saved to a file and later loaded back ...
4
votes
2answers
168 views

What is a stream, and how does it work?

Like a data stream in general, this can be an I/O stream or memory stream or a file stream or what not. I understand that it's some sort of stream of bytes, that you can read to parse very large ...
4
votes
3answers
661 views

Any good reason to open files in text mode?

(Almost-)POSIX-compliant operating systems and Windows are known to distinguish between 'binary mode' and 'text mode' file I/O. While the former mode doesn't transform any data between the actual file ...
4
votes
2answers
6k views

PHP file_put_contents File Locking

The Senario: You have a file with a string (average sentence worth) on each line. For arguments sake lets say this file is 1Mb in size (thousands of lines). You have a script that reads the file, ...
4
votes
1answer
187 views

How should I recursively process a file in Haskell?

Basically I'm expanding a literate program in Haskell. There are two things I need to look for to expand. Here's an example file: program.c.lit: ============== Some comments in the source file that I ...
4
votes
1answer
593 views

Designing status management for a file processing module

The background One of the functionality of a product that I am currently working on is to process a set of compressed files ( containing XML files ) that will be made available at a fixed location ...
3
votes
2answers
248 views

Why do binary files load quicker than alphanumeric text files?

I've noticed that when I load/store large data files in a binary format, the program runs much faster than if I load data from an ASCII encoded file. Why is this the case? The data in my case is ...
3
votes
3answers
503 views

Was API hooking done as needed for Stuxnet to work? I don't think so

Caveat: I am a political science student and I have tried my level best to understand the technicalities; if I still sound naive please overlook that. In the Symantec report on Stuxnet, the authors ...
3
votes
3answers
473 views

Design: Method that loads data from one of two file formats?

Say I have a class Person: class Person { string FullName; int Age; } And I need to load a list of Person's from a plain text file. The text file could be in one of two formats. I want a ...
3
votes
1answer
127 views

What does it mean that file IO binary flag is only relevant on Windows

I was reading a question and one of the comments mentioned that "The binary flag is only relevant on Windows". Given the context of the question and answer, I interpret this statement as on ...
3
votes
3answers
245 views

Storing lots of large strings with frequent “appends” and few reads

In my current project, I need to store a very long ASCII string to each instance of a given object. This string will receive an 2 appends per minute and will not be retrieved so frequently. The ...
3
votes
2answers
192 views

Design, how to utilize The Hardware (multiple threads and/or GPU) while indexing (via a database) a very large set of binary files

Problem How can design my file parser to maximize the hardware when processing (creating meta data for) a large file (i.e. how to avoid being blocked by IO and running out of memory) preferably ...
3
votes
2answers
741 views

How to simulate inner join on very large files in java (without running out of memory)

I am trying to simulate SQL joins using java and very large text files (INNER, RIGHT OUTER and LEFT OUTER). The files have already been sorted using an external sort routine. The issue I have is I am ...
3
votes
1answer
120 views

What does “windowed streaming” stand for?

So I was asking around the Mercurial development mailing list about binary diffing and patch handling and I got the following examples: whole file approaches (classic diff, bsdiff, Mercurial's ...
2
votes
5answers
240 views

Closest file match?

I am looking for a way to compare a set of files against a given file with each comparison giving me a "closeness" metric. I should then be able to order based upon the metric to find the closest ...
2
votes
2answers
894 views

Optimize Reading of Multiple files

I have a unique problem which I'm hoping someone can assist with. I have One big text file, our Production file. The data in the file is delimited in the following format Reference|Cost ...
2
votes
4answers
264 views

Where can I learn about how programs handle file systems like NTFS?

I'm writing a program to handle the xbox 360's STFS files. I'm encountering all sorts of problems about how to keep track of used vs unused blocks, reading non-sequential files, etc. These aren't so ...
2
votes
3answers
99 views

Smart fallback for line endings in code files? [closed]

What do you recommend as a default fallback for line endings if the operating system cannot be determined? Example Mac OS up to version 9 uses carriage returns (\r) for line endings. Unix-like ...
2
votes
2answers
119 views

Criteria for selecting timeout value?

Situation: a piece of software reads frames of data from a file in a seperate thread and puts it on a queue, emptied by another thread. That second thread periodically checks on the queue and fails ...
2
votes
3answers
371 views

Quickly compute added and removed lines

I'm trying to compare two text files. I want to compute how many lines were added and removed. Basically what git diff --stat is doing. Bonus points for not having to store the entire file contents in ...
2
votes
1answer
182 views

How to name filename parameter to make clear it should contain full path? [closed]

I'm seeking vital naming of string filename parameter in parameter list used in various methods where filename with full path is expected. In many cases also UNC path can be actually supplied as full ...
2
votes
2answers
179 views

Recreating files from shell script or java?

I have java application which takes a file, which is created by a process running on terminal. I start the process using a small shell script. Then run the Java application and it reads the file ...
2
votes
1answer
95 views

File Processing Times

I have a very simple program in Ruby that opens a dictionary file, sorted-words.txt and prints out all the words in pig-latin. I tested its speed, and the first time I ran it, the program finished in ...
2
votes
1answer
99 views

How to share a file over multiple projects

I'm using Visual Studio 2010 to amend a C# project. Part of the application requires the user to be able to save and read a settings file (1 Project reads them in (as read only) from a file. The ...
1
vote
2answers
323 views

Is there actually an advantage to using Win API calls for file access?

I remember coding file access using the Win API about 15 years ago. It was much faster than using the FileStream of my language at the time (Delphi). I spent a while today experimenting in C# under ...
1
vote
1answer
287 views

Editing files without race conditions?

I have a CSV file that needs to be edited by multiple processes at the same time. My question is, how can I do this without introducing race conditions? It's easy to write to the end of the file ...
1
vote
2answers
537 views

How do I / what is the best way to interact with a desktop application from a browser web application?

Situation: We have a desktop application. We also have a web application. We need the two to interact with each other. So when a button is pushed in the web application, the desktop application ...
1
vote
3answers
122 views

Which part of the code should rewind a file pointer?

If you had a function that took a handler to a file and then performed some actions that moved the pointer, would you expect it to put the pointer back when it was finished? Additionally, would you ...
1
vote
3answers
300 views

How to make a random-access archive of text files?

I wrote an application that tests the performance of evolutionary algorithms. This application performs a run of the algorithm which consists of several generations. The data which is produced by my ...
1
vote
2answers
373 views

Retrying a statement or call in a catch block - code smell or anti-pattern?

I'm wondering how better to perform this operation for a large amount of files. The bit I'd like some thoughts on whether this copy/paste is acceptable enough of a tradeoff. try to write a file if ...
1
vote
2answers
291 views

Is creating and writing to one large file faster than creating and writing to many smaller files in Python on Linux?

If using Python on a Linux machine, which of the following would be faster? Why? Creating a file at the very beginning of the program, writing very large amounts of data (text), closing it, then ...