Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We have different ways to store program data (save files in games, employee databases, program configuration etc.):

  • Plain text (think .ini and .conf)
  • XML
  • Databases (MySQL, SQLite...)
  • .zip and similar containing several files (with different formats)
  • Binary files (think .doc etc., for instance created by a serialization tool)

What are different use cases for the formats listed above, and what are their advantages contra disadvantages (think speed, flexibility, file size, ease of use...)? How to decide between them for different tasks?

About the zipping format: This is just used for containing other files. It could be another compression format as well. This allows for a structure of several files, including image files, sound files and text files. As an example, say you have a storage format for messages, which may contain files. You can have the following files inside a zipped file:

message.txt (containing the message)
attachments (folder containing attachments)
  audio.wav
  picture.jpg
share|improve this question
    
wrt binary, consider Google Protocol Buffer. The lazy deserialization ability is awesome, and you always have the possibility to extract it and resave it as formatted text (in several languages C++/Java/Python). –  Matthieu M. Apr 16 '11 at 15:28

5 Answers 5

I use as follows:

Plain text

For configuration - usually using YAML or .ini. Deprecated by me for most uses except when a text file is the desired result (e.g. print to text, save to text etc.)

XML

For configuration and transportation of data; e.g. export, format via XSLT etc. Good as a portable file format (e.g. SVG). Excellent manipulation tools and filters.

Databases

Main data storage from inside app/webapp. Use it all the time as storage of choice. It's reliable, robust, and you get a lot built in (transactions, referential integrity, cascading delete/update, indexes, speed). Best used with a layer, or ORM (IMO).

Single file archive (e.g. .zip)

Suitable for storing related multiple binary streams compactly, e.g. ROM images for an emulator. Best for things that don't often or never have to be updated. It is heavyweight, slow and hard to manipulate;

Binary

Only where a database is not available for storing app data. Easiest with serialization (C++). A highly tuned binary format will outperform everything else for both speed and size.

share|improve this answer

There's no silver bullet. In my experience:

Plain text as a storage medium is an automatic no. The few cases I would even consider it would be better covered by a .config file where I have a schema and type safety. It seems the need for type safety and data extraction almost always comes up. Plain text makes this process a nightmare.

XML: Type safety, data validation, low volume, and in some cases I use it because .NET has powerful built in support for XML serialization of objects.

Databases: My default. Type safety, speed, transactions, well trusted and hard to get blame for picking a DB as a storage medium if something doesnt go according to plan.

.zip is a compression format, not sure how this fits into persistance..?

Binary: I only use binary when I need to create a temporary memorystream. Binary doesn't add value in the way of query-ability as compared to a DB or XML where my data is organized with schema.

Ease of use is relative and depends on what specifically you want to accomplish. Speed is similar outside what I said above regarding volume. If file size is a concern and proper normalization applied, I will compress it via zip or some other compression format, but this is a seperate process.

share|improve this answer

I use them as follows:

Plain Text

If that category includes slightly more elaborate formats, like YAML or properties files, then it's the best option for whatever you expect people to read and edit by hand. Another huge advantage is the simplicity of modifying it via a small script (e.g. sed).

Nothing beats the simplicity and ease of use. When the support team has to configure something on a remote machine (e.g. solve a client's problem), or IT has to reconfigure a bunch of servers that run your software, they'll thank you for choosing this format. It'll also save you from writing some one-off software that does that for them.

XML

I agree with @Ingo here - unlike plain text XML is harder to process via scripting, and a nightmare to edit by hand imo.

Still, if you have data with some elaborate structure where YAML becomes indecipherable and still want it to be human-readable and editable, then XML is probably the best choice.

Relational Database

A great choice for when you have lots of data (that would make plain text and XML cumbersome) that you may still want to allow 3rd parties to edit manually - via SQL commands and even GUIs.

Another advantage is that your code that manages the contents is very readable. @Richard-Harrison gave a good list of other advantages in his excellent answer.

NoSQL Database

One advantage over RDBMS is scalability through distribution, which is probably not very relevant to your question. The advantages that are probably more relevant are the simplicity of a key-value store and the flexibility of schemalessness (is this a word?). When you find yourself breaking the relational paradigm: just storing blobs to the database, accessing them by key, and processing them through code, then consider this option. Some choices (e.g. CouchDB) are very portable, have a small footprint and can also scale so they offer a good non-relational alternative to MySQL and SQLite.

Binary

The advantage of binary is that it's fast and compact. When the only thing that needs to read and modify your file is a program and the data doesn't fit the relational paradigm or speed is really important then this may be a good choice. Probably the best fit for media files.

I should point though that I've yet to encounter a case where simple access to program data isn't required at some point for reasons that weren't considered during the initial design. Nowadays I personally go for the database option for anything other then files that have standard formats and need to be encoded/decoded by other software (e.g. audio, video).

Note: there's a common misconception that binary is opaque and thus somehow more secure. Without additional protection it isn't - if someone wants to hack your software then simply storing your configurations or whatever in binary won't stop them.

Compressed Archive

Not really an alternative to the above, but rather an extra measure.

Advantageous when you need to transmit things over the network, or when you store lots and lots of data and want to save space. Note that storage space is usually abundant these days, so consider your target platform.

Performs very fast on almost anything today (Moore's law in action, baby), so the only reason not to use it is that it adds complexity to your code. Not a lot of complexity, but still a violation of the KISS principle. Especially cumbersome for configuration files that need to be edited manually or via scripting - and if you really need to save space there, then you should probably use the database option.

share|improve this answer

I have heard that XML combines the worst features of text (difficult/slow to process) and binary (unreadable).

share|improve this answer
    
Not a complete answer –  Anto Apr 16 '11 at 13:15

I would use them as follows:

  • Plain text : The application has small size of simply structured data (name value pairs for ex.). Data are not modified concurrently by multiple users.
  • XML : Small size of structured data that are not modified concurrently or frequently.
  • Database : large structured data or concurrent access is needed. Need for querying and search is a must in the application.
  • Binary data: I would use that just for streaming objects.
  • zipping is compression that may be added as another process for any of the above except databases on servers.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.