Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm about to start writing a process for saving some data structure from code in to a file of some proprietary, as-yet-undefined type. However, I've never designed a file type or structure before.

  • Are there any things, generally speaking, that I should consider before starting my design?
  • Are there any accepted good practices here? Bad practices I should avoid?
  • Any absolute do's and don'ts?
  • Can anybody recommend any good reading on this topic?
share|improve this question
add comment

2 Answers 2

up vote 12 down vote accepted

First, try to find a format that is close enough to what you are about to build. In general, it is better to use someone's format than to invent your own, even if the format appears to be slightly more complex than what you need1.

If you cannot find a suitable ready-made format, see if you can build your own on top of an existing general-purpose format, such as XML or Binary XML. This should be possible in nearly all cases when you are about to start a new file format. Text-based XML takes more space, but gives humans some measure of readability. However, if you find yourself using Base-64 encoding inside an XML file, that's a clear indication that you should have used a binary encoding instead.

As far as good and bad practices go, make sure that you do not "bake in" the hardware feature of your initial target platform into the design of your file format. Specifically, make sure that your numbers are stored in a format that can be read correctly on platforms with endianness that is different from that of the writer, and that your user-facing strings are stored in UNICODE.

Another good practice is to include a header from which it is possible to determine the type of your file in case its extension is missing or incorrect. It is a good idea to include a version of your file format in the header. This would let you change the format later, and stay backward-compatible.

If possible, do not make your format dependent on the specifics of the default serialization mechanism built into your platform. For example, binary-serialized Java objects do not make a good file format2.

Finally, decide if your files need to be streamable. This introduces additional complexity, because one should be able to interpret individual "frames" of your file in isolation. In cases when you need streamability, however, you should almost always be able to locate a suitable file format that already exists.


1 On the other hand, you should avoid formats that require extraordinary efforts to support the complexity that your application requires.

2 This does not mean, however, that you should not attempt to custom-integrate reading and writing of your new format with the serialization scheme of your platform, only that you should not rely on the default mechanisms of serialization.

share|improve this answer
add comment

The first thing you should consider is whether you actually need a new format or if you can get by using an already existing format. Consider using SQLite; if you can adapt your needs to fit the RDBMS model, this could save you a lot of headaches. Also, consider using XML or JSON, this will save you from having to write your own parser.

If you do have to make your own format, the first consideration is whether you want a text format or a binary format. There are advantages to both. A text format is a big win for portability and has the advantage of being easier for a human to read or edit. A binary format could be more efficient, but it has a lot of portability problems that come with it. Don't be tempted to read bytes directly into variables, you'll regret it if you need to port the code to another platform.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.