Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm about to start writing a process for saving some data structure from code in to a file of some proprietary, as-yet-undefined type. However, I've never designed a file type or structure before.

  • Are there any things, generally speaking, that I should consider before starting my design?
  • Are there any accepted good practices here? Bad practices I should avoid?
  • Any absolute do's and don'ts?
share|improve this question
3  
Hello from 2016! This is the textbook definition of a question that is too broad :) It would help if you provided specifics about what you're trying to achieve. Most proprietary file formats are designed for something. What are your requirements? – Andres F. Mar 3 at 21:27
    
Don't forget the key benefit of using a standard format: other tools and applications will be able to ingest it more easily. And you might not be the only person working with that data. If you work with that principle in mind, your software will be more flexible, and that has a way of simplifying things. – Joey Adams Mar 4 at 17:08
up vote 18 down vote accepted

First, try to find a format that is close enough to what you are about to build. In general, it is better to use someone's format than to invent your own, even if the format appears to be slightly more complex than what you need1.

If you cannot find a suitable ready-made format, see if you can build your own on top of an existing general-purpose format, such as XML or Binary XML. This should be possible in nearly all cases when you are about to start a new file format. Text-based XML takes more space, but gives humans some measure of readability. However, if you find yourself using Base-64 encoding inside an XML file, that's a clear indication that you should have used a binary encoding instead.

As far as good and bad practices go, make sure that you do not "bake in" the hardware feature of your initial target platform into the design of your file format. Specifically, make sure that your numbers are stored in a format that can be read correctly on platforms with endianness that is different from that of the writer, and that your user-facing strings are stored in UNICODE.

Another good practice is to include a header from which it is possible to determine the type of your file in case its extension is missing or incorrect. It is a good idea to include a version of your file format in the header. This would let you change the format later, and stay backward-compatible.

If possible, do not make your format dependent on the specifics of the default serialization mechanism built into your platform. For example, binary-serialized Java objects do not make a good file format2.

Finally, decide if your files need to be streamable. This introduces additional complexity, because one should be able to interpret individual "frames" of your file in isolation. In cases when you need streamability, however, you should almost always be able to locate a suitable file format that already exists.


1 On the other hand, you should avoid formats that require extraordinary efforts to support the complexity that your application requires.

2 This does not mean, however, that you should not attempt to custom-integrate reading and writing of your new format with the serialization scheme of your platform, only that you should not rely on the default mechanisms of serialization.

share|improve this answer
2  
Unicode is not a format. It is just a mapping. UTF-8 is a format and is generally considered to be the proper format for portable text strings (unless you're handling CJK text in which case I think UTF-16LE is probably the next most common format for strings) – squarewav Mar 4 at 3:00

The first thing you should consider is whether you actually need a new format or if you can get by using an already existing format. Consider using SQLite; if you can adapt your needs to fit the RDBMS model, this could save you a lot of headaches. Also, consider using XML or JSON, this will save you from having to write your own parser.

If you do have to make your own format, the first consideration is whether you want a text format or a binary format. There are advantages to both. A text format is a big win for portability and has the advantage of being easier for a human to read or edit. A binary format could be more efficient, but it has a lot of portability problems that come with it. Don't be tempted to read bytes directly into variables, you'll regret it if you need to port the code to another platform.

share|improve this answer
    
Remember also, when designing your own format, that you MUST account for broken files and making sure they won't break your app. Also, if using XML, please, for the love of all that is precious to you: USE AN XML PARSER. Doesn't matter which one (your language may well provide one), but DO NOT PARSE XML YOURSELF. – Michael Kohne Mar 3 at 21:44
    
SQLite if you need a database with individual records that can be changed instantly. JSON if it is Ok to read all the data into memory, modify it in memory, then write it back to the file. – gnasher729 Mar 3 at 22:29
1  
@MichaelKohne: Same for JSON. I'd estimate 10% of iOS questions are about crashes when some JSON contains a null where they expect a string. – gnasher729 Mar 4 at 0:09

Your first and most important decision is whether to use a binary format or a text based one. Binary is the way to go when you have to dump big amounts on non-string data. But it has significant downsides:

  • Binary data is not human readable. As such, it makes debugging and/or tweaking data that's already on disk much harder. This is one of the reasons why the UNIX philosophy so strongly embraces text based files.

  • Binary formats don't lend themselves to future expansion. While this can be done, points for expandability need to be built into the format from the very beginning. Typically, these are

    1. a magic number/string that identifies the format

    2. a format version number

    3. reserved fields at strategical positions, which must be initialized to zero

    The first two typically appear right at the beginning of the file, while reserved fields are usually scattered across the file.


Now, if you go the text-based route, here are some things to think about:

  • Any text based format defines a new mini-language. Know this and use it to your advantage.

  • Try to keep the rules of your mini-language as simple as possible. There is no place where the KISS principle is more important than when designing a text based file format.

  • Try to make your files self explanatory.

  • Don't impose unnecessary restrictions, like where whitespace may appear, and how much of it.

  • Take a good look at a number of different file formats developed for UNIX. This can give you quite a few good ideas.

  • If possible use or adapt/expand/constrain an existing file format. The json format is a quite readable, good starting point. (At least much better than XML, which is a pain to read for humans.)

  • If file size is an issue, you might consider using a text based format anyway, but pass it through one of the standard compressors like gzip or lzma. The standard compressors love input like that.


If you go the binary route, here are some things to watch out for:

  • You should have a header with a magic number/string and a version number. Usually this goes to the beginning of the file, but it may also go to the end of the file. Some files may even have two different headers in front and back, giving two independent views on the data within.

  • You should have an index, and you should try to keep its parts close together. This allows a reader to quickly find out what's inside the file, without having to scan the entire thing. If you fail to do that, you might end up reading everything two times.

  • If you have bits of the file that are only accessible as a sequence instead of via an index structure, include at least a length field to each record in the sequence. Either an index or such length fields are requirement for readers that do not understand all the details of your format, and need to skip parts of it as black boxes. (Thanks to Jules for this one.)

  • Every data object within the file needs to contain at least one reserved field for future expansion. This does not need to be big, but it needs to be there. Because, if it isn't, there is no place where you could recognize the future features.

  • You need to take endianess into account. Typically that means that you decide once whether your files should be encoded in big endian or little endian byte order, and stick to that decision: Handling endianess like this is a nuisance, but it's nowhere near as bad as having to account for two different versions of endianess in the file.

  • Be generous in the widths of the fields that you provide. Especially, when you need to encode offsets within the file, always use 64 bits. Many a headache has been caused by file format designers that were too conservative with the amount of bits they allocated.

share|improve this answer
    
I'll add another one for binary files: if you have a sequence of records of varying type (which is a very useful structure) DO NOT assume that any reader of the file understands all types. Use length indicators so unknown records can be skipped. – Jules Mar 4 at 12:25
    
@Jules Right you are :-) I added that as point 3 for the binary files. – cmaster Mar 4 at 17:30

It really depends on what you're doing. It should be as simple as possible and not simpler. I see a lot of other people pushing XML. I strongly discourage the use of XML. XML is an over-specified mess. The first question would be do your data structures have branches. Meaning are they lists of lists or lists of maps or such? If no, then a simple sequence of text records might be good. CSV maybe.

If you need performance or random access, binary is good. Define a sequence of records where each record contains a sequence of datums that have a specific size like a 4 byte little endian integer for some number or a 2 byte integer that specifies the number of bytes for a UTF-8 string. Make each record begin with an integer specifying the size of the record so that the entire file can be scanned without actually reading the content of the records. This also allows you to encode records in-situ (meaning you can mmap the file, encode the record and then update the size after so as to minimize unnecessary copying). This is the sort of thing that you cannot do with XML.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.