Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I am desiging a file format and I want to do it right. Since it is a binary format, the very first byte (or bytes) of the file should not form valid textual characters (just like in the PNG file header1). This allows tools that do not recognize the format to still see that its not a text file by looking at the first few bytes.

Any codepoint above 0x7F is invalid US-ASCII, so that's easy. But for Unicode it's a whole different story. Apart from valid Unicode characters there are private-use characters, noncharacters and sentinels, as I found in the Unicode Private-Use Characters, Noncharacters & Sentinels FAQ.

What would be a sentinel sequence of bytes that I can use at the start of the file that would result in invalid US-ASCII, UTF-8, UTF-16LE and UTF-16BE?

  • Obviously the first byte cannot have a value below 0x80 as that would be a valid US-ASCII (control)character, so 0x00 cannot be used.
  • Also, since private-use characters are valid Unicode characters, I can't use those codepoints either.
  • Since it must work with both little-endian and big-endian UTF-16, a noncharacter such as 0xFFFE is also not possible as its reverse 0xFEFF is a valid Unicode character.
  • The above mentioned FAQ suggests not using any of the noncharacters as that would still result in a valid Unicode sequence, so something like 0xFFFF is also out of the picture.

What would be the future-proof sentinel values that are left for me to use?


1) The PNG format has as its very first byte the non-ASCII 0x89 value, followed by the string PNG. A tool that read the first few bytes of a PNG may determine it is a binary file since it cannot interpret 0x89. A GIF file, on the other hand, starts directly with the valid and readable ASCII string GIF followed by three more valid ASCII characters. For GIF a tool might determine it is a readable text file. This is wrong and the idea of starting the file with a non-textural byte sequence came from Designing File Formats by Andy McFadden.

share|improve this question
3  
Since it is a binary format, the first bytes of the file should not form valid textual characters - You should look at the magic file (/usr/share/magic, or /etc/magic on many unix systems) that shows how this application identifies file types. A PNG file starts out with \x89PNG\x0d\0a\x1a\x0a -- note the "PNG" in there, that's a raw string. The sequences \x89 and the like are non-printable bytes. –  MichaelT Mar 13 '13 at 15:29
    
@MichaelT Yes, since PNG is a binary format, the first byte does not form a valid textual character. That's what I meant. I fail to see your point? –  Virtlink Mar 13 '13 at 15:36
6  
That was an example. A .gif starts out with GIF8. A SGI movi file starts out with MOVI. One style of zip archive file starts out with ZZ, the more popular pkzip format starts out with PK. The constraint that the first byte be an invalid text character does not seem to match what is found in the wild. I am curious why this is a requirement. –  MichaelT Mar 13 '13 at 15:51
3  
Do you really care how other programs behave when they see a unknown file? To me, a signature sequence (like PNG files) is much more useful than a sentinel sequence - when the content is sent through a simple stream protocol, the receiver can immediately decide how to handle the following bytes. A Omani-sentinel sequence is next to no-sequence once everyone starts using it to identify their own format. –  Codism Mar 13 '13 at 16:51
1  
@Virtlink, I dont particularly care what bytes you use in your file format. But you made an assertion that its 'wrong' to use ascii characters... yet I've not seen anything here that supports that claim, and there's plenty of empirical experience that shows it really doesn't matter (ie, the countless file formats that have been using ASCII characters without a problem for decades) –  GrandmasterB Mar 13 '13 at 19:49
show 7 more comments

3 Answers 3

0xDC 0xDC

  • Obviously invalid UTF-8 and ASCII
  • Unpaired trail surrogate in lead position regardless of endianess in UTF-16. It doesn't get more invalid UTF-16 than that.
share|improve this answer
    
But perfectly reasonable ISO-8859-1, and probably reasonable in any other character set that uses an 8-bit encoding. –  parsifal Mar 13 '13 at 23:06
2  
+1 OP didn't ask for ISO 8859-1, just US-ASCII and UTF-*. –  Ross Patterson Mar 14 '13 at 0:00
    
@RossPatterson - true, but I suspect that's mostly because the OP hasn't really thought through the problem. Without any statistics to back me up, I'm willing to bet that a random "is this text" algorithm is more likely to give preference to ISO-8859-1 than UTF-16, simply because there's an enormous amount of 8-bit text in the world. –  parsifal Mar 14 '13 at 12:57
1  
@parsifal Any binary is valid ISO-8859-1 so it doesn't need to be considered simply because it's impossible to make invalid ISO-8859-1. –  Esailija Mar 14 '13 at 13:05
    
@Esailija - valid, yes, but "text" files don't usually contain control characters (outside of the limited set of whitespace characters). –  parsifal Mar 14 '13 at 14:40
show 1 more comment
  • In UTF-8, the bytes C0, C1, and F5 - FF are illegal. The first byte must either be ASCII or a byte in the range C2-F4, any other starting byte is not valid UTF-8.

  • In UTF-16, the file normally starts with the Byte Order Mark (U+FEFF), otherwise applications have to guess at the byte order. Codepoints in the range D800-DBFF are lead bytes for a surrogate pair, and DC00-DFFF are the trailing bytes for a surrogate pair.

Thus, I'd use the byte combo F5DC. These two values are:

  • Not ASCII
  • Not valid UTF-8
  • Either interpreted as a UTF-16 trailing byte in a surrogate pair (not legal), or the codepoint U+F5DC, which is a private use character, but only by applications that stubbornly try to interpret this as UTF-16 even without a BOM.

If you need more options, F5DD through to F5DF all have the same 3 properties, as do F6DC - F6DF, F7DC - F7DF and F8DC - F8DF, for a total of 16 different byte combos to pick from.

share|improve this answer
    
So, by Esailija's suggestion to use U+DCDC, 0xDC would be valid UTF-8? –  Virtlink Mar 13 '13 at 16:38
1  
@Virtlink 0xDC is a UTF-8 lead byte for a 2-byte sequence. It must be followed by a 10xxxxxx continuation byte for it to be valid. 0xDC is not a valid continuation byte, so 0xDC 0xDC is not valid UTF-8. –  Esailija Mar 13 '13 at 16:40
    
@Virtlink: No, because the second byte is not valid, it would have to be in the range 80 - BF. –  Martijn Pieters Mar 13 '13 at 16:45
add comment

If you're trying to use a non-printable character to indicate "not text," then you'll find it hard to beat 0x89:

  • It's outside the US-ASCII range
  • In ISO-8859-1 it's a non-printable character ("CHARACTER TABULATION WITH JUSTIFICATION "). Likewise with Shift-JIS, which I believe is still in common use. Other 8-bit encodings may, however treat this as a valid character.
  • In UTF-8 it's an invalid first-byte for a multi-byte sequence (top bits are 10, which are reserved for characters 2..N of a multi-byte sequence)

Generally, when you form magic numbers, "non-text" is a minor point. I'll have to look up the reference, but one of the standard graphics formats (TIFF, I think) has something like six different pieces of useful information from its magic number.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.