Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

PNG is one of a family of container formats generally called FourCC or OSType. [A]IFF and WAV are two other common instances of the format.

I want to develop a new format for a specialized data type. I modeled the format closely on PNG. PNG is unusual for the FourCC family in that each block ends with a checksum. In theory this can be used to check for corrupted data.

Is that checksum useful for practical cases? New formats, like Google's WebP (based on RIFF), don't have a checksum. I can't find guidelines on what to do with it, including:

  • should all readers verify the checksum, or do most readers ignore it?
  • what should a reader do if it's wrong?

Since I'm not convinced it's useful, and supporting the checksum makes things more complicated, I'm going to omit the checksum in the next revision. But since it's in PNG, I figure there's a good reason, I wanted to get some feedback about how people actually use that field and its relative importance.

share|improve this question
add comment

1 Answer 1

up vote 1 down vote accepted

Is that checksum useful for practical cases?

As a lossless raster graphics compression format, PNG was developed in 1995 in response to Unisys waving a patent around and demanding royalties from the myriad software companies who were supporting GIF.

Back in those days of yore, when dinosaurs roamed the land and we carved our code onto the raw silicon with chisels, reliable, hard-wired Internet connections were uncommon and quite expensive. (I remember calling PacBell in the Bay Area in 1995 and asking about a DSL Internet connection, and getting a lot of "Huh? What's that?".) Most individuals and even small companies had their connectivity - whether Internet, UUCP, or BBS - via modems using dialup over flaky phone lines, and data corruption during transfers was endemic.

Hence the CRC values (not less-reliable checksums) in PNG - that was a way to see if the image you'd just downloaded had been corrupted and you needed to re-download it. You'll find the same kinds of checksums in other binary compression formats that date back to that era or earlier, notably zip archives.

Do you need it today? If you're positive all the protocols over which your data will be transferred will preserve its fidelity in all circumstances, perhaps not. But I still get corrupted zip archives occasionally, so when designing binary file formats for data that really matters, I consider CRC values to be cheap insurance against corruption.

should all readers verify the checksum, or do most readers ignore it?

With PNG, yes, they should.

what should a reader do if it's wrong?

Reject the image as corrupted and notify the user.

share|improve this answer
    
Thanks for pointing out the difference between a "checksum" and a "check value." Stone and Partridge in SIGCOMM'00 point out that the network stack will fail for one packet out of 16 million to 10 billion packets, or about 20GB to 15TB, and Bram Cohen, in his Stanford EE380/2005 talk on BitTorrent, concurs with the TB range. My data sets will be in the 10 GB range, which means CRC-32 isn't strong enough to detect corruptions after a few thousand transfers. If I go this route I should use something larger, and SHA-256 is more easily available than CRC-64 variants. –  Andrew Dalke Jun 13 '11 at 6:37
    
I forgot to address intended use. If the data will be something where it's not the end of the world if corruption is undetected, like soft drink ads for web users or pr0n, CRC or even skipping checks is probably okay. For something like medical imaging where the consequences of undetected corruption are dire, yes, SHA-256 or even SHA-512 is probably the way to go. –  Bob Murphy Jun 13 '11 at 14:25
    
Early stage drug discovery. Really early stage. The file contains something akin to a 4k Bloom filter for several tens of million chemical structures. It's not dire if one bit is wrong - there's so much noise that any small bit difference near the search threshold shouldn't make a difference. –  Andrew Dalke Jun 13 '11 at 18:03
    
@Andrew: Very cool! I'm an organic chemist by background and had some graduate courses in pharmaceutical design. Don't really do much with it any more - although I'm a wizard at getting out laundry stains - but I'm a very interested bystander around things like drug discovery, protein folding, structure-activity relationships, etc. –  Bob Murphy Jun 13 '11 at 18:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.