Serializing Data Structures in C

Question

I've recently read three separate books on algorithms and data structures, tcp/ip socket programming, and programming with memory. The book about memory briefly discussed the topic of serializing data structures for the purposes of storing it to disk, or sending it across a network. I can't help but wonder why the the other two books didn't discuss serialization at all.

After an unsuccessful web/book search I'm left wondering where I can find a good book/paper/tutorial on serializing data structures in C? Where or how did you learn it?

Answer 1

Have a look at the work Google has done with Protocol Buffers.

You write a .proto file like this:

message Person {
  required int32 id = 1;
  required string name = 2;
  optional string email = 3;
}

Then you compile it with protoc, the protocol buffer compiler, to produce code in C++, Java, or Python.

Then, if you are using C++, you use that code like this:

Person person;
person.set_id(123);
person.set_name("Bob");
person.set_email("bob@example.com");

fstream out("person.pb", ios::out | ios::binary | ios::trunc);
person.SerializeToOstream(&out);
out.close();

You can examine the SerializeToOstream method to understand how Google generates the serialization code. Yes, it's C++ code, but it should still be pretty close to C code.

Answer 2

C has no native support for serializing structures, so you're on your own.
The first order approximation is (as stated in other replies) to define it for primitive types, and apply it recursively to larger structures.

However, there are lots of devilish details that have to be addressed beyond the simple concept. To name a few:

• endian order of integers, and sizes of various common types of integers depending on machine architectures. This isn't much of a problem if all the consumers of serializatin are the same binary, but consider reading data produced by a 32 bit PPC Mac on a 64 Bit Windows machine, or if a "long" is 32 or 64 bits.
• Different representations for common data types. Color bitmaps have 3 components on a PC, but 4 components, in a different order, on Macs
• Representation and precision of floating point numbers.
• If strings with the same letters are idential or only similar.
• dealing with cyclic or self-referential data structures.

Answer 3

Serialization is usually pretty simple and recursive. You just figure out what fields you need to send before the actual data so you can reconstruct the structure at the other end. Issues would mainly be endianness. Learn it by trial and error, I doubt you need a book for this sort of thing.

Answer 4

The Wikipedia article Serialization covers the topic fairly well, though oddly it does not mention ASN.1 which is a widely hated, but extremely well defined and well known standard for describing efficient data serialization protocols. ASN compilers typically generate code (e.g. C code) for encoding and decoding the described data structures in a canonical way.

BTW, the endian issue can be trivially dealt with in C, as Rob Pike has shown nicely in his article The Byte Order Fallacy, though some C compilers don't yet always generate the most optimal object code when using this technique.