Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I have to read and write binary "chunks" of approximately 1Mb each. The data can come in the form of a stream or a in-memory byte[].

Normally I would use a Struct with a fixed layout, but there are a few properties of this binary data that are making this harder:

  1. Byte 4 describes the "version" of the chunk
  2. Byte 5 starts either as variable array of sub-chunks or as a human readable blob
  3. Each sub-chunk has a version, and has a variable array of data within it.
  4. The endian-ness of the data is inconsistent

How should I structure a program that needs to handle binary data thats in this format?

share|improve this question
    
What interface do you need to provide to the data? –  Telastyn Mar 21 '13 at 14:14
    
Does the chunk "version" give information about points 2, 3, and 4 on your list? –  Phil Mar 21 '13 at 14:35
    
@Telastyn If I read that data it will go to SQL, Azure Table, or to an in-memory array. If I write the data, it will go back into a binary format on disk or to a TCP stream –  makerofthings7 Mar 21 '13 at 14:55
    
@Phil I am still researching this, though #4 is consistent for specific known data values. Some types are Big, some are little endian. The protocol isn't well defined and I'm not sure if the structure will need to adapt on the fly. (e.g. I read N bytes and might need to backtrack and discover this is "sub chunk type 3" and reprocess) –  makerofthings7 Mar 21 '13 at 14:56

2 Answers 2

I once had to implement a versioned de/serialization abstractions similarly to what you're describing before, I'll detail what I (can remember I) did:

I created an unversioned form of the objects that were to be serialized/deserialized which just decorated the latest version so all consuming code always pointed to the versionless ones which were always the latest versions; this way when new versions came out I didn't have to change any consuming code, just those versionless abstractions had to be pointed to the new versions.

So let's say an example like..

public class Person : V4.Person { }

Then in the implementation, I had each versioned object upconvert on deserialization by passing down to older until one of them recognized the version so it was something like..

public class V4.Person : V3.Person
{
    public string SomethingNewInV4 { get; set; }

    public static Person FromBytes(BinaryReader personReader)
    {
        Person person = base.FromBytes(personReader);

        if (person.Version >= 4) // If this version or newer, it will have the SomethingNewInV4 property, deserialize it.
        {
            int stringLength = personReader.ReadInt();
            person.SomethingNewInV4 = personReader.ReadString(stringlength);
        }
        else
        {
            person.SomethingNewInV4 = "default string used when upconverting from V3";
        }

        return person;
    }
}

In this way, each new version is just a modification of the previous, inheriting the previous version and adding (or hiding) parts from it, and using it's serialization/deserialization which it then adds onto it's particular parts special to it's version. There is the possibility for a breaking change in the chain but that just means that particular version may have to reimplement serialization/deserialization from scratch including all the intelligence for branching if previous versions should do it or not, but scenarios like that are likely minimal.

The versionless facade in front ensures all new serializations are serialized in the latest format, and all old deserializations start at the top of the stack which will call down to the bottom through inheritance, and each version will upconvert from there, so that versionless facade redirecting to the latest is necessary (unless each time a new version comes out you want to update all consuming code).

V1 will have to do the version picking and full deserialization etc, but above that each version will check which version and choose whether the data for it is there or it needs to fake it up.

When I did this, the subchunking you're referring to was totally there as well and I handled this by having this same layout for each subchunk type, so you could have Person's deserialization call to Arm's deserializer when it got to the Arm chunk, and if it had a numeral in front counting the Arms it would use that to deserialize in a loop. Each versioned subchunk needs a versionless front piece where all the main chunks call to, so V4.Person would call to the versionless Arm's deserializer which may point to V3 if that's the latest Arm, or etc.

Last point: I strongly suggest memory stream for this, I remember going back and forth on this and realizing the byte array was terrible because the forward movement meant I had to pass pointers around between versions, with the memory stream version 1 takes all the initial V1 data, V2 takes all the V2 data which is layered after that, and so on and so forth without needing to maintain the pointer to your current position in the array.

Your problem is almost step by step identical to the one I faced and solved in this manner, it worked pretty well, hope this helps! To make things a little easier I made extension methods on BinaryReader for some of the types that pointed to the versionless stuff so there would be like

public Arm ReadArm(this BinaryReader target) { return Arm.FromBytes(target); }

then in person you would have..

int numberOfArms = personReader.ReadInt();
List<Arm> arms = new List<Arm>();
for(int i = 0; i < 1; i++) arms.Add(personReader.ReadArm());

I made it fit with BinaryReader's normal naming, don't remember if it's Read or To.. Either way it was a handy shortcut.

share|improve this answer

I wonder if some sort of lexer/parser set-up would be in order, since:

  • it seems like you're going from an input stream (of some sort) to a tree-like structure (if I'm reading things right)

  • it would be a way to do the back-tracking you mentioned you might need to do

  • it could probably handle some of the endianness uncertainty/switching you might encounter

I don't know if you could use one that's pre-written in this use case, but it might be a place to start with thinking about structuring things.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.