Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We are starting a project where we will need to write parsers for a bunch of binary file formats, each of them representing very similar data (time-value series from different measurement devices).

Since we are starting from scratch, I would like to get it right, and I see two possible approaches:

  1. write dedicated, home grown binary parsers for each format separately, or

  2. represent binary formats using a grammar, and then use some standard algorithms for lexical analysis/tokenization.

Whenever I seek for advice on how to build a parser, I find most guys advocating the latter approach. However, I don't have much experience with formal grammars and languages, and I am afraid that there might be a learning curve before we get results.

So, I basically have these questions:

  • What is the problem with coding parsers "by hand"?
  • Is there a practical "size limit" of a problem when it pays off to invest in learning the "formal approach"?
  • Most parsing examples focus on textual files. What is the good way to specify grammar for a binary parser?
share|improve this question

2 Answers 2

up vote 4 down vote accepted

I don't have much experience with formal grammars and languages, and I am afraid that there might be a learning curve before we get results.

Spend a couple afternoons building a simple project to try it out. Given that you need to support "a bunch" of formats, it's a good bet your investment in learning to use tools that do exactly what you need will pay off quickly.

What is the problem with coding parsers "by hand"?

The problems are mainly:

  • writing and maintaining parsers by hand is time consuming, difficult, and error-prone

  • writing the code yourself means that there's another layer of indirection between the two things you care about: the format and the parser. If the parser doesn't work quite right, you have to look at the code to figure out why. If you can specify the format as a grammar instead, it should be easier to see where the problem is (or avoid problems in the first place).

Is there a practical "size limit" of a problem when it pays off to invest in learning the "formal approach"?

I suspect that the where the line is depends on your situation. Coding parsers by hand will always feel easier if you don't know any other way. As you get more comfortable with tools like flex and bison (or whatever you choose), the line will move.

Most parsing examples focus on textual files. What is the good way to specify grammar for a binary parser?

I don't think that needs to be the case. For example, flex lets you specify input characters in terms of octal or hex values.

share|improve this answer
    
As an aside, lex and yacc compile to native c code. You can then compile that into a c dll which can be called from most other languages. You can even have multiple parsers and have the calling language switch between them based upon user input, or from testing the file. –  Spencer Rathbun Jul 11 '12 at 19:11
    
Thanks, +1. Since we have tight deadlines, I will certainly try to play with some simple projects before we start to see if it's feasible to go this way. Considering that no member of our team has experience with that, we might also end up doing it by hand like we did so far. –  Dilbert Jul 12 '12 at 8:23

Depending on how complex your file formats are, I suspect you'd be better off writing a utility parsing library and writing your lexers/parsers by hand. I've done this before for parsing BAF records from phone switches and you can crank out the basic framework pretty quickly (a couple of days). If you were already familiar with parsing tools like flex and bison then I'd second Caleb's suggestion, but since you're not I'd suggest sticking with what you know. I assume that nobody on your team is familiar with parsing tools either, so this way everyone should be able to maintain the code going forward (no 'single point of failure' when a parsing error/bug crops up).

share|improve this answer
    
+1 Thanks. We have tight deadlines, and we didn't consider this formal approach much, until now. So I believe we might end up doing the old way again. –  Dilbert Jul 12 '12 at 8:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.