I need to develop a parser. Can I use Lex and Yacc for the purpose?

Question

I need to extract very particular data from log files(of different types and formats). Since I am a recent college passout; my mind ran to using Lex and Yacc for the purpose. Now I have the following Questions

1. Will it be legal to do so ? (This product I am working for belongs to one of the biggest tech companies in the world.)

2. Also ; I would like to know if I am being too afraid to write my own parser?

3. How can I use Lex and Yacc if my product is Windows based?

Please tell me if you need any clarification or extra information.

Answer 1

Before I answer your questions let me first start to say that there are two major classes of computing languages:

• Regular languages: Most log files follow this format. Each line is a full expression, and can be parsed separately. Essentially, all your needs can be met by just a lexer or regular expressions.
• Syntactical languages: There is a specific grammar and rules surrounding what makes proper syntax (or sentence structure). Most programming languages and even some markup languages follow this format. Essentially you need both a lexer and a parser.

If your need is parsing log files, you can probably get away with reading them line by line and using a regular expression to get the fields. If the log line is very simple, you can easily hand roll your parser and avoid the syntax ugliness of regular expressions. I've done both approaches and it's fairly quick work.

If you need something a bit more robust, then you can use ANTLR, Flex, Yacc, etc. depending on your platform needs. Understand that ANTLR, while itself is a Java tool, can generate C/C++ and C# code as well as Java code.

Now for your questions one by one:

1. Is it legal? Absolutely. Most parser generators explicitly put in their licenses that the license does not apply to the generated code. If the source code is proprietary, so is the generated code that came from it.
2. Am I just being afraid of writing my own parser? Only you can answer that. However, common sense says that if there is a tool to help you save time and avoid errors, you should probably use it. All the more if it doesn't cost extra money. Just make sure the tool you pick fits the job. If all you need to parse is a regular language, then don't opt for the full fledged parser generators.
3. Yacc and Lex on Windows? You can use an alternative like ANTLR, or you can use the Cygwin based tools, or the Bumble-Bee port. You have options.

Answer 2

There are much easier to learn and more powerful tools available today.

1. ANTLR this is one of the easiest to use and best supported tools on the market. Great books on how to use it by the author as well.
2. PyParsing is useful for lighter weight tasks and prototyping things you can then go and re-implement in ANTLR.

Answer 3

1. Yes, it is. Using GPL'd software places no obligation on you, except that if you extend GPL'd code and distribute the resulting program, you must also distribute the extended source code. So for internal use the GPL is irrelevant. Also, if you don't extend flex or yacc, but just use them, you are under no obligations either (but be aware that the question "What constitutes a derived work?" is controversial.)
2. Probably. It's not that hard, although parsing log files is usually simple enough that regular expressions are even simpler and faster.
3. Why wouldn't you? Get the Cygwin versions, or a dedicated Windows port (like e.g. Bumble-Bee), and you're good. The few calls you need can easily be integrated into your build whether you use Make/Ant or Eclipse/Visual Studio/whatever IDE.

Answer 4

I think if you use a parser generator, that would be a reasonable and practical way to build a parser. Doing so has its advantages as it's easier to work with a BNF grammar than a lot of code.

That said, it's also at times very educational to write your own parser. One can begin to appreciate lex and yacc and the science behind it once one tackles parsing challenges directly.

Writing your own parser can also have other benefits. First, it can mesh with your existing code base better (if you design it that way). Second, you might see better performance with your own hand-coded tokenizer if it's written pragmatically (rather than, say, regexing every token).

Answer 5

Learn Perl. Perl is the super deluxe version of the Swiss Armyknife of text file wrangling, and by having a few regular expressions with parentesis marking what you need, it is very frequently possible to do even advanced log file extraction in a few lines of Perl.