a lexer is a program performing lexical analysis: it converts a sequence of characters into a sequence of tokens.

learn more… | top users | synonyms

15
votes
4answers
1k views

Why implement a lexer as a 2d array and a giant switch?

I'm slowly working to finish my degree, and this semester is Compilers 101. We're using the Dragon Book. Shortly into the course and we're talking about lexical analysis and how it can be implemented ...
1
vote
2answers
194 views

Lexer/Parser for multidimensional Languages

How does Lexer/Parser work in a 2D programming language like Funciton in order to transform such an unusual source-code to the correct AST?
1
vote
1answer
72 views

Should a lexer un-escape strings?

Is it a lexer's job to undo any escaping done to a string literal? For example: "Me: \"Hello World!\"" Becomes: Me: "Hello World!" Should this conversion be done inside the lexer? I am guessing ...
2
votes
4answers
326 views

Lexing: One token per operator, or one universal operator token?

When lexing, what would be the best way to tokenize operators? Would one just create a BinaryOperator token, or a separate token for every single binary operator? Examples: PlusOperator, ...
0
votes
2answers
79 views

Storing tokens during lexing stage

I am currently implementing a lexer that breaks XML files up into tokens, I'm considering ways of passing the tokens onto a parser to create a more useful data structure out of said tokens - my ...
0
votes
0answers
221 views

Parsing Razor-style Templates

I want to build a template engine (ITT not another template engine...) based on Razor. I've been at it for quite a long time not getting anywhere and quite frankly I'm at my limit. I've tried rolling ...
4
votes
2answers
333 views

Can you apply the same lexer rules to all programming languages?

I'm trying to understand the theory behind a lexer with the purpose of building one (just for my own fun and experience and to compensate for not taking proper CS courses :)). What I have yet to ...
6
votes
3answers
344 views

Clarification about Grammars , Lexers and Parsers

Background info (May Skip): I am working on a task we have been set at uni in which we have to design a grammar for a DSL we have been provided with. The grammar must be in BNF or EBNF. As well as ...
3
votes
3answers
254 views

What is the proper way to distinguish between keywords and identifiers?

I'm aware that most modern languages use reserved words to prevent things like keywords from being used as identifiers. Reserved words aside, let's assume a language that allows keywords to be used ...
4
votes
3answers
169 views

Is this a viable approach to resolving multiple matches in a lexer?

I'm writing a lexer in JavaScript. It's pretty typical - rules are specified with regular expressions and produce a token. I am unsure of the best way to handle when multiple rules are matched. The ...
3
votes
1answer
250 views

Chosing a parser for a code beautifier

I'm in the planning stage of making a code beautifier (similar to AStyle or Uncrustify) - originally I was going to just contribute to one of those projects, but reviewing their source led me to the ...
1
vote
4answers
383 views

Is it possible to create a single tokenizer to parse this?

This extends off this other Q&A thread, but is going into details that are out of scope from the original question. I am generating a parser that is to parse a context-sensitive grammar which can ...
-3
votes
2answers
285 views

What is the name of a grammar which can change its tokenizer in mid parse?

I was creating a language and discovered that my language tokenizer would have to change depending where in the parse it is. I.e. abc[1] would be parsed as 4 tokens (abc, [, 1, ]), where as { abc[1] ...
0
votes
0answers
69 views

Considerations for beginning work on a unified search

I have become interested in creating a unified search for a corporate asset management database. My goal is to allow users to submit queries like: stuff in building 3210 stuff in building 3210 owned ...
2
votes
1answer
241 views

Practical reference for learning about graph reduction

Are there any practical references (with actual examples) for getting started implementing a small, lazy functional programming language with graph reduction? A reference that included the lexing and ...
2
votes
1answer
327 views

How are “Json.org”-like specs graphs called and how can I generate them?

In http://www.json.org Douglas Crockford shows the specs of the JSON format in two interesting ways: In the right side column he lists a text spec that looks like a YACC or LEX listing. In the main ...
2
votes
1answer
212 views

What follows after lexical analysis?

I'm working on a toy compiler (for some simple language like PL/0) and I have my lexer up and running. At this point I should start working on building the parse tree, but before I start I was ...
2
votes
5answers
2k views

Understanding hand written lexers

I am going to make a compiler for C (C99; I own the standards PDF), written in C (go figure) and looking up on how compilers work on Wikipedia has told me a lot. However, after reading up on lexers ...
9
votes
5answers
2k views

Coming up with tokens for a lexer

I'm writing a parser for a markup language that I have created (writing in python, but that's not really relevant to this question -- in fact if this seems like a bad idea, I'd love a suggestion for a ...
7
votes
5answers
905 views

Lexical Analysis without regular expressions

I've been looking at a few lexers in various higher level langauges (Python, PHP, Javascript among others) and they all seem to use regular expressions in one form or another. While I'm sure regex's ...
11
votes
5answers
2k views

Are separate parsing and lexing passes good practice with parser combinators?

When I began to use parser combinators my first reaction was a sense of liberation from what felt like an artificial distinction between parsing and lexing. All of a sudden everything was just ...
8
votes
4answers
9k views

Writing a lexer in C++

What are good resources on how to write a lexer in C++ (books, tutorials, documents), what are some good techniques and practices? I have looked on the internet and everyone says to use a lexer ...