a lexer is a program performing lexical analysis: it converts a sequence of characters into a sequence of tokens.

learn more… | top users | synonyms

0
votes
2answers
157 views

Lexer/Parser for multidimensional Languages

How does Lexer/Parser work in a 2D programming language like Funciton in order to transform such an unusual source-code to the correct AST?
1
vote
1answer
62 views

Should a lexer un-escape strings?

Is it a lexer's job to undo any escaping done to a string literal? For example: "Me: \"Hello World!\"" Becomes: Me: "Hello World!" Should this conversion be done inside the lexer? I am guessing ...
2
votes
4answers
292 views

Lexing: One token per operator, or one universal operator token?

When lexing, what would be the best way to tokenize operators? Would one just create a BinaryOperator token, or a separate token for every single binary operator? Examples: PlusOperator, ...
0
votes
2answers
70 views

Storing tokens during lexing stage

I am currently implementing a lexer that breaks XML files up into tokens, I'm considering ways of passing the tokens onto a parser to create a more useful data structure out of said tokens - my ...
0
votes
0answers
142 views

Parsing Razor-style Templates

I want to build a template engine (ITT not another template engine...) based on Razor. I've been at it for quite a long time not getting anywhere and quite frankly I'm at my limit. I've tried rolling ...
4
votes
2answers
286 views

Can you apply the same lexer rules to all programming languages?

I'm trying to understand the theory behind a lexer with the purpose of building one (just for my own fun and experience and to compensate for not taking proper CS courses :)). What I have yet to ...
6
votes
3answers
284 views

Clarification about Grammars , Lexers and Parsers

Background info (May Skip): I am working on a task we have been set at uni in which we have to design a grammar for a DSL we have been provided with. The grammar must be in BNF or EBNF. As well as ...
4
votes
3answers
234 views

What is the proper way to distinguish between keywords and identifiers?

I'm aware that most modern languages use reserved words to prevent things like keywords from being used as identifiers. Reserved words aside, let's assume a language that allows keywords to be used ...
4
votes
3answers
166 views

Is this a viable approach to resolving multiple matches in a lexer?

I'm writing a lexer in JavaScript. It's pretty typical - rules are specified with regular expressions and produce a token. I am unsure of the best way to handle when multiple rules are matched. The ...
3
votes
1answer
195 views

Chosing a parser for a code beautifier

I'm in the planning stage of making a code beautifier (similar to AStyle or Uncrustify) - originally I was going to just contribute to one of those projects, but reviewing their source led me to the ...
2
votes
4answers
340 views

Is it possible to create a single tokenizer to parse this?

This extends off this other Q&A thread, but is going into details that are out of scope from the original question. I am generating a parser that is to parse a context-sensitive grammar which can ...
-2
votes
2answers
279 views

What is the name of a grammar which can change its tokenizer in mid parse?

I was creating a language and discovered that my language tokenizer would have to change depending where in the parse it is. I.e. abc[1] would be parsed as 4 tokens (abc, [, 1, ]), where as { abc[1] ...
0
votes
0answers
68 views

Considerations for beginning work on a unified search

I have become interested in creating a unified search for a corporate asset management database. My goal is to allow users to submit queries like: stuff in building 3210 stuff in building 3210 owned ...
2
votes
1answer
216 views

Practical reference for learning about graph reduction

Are there any practical references (with actual examples) for getting started implementing a small, lazy functional programming language with graph reduction? A reference that included the lexing and ...
2
votes
1answer
271 views

How are “Json.org”-like specs graphs called and how can I generate them?

In http://www.json.org Douglas Crockford shows the specs of the JSON format in two interesting ways: In the right side column he lists a text spec that looks like a YACC or LEX listing. In the main ...
3
votes
1answer
207 views

What follows after lexical analysis?

I'm working on a toy compiler (for some simple language like PL/0) and I have my lexer up and running. At this point I should start working on building the parse tree, but before I start I was ...
2
votes
5answers
1k views

Understanding hand written lexers

I am going to make a compiler for C (C99; I own the standards PDF), written in C (go figure) and looking up on how compilers work on Wikipedia has told me a lot. However, after reading up on lexers ...
9
votes
5answers
1k views

Coming up with tokens for a lexer

I'm writing a parser for a markup language that I have created (writing in python, but that's not really relevant to this question -- in fact if this seems like a bad idea, I'd love a suggestion for a ...
7
votes
5answers
817 views

Lexical Analysis without regular expressions

I've been looking at a few lexers in various higher level langauges (Python, PHP, Javascript among others) and they all seem to use regular expressions in one form or another. While I'm sure regex's ...
12
votes
5answers
2k views

Are separate parsing and lexing passes good practice with parser combinators?

When I began to use parser combinators my first reaction was a sense of liberation from what felt like an artificial distinction between parsing and lexing. All of a sudden everything was just ...
7
votes
4answers
7k views

Writing a lexer in C++

What are good resources on how to write a lexer in C++ (books, tutorials, documents), what are some good techniques and practices? I have looked on the internet and everyone says to use a lexer ...