Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm learning how to build parsers using grammars, but I got stuck trying to express comments, because they can appear almost anywhere.

This indicates that comments can be stripped from the token stream before parsing takes place.

Is that the standard practice, or are comments ever/often specified in grammars?

share|improve this question
    
"They can appear alomost everywhere" is somewhat of a stretch. There are quite a few languages that only allow comments between statements. –  MSalters Jun 22 '12 at 14:45
1  
As an example of a language that does NOT remove the comments before parsing, see TCL - the comments are part of the grammar and if you put them in the wrong place, you can expect the parser to yell at you. –  Michael Kohne Jun 22 '12 at 18:32
    
@Michael I can't find a grammar for TCL, though: this page seems to indicate there isn't a static one. –  Matt Fenwick Jun 25 '12 at 15:04
1  
@MattFenwick - good gravy. I didn't know it was that malleable. According to what that says, you can't have a TCL grammer, it doesn't make sense. I had no idea, sorry for the false lead. –  Michael Kohne Jun 25 '12 at 17:27

4 Answers 4

up vote 2 down vote accepted

It is very common to treat it as some form of white space. Much in the same way as newlines in semicolon-oriented languages like C.

Once it is some form of white-space, you frequently just ignore it higher in the parser.

share|improve this answer

They usually aren't.

They are removed by the lexer, when source code is transformed from characters to tokens.

Then, the parser will get tokens and build an AST. When the parser does its job, comment are already gone, so they don't have to appear in the grammar.

share|improve this answer

Quick Short Answer

Yes, its the standard practice, to detect comments in the "lexer" or the "parser"

Sometimes, the "parser" has a built-in "lexer" or is mixed with the "parser" as a single tool ("lexer-parser").

Extended Answer

I just working in that case.

Most "scanners" ( also know as "tokenizers" or "lexers" ), detect comments, but, removed when returning tokens to the "parser".

Sometimes, a programming language, uses some comments with a special meaning, like "compiler directives" or "documentation".

Standard Comment Example:

/*
 This function does something cool.
*/
int doSomething()
{
  return 0;
}

Directive Comment Example:

/*
 ##override
*/
int doSomething()
{
  return 0;
}

Comment for documentation generator Example:

/*
 @description: This Function text will be turn,
  into an external pdf file, togheter with other
  similar comments.
*/
int doSomething()
{
  return 0;
}

Most compiler-related tools, detect this special comments, with an analyzer or preprocessor, that is not the main lezxer or parser, and, even has a small lexer of its own.

Cheers.

share|improve this answer

Some parsers that retain comments and/or illegal inputs in the AST add them as properties to all nodes of the AST.

You could take a look at the Microsoft roslyn project which does that if I recall correctly.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.