Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've been developing a little language in my own time, and I've got a fairly bare-bones prototype that implements some of the most basic features. Now I'm looking at scaling up, and I don't want to make a nasty architectural mistake where I have to massively refactor everything to support these features. So I'd like to know about the key API features required. Right now, I have

The lexer can operate on any input buffer using iterators, the tokens are available for inspection before passing to the parser, and I also provide a hook for comments.

The parser can operate on any token stream (not hard coded to my lexer at all), and the AST construction can be swapped out. I also annotated my AST with location information. Also, the AST is readily available for inspection prior to analysis.

Finally, many of the analysis features are lazily evaluated, so it's easy to get their semantic representation from the AST. But I'm not sure what features those semantic representations need to be readily available for such efforts.

Any suggestions as to what is necessary in this regard?

share|improve this question
Incrementally updating your AST on small changes to the file might be useful. – CodesInChaos Mar 8 '13 at 14:29
ctags and appear to be used to help in this capacity. – user40980 Mar 8 '13 at 14:34

For examples of what functionality can be offered, see Using Clang as a Library. Clang's AST matching uses C++ templates and macros, which makes it look fairy different compiler examples in ML or Haskell. Consider this example that "will capture all for statements that define a new variable initialized to zero."

StatementMatcher LoopMatcher =
share|improve this answer

In regards to your question headline: "What API is required for code completion/syntax highlighting/similar IDE features?", it might be useful to check out the Qt Creator Development Suite if you looking to develop your own IDE. Qt allows for the deployment of cross-platform software, and it supports many of the features listed above (code completion, syntax highlighting, etc.). Not sure if this is what you were looking for, but between the question headline and the question description I got a little confused.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.