I've been developing a little language in my own time, and I've got a fairly bare-bones prototype that implements some of the most basic features. Now I'm looking at scaling up, and I don't want to make a nasty architectural mistake where I have to massively refactor everything to support these features. So I'd like to know about the key API features required. Right now, I have
The lexer can operate on any input buffer using iterators, the tokens are available for inspection before passing to the parser, and I also provide a hook for comments.
The parser can operate on any token stream (not hard coded to my lexer at all), and the AST construction can be swapped out. I also annotated my AST with location information. Also, the AST is readily available for inspection prior to analysis.
Finally, many of the analysis features are lazily evaluated, so it's easy to get their semantic representation from the AST. But I'm not sure what features those semantic representations need to be readily available for such efforts.
Any suggestions as to what is necessary in this regard?