lexer

I'm writing an Eclipse/Xtext plugin for CoffeeScript, and I realized I'll probably need to write a lexer for it by hand. CoffeeScript parser also uses a hand-written lexer to handle indentation and other tricks in the grammar. Xtext generates a class that extends org.eclipse.xtext.parser.antlr.Lexer which in turn extends org.antlr.runtime.Lexer . So I suppose I'll have extend it. I can see two ways to do that Override mTokens() . This is done by the generated code, changing the internal state. Override nextToken() which seems a natural approach, but then I'll have to keep track of the internal

I was writing a parser to parse C-like grammars. First, it could now parse code like: a = 1; b = 2; Now I want to make the semicolon at the end of line optional. The original YACC rule was: stmt: expr ';' { ... } Where the new line is processed by the lexer that written by myself(the code are simplified): rule(/\r\n|\r|\n/) { increase_lineno(); return :PASS } the instruction :PASS here is equivalent to return nothing in LEX, which drop current matched text and skip to the next rule, just like what is usually done with whitespaces. Because of this, I can't just simply change my YACC rule into:

Are there any existing C++ grammar files for ANTLR? I'm looking to lex, not parse some C++ source code files. I've looked on the ANTLR grammar page and it looks like there is one listed created by Sun Microsystems here . However, it seems to be a generated Parser. Can anyone point me to a C++ ANTLR lexer or grammar file? C++ parsers are tough to build. I can't speak with experience about using ANTLR's C++ grammars. Here I discuss what I learned by reading the notes attached to the the one I did see at the ANTLR site; in essence, the author produced an incomplete grammar. And that was for just

I'm writing a lexer for the IMAP protocol for educational purposes and I'm stumped as to where I should draw the line between lexer and parser. Take this example of an IMAP server response: * FLAGS (\Answered \Deleted) This response is defined in the formal syntax like this: mailbox-data = "FLAGS" SP flag-list flag-list = "(" [flag *(SP flag)] ")" flag = "\Answered" / "\Deleted" Since they are specified as string literals (aka "terminal" tokens) would it be more correct for the lexer to emit a unique token for each, like: (TknAnsweredFlag) (TknSpace) (TknDeletedFlag) Or would it be just as

I would like to create an automatic Flowchart-like visualization to simple Java Logic, for this I need to parse Java Source code, I have 2 candidates, ANTLR and javax.lang.model of Java 6. Neither are easy. I have yet to find a single working example that will be even remotely close to what I want to achieve. I want to find simple variable declarations, assignments, and flows (if, for, switch, boolean conditions etc) Is there a simple example or tutorial for either of these? I found very few ANTLR examples (non of them are working out of the box without significant "homework") and absolutely

I'm taking a course in compiler construction, and my current assignment is to write the lexer for the language we're implementing. I can't figure out how to satisfy the requirement that the lexer must recognize concatenated tokens. That is, tokens not separated by whitespace. E.g.: the string 39if is supposed to be recognized as the number 39 and the keyword if . Simultaneously, the lexer must also exit(1) when it encounters invalid input. A simplified version of the code I have: %{ #include <stdio.h> %} %option main warn debug %% if | then | else printf("keyword: %s\n", yytext); [[:digit:]]+