Antlr3: Could not match token in parser rules which is used in lexer rule

Question

I have lexer rules in Antlr3 as:

HYPHEN : '-';

TOKEN : HYPHEN CHARS;

CHARS : 'a' ..'z';

Parser rule is as:

exp : CHARS | some complex expression;
parser_rule : exp HYPHEN exp;

If I try to match 'abc-abc' with parser_rule, It fails. Because lexer creates TOKEN for HYPHEN exp. How can I match it correctly with parser_rule.

Answer 1

In ANTLR lexer, the lexer rule that can match the longest sub-sequence of input is used. So your input

abc-abc

will be tokenized as

CHARS("abc")
TOKEN("-abc")

and therefore will not match the expected CHARS HYPHEN CHARS.

You should consider making TOKEN a parser rule instead of lexer rule.