Can JavaCC distinguish token by its context?

Question

Basic requirement is use keyword as identifier, so I want to distinguish the token from it\'s context.(e.g.class is a keyword, but we allowed a variable named <

Answer 1

If you merge the lexer and parser into a character-oriented parser, then it is relatively easy to distinguish keywords in context, because the parser is all about retaining context. You could operate JavaCC on character tokens to achieve this effect, but its LL nature would probably make it impossible to write practical grammars for other reasons.

If you separate lexer and parser, this isn't easy.

You are asking the lexer to know when something is an identifier or a keyword, which it can only do by knowing the context which the Id/keyword is found.

Ideally the lexer would simply ask the parser for its state, and that would identify the contexts in which the choice is made. That's hard to organize; most parsers aren't designed to reveal their state easily or in a form easy to interpret for extracting the context signal needed. JavaCC isn't obviously organized this way.

Your other obvious choice is to model the different contexts as states in the lexer, with transitions between lexing states corresponding to transitions between interesting contexts. This may or may not be easy depending the context. If you can do it, you have to code the states and the transitions in your lexer and keep them up to date. When you can do this "easily", it is not a bad solution. This can be hard or impossible depending the specific contexts.

For OPs purpose (apparantly a parser for an assembler), the context is usually determined by the position within the source line. One can qualitatively divide assembler input into Label, Opcode, Operand, Comment contexts by watching whitespace: A newline sets the context to Label, whitespace in Label mode sets context to Opcode, whitespace in Opcode sets Operand context, and whitespace in Operand context sets Comment context. With these state transitions, one can write different sublexers for each context, thus having different keywords in each subcontext.

This trick doesn't work for languages like PL/I, which have vast numbers of keywords in context (for PL/I, in fact, every keyword is only in context!).

A non-obvious choices is to not try to differentiate at all. When an Id/Keyword is found, feed both tokens to the parser, and let it sort out which one leads to a viable parse. (Note: it may have handle the cross product of multiple ambiguous tokens, thus many possible parses while sorting this out.) This requires a parser that can handle ambiguity, both while parsing, and in the tokens it accepts (or it can't accept both an ID and a Keyword token at the same time). This is a beautifully simple solution to use when you have the right parsing machinery. JavaCC isn't that machinery.

[See my bio for a GLR parsing engine in which all 3 solutions are easily accessible. It handles Pl/I easily.