ANTLR4: How to inject tokens

Question

I\'m trying to implement a preprocessor for a DSL, modeled after the CPP example in code/extras. However, I\'m not using token factory. Is one required? Calling emit(toke

Answer 1

You need to override Lexer.nextToken to provide this feature. In your lexer, keep a Deque of injected tokens that have not yet been returned by nextToken. When the queue is empty, your implementation of nextToken should return the next token according to the superclass implementation.

Here's some sample code. I have not tried to compile or run it so it might not be perfect.

private final Deque pendingTokens = new ArrayDeque<>();

@Override
public Token nextToken() {
    Token pending = pendingTokens.pollFirst();
    if (pending != null) {
        return pending;
    }

    Token next = super.nextToken();
    pending = pendingTokens.pollFirst();
    if (pending != null) {
        pendingTokens.addLast(next);
        return pending;
    }

    return next;
}