How do I tokenize input using Java's Scanner class and regular expressions?

Question

Just for my own purposes, I\'m trying to build a tokenizer in Java where I can define a regular grammar and have it tokenize input based on that. The StringTokenizer class i

Answer 1

The name "Scanner" is a bit misleading, because the word is often used to mean a lexical analyzer, and that's not what Scanner is for. All it is is a substitute for the scanf() function you find in C, Perl, et al. Like StringTokenizer and split(), it's designed to scan ahead until it finds a match for a given pattern, and whatever it skipped over on the way is returned as a token.

A lexical analyzer, on the other hand, has to examine and classify every character, even if it's only to decide whether it can safely ignore them. That means, after each match, it may apply several patterns until it finds one that matches starting at that point. Otherwise, it may find the sequence "//" and think it's found the beginning of a comment, when it's really inside a string literal and it just failed to notice the opening quotation mark.

It's actually much more complicated than that, of course, but I'm just illustrating why the built-in tools like StringTokenizer, split() and Scanner aren't suitable for this kind of task. It is, however, possible to use Java's regex classes for a limited form of lexical analysis. In fact, the addition of the Scanner class made it much easier, because of the new Matcher API that was added to support it, i.e., regions and the usePattern() method. Here's an example of a rudimentary scanner built on top of Java's regex classes.

import java.util.*;
import java.util.regex.*;

public class RETokenizer
{
  static List tokenize(String source, List rules)
  {
    List tokens = new ArrayList();
    int pos = 0;
    final int end = source.length();
    Matcher m = Pattern.compile("dummy").matcher(source);
    m.useTransparentBounds(true).useAnchoringBounds(false);
    while (pos < end)
    {
      m.region(pos, end);
      for (Rule r : rules)
      {
        if (m.usePattern(r.pattern).lookingAt())
        {
          tokens.add(new Token(r.name, m.start(), m.end()));
          pos = m.end();
          break;
        }
      }
      pos++;  // bump-along, in case no rule matched
    }
    return tokens;
  }

  static class Rule
  {
    final String name;
    final Pattern pattern;

    Rule(String name, String regex)
    {
      this.name = name;
      pattern = Pattern.compile(regex);
    }
  }

  static class Token
  {
    final String name;
    final int startPos;
    final int endPos;

    Token(String name, int startPos, int endPos)
    {
      this.name = name;
      this.startPos = startPos;
      this.endPos = endPos;
    }

    @Override
    public String toString()
    {
      return String.format("Token [%2d, %2d, %s]", startPos, endPos, name);
    }
  }

  public static void main(String[] args) throws Exception
  {
    List rules = new ArrayList();
    rules.add(new Rule("WORD", "[A-Za-z]+"));
    rules.add(new Rule("QUOTED", "\"[^\"]*+\""));
    rules.add(new Rule("COMMENT", "//.*"));
    rules.add(new Rule("WHITESPACE", "\\s+"));

    String str = "foo //in \"comment\"\nbar \"no //comment\" end";
    List result = RETokenizer.tokenize(str, rules);
    for (Token t : result)
    {
      System.out.println(t);
    }
  }
}

This, by the way, is the only good use I've ever found for the lookingAt() method. :D