How do I tokenize input using Java's Scanner class and regular expressions?

Question

Just for my own purposes, I\'m trying to build a tokenizer in Java where I can define a regular grammar and have it tokenize input based on that. The StringTokenizer class i

Answer 1

Most of the answers here are already excellent but I would be remiss if I didn't point out ANTLR. I've created entire compilers around this excellent tool. Version 3 has some amazing features and I'd recommend it for any project that required you to parse input based on a well defined grammar.