Pythonic way to implement a tokenizer

Question

I\'m going to implement a tokenizer in Python and I was wondering if you could offer some style advice?

I\'ve implemented a tokenizer before in C and in Java so I\'m

Answer 1

When I start something new in Python I usually look first at some modules or libraries to use. There's 90%+ chance that there already is somthing available.

For tokenizers and parsers this is certainly so. Have you looked at PyParsing ?